You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/09/25 21:46:11 UTC
(datafusion-comet) branch asf-site updated: Publish built docs triggered by b01c168b846cd55286b523d4cd2b458308cabb97
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new adccd5d058 Publish built docs triggered by b01c168b846cd55286b523d4cd2b458308cabb97
adccd5d058 is described below
commit adccd5d0581dca6fe0ecd4df36c212807fd20cb5
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Fri Sep 25 21:46:05 2026 +0000
Publish built docs triggered by b01c168b846cd55286b523d4cd2b458308cabb97
---
...id-013012468fa27c9e9fdb0a14196d357c619c68a8.svg | 2 +-
_sources/contributor-guide/release_process.md.txt | 6 -
.../user-guide/1.0/s3-credential-providers.md.txt | 29 +++++
.../latest/compatibility/expressions/index.md.txt | 3 +
.../expressions/spark-3.4/cast.md.txt | 34 +++--
.../expressions/spark-3.4/index.md.txt | 3 +
.../compatibility/expressions/spark-3.4/map.md.txt | 2 +-
.../expressions/spark-3.5/cast.md.txt | 34 +++--
.../compatibility/expressions/spark-3.5/map.md.txt | 2 +-
.../expressions/spark-4.0/cast.md.txt | 34 +++--
.../compatibility/expressions/spark-4.0/map.md.txt | 2 +-
.../expressions/spark-4.1/cast.md.txt | 34 +++--
.../compatibility/expressions/spark-4.1/map.md.txt | 2 +-
.../user-guide/latest/compatibility/index.md.txt | 30 +----
.../latest/compatibility/operators.md.txt | 19 ++-
.../user-guide/latest/compatibility/scans.md.txt | 2 +
.../latest/compatibility/spark-versions.md.txt | 33 ++---
_sources/user-guide/latest/datasources.md.txt | 12 +-
_sources/user-guide/latest/datatypes.md.txt | 19 +--
_sources/user-guide/latest/expressions.md.txt | 69 +++++-----
_sources/user-guide/latest/iceberg-writes.md.txt | 11 +-
_sources/user-guide/latest/iceberg.md.txt | 16 ++-
_sources/user-guide/latest/in-memory-cache.md.txt | 5 +-
_sources/user-guide/latest/installation.md.txt | 39 ++++--
_sources/user-guide/latest/kubernetes.md.txt | 16 ++-
_sources/user-guide/latest/metrics.md.txt | 92 ++++++++-----
_sources/user-guide/latest/migration-guide.md.txt | 7 +
_sources/user-guide/latest/operators.md.txt | 44 ++++---
_sources/user-guide/latest/pyarrow-udfs.md.txt | 30 ++---
.../latest/s3-credential-providers.md.txt | 14 +-
_sources/user-guide/latest/scala_java_udfs.md.txt | 4 +-
_sources/user-guide/latest/source.md.txt | 2 +-
_sources/user-guide/latest/tuning.md.txt | 53 ++++----
.../latest/understanding-comet-plans.md.txt | 50 ++++---
contributor-guide/release_process.html | 4 -
searchindex.js | 2 +-
user-guide/1.0/s3-credential-providers.html | 24 ++++
.../latest/compatibility/expressions/index.html | 2 +
.../compatibility/expressions/spark-3.4/cast.html | 38 +++---
.../compatibility/expressions/spark-3.4/index.html | 2 +
.../compatibility/expressions/spark-3.4/map.html | 2 +-
.../compatibility/expressions/spark-3.5/cast.html | 38 +++---
.../compatibility/expressions/spark-3.5/map.html | 2 +-
.../compatibility/expressions/spark-4.0/cast.html | 38 +++---
.../compatibility/expressions/spark-4.0/map.html | 2 +-
.../compatibility/expressions/spark-4.1/cast.html | 38 +++---
.../compatibility/expressions/spark-4.1/map.html | 2 +-
user-guide/latest/compatibility/index.html | 31 +----
user-guide/latest/compatibility/operators.html | 19 ++-
user-guide/latest/compatibility/scans.html | 2 +
.../latest/compatibility/spark-versions.html | 31 ++---
user-guide/latest/datasources.html | 11 +-
user-guide/latest/datatypes.html | 9 +-
user-guide/latest/expressions.html | 145 ++++++++++++---------
user-guide/latest/iceberg-writes.html | 11 +-
user-guide/latest/iceberg.html | 15 ++-
user-guide/latest/in-memory-cache.html | 5 +-
user-guide/latest/installation.html | 44 +++++--
user-guide/latest/kubernetes.html | 15 ++-
user-guide/latest/metrics.html | 87 +++++++++----
user-guide/latest/migration-guide.html | 7 +
user-guide/latest/operators.html | 24 ++--
user-guide/latest/pyarrow-udfs.html | 30 ++---
user-guide/latest/s3-credential-providers.html | 9 +-
user-guide/latest/scala_java_udfs.html | 4 +-
user-guide/latest/source.html | 2 +-
user-guide/latest/tuning.html | 56 ++++----
user-guide/latest/understanding-comet-plans.html | 58 +++++++--
68 files changed, 913 insertions(+), 650 deletions(-)
diff --git a/_images/mermaid-013012468fa27c9e9fdb0a14196d357c619c68a8.svg b/_images/mermaid-013012468fa27c9e9fdb0a14196d357c619c68a8.svg
index 81bb1c9807..5fab12d554 100644
--- a/_images/mermaid-013012468fa27c9e9fdb0a14196d357c619c68a8.svg
+++ b/_images/mermaid-013012468fa27c9e9fdb0a14196d357c619c68a8.svg
@@ -1 +1 @@
-<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 942" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
+<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 942" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
diff --git a/_sources/contributor-guide/release_process.md.txt b/_sources/contributor-guide/release_process.md.txt
index e490cbec25..842ace9dcd 100644
--- a/_sources/contributor-guide/release_process.md.txt
+++ b/_sources/contributor-guide/release_process.md.txt
@@ -287,9 +287,6 @@ git tag 0.13.0-rc1
git push apache 0.13.0-rc1
```
-Note that pushing a release candidate tag will trigger a GitHub workflow that will build a Docker image and publish
-it to GitHub Container Registry at https://github.com/apache/datafusion-comet/pkgs/container/datafusion-comet
-
### Publishing Documentation
In `docs` directory:
@@ -468,9 +465,6 @@ git tag 0.13.0
git push apache 0.13.0
```
-Note that pushing a release tag will trigger a GitHub workflow that will build a Docker image and publish
-it to GitHub Container Registry at https://github.com/apache/datafusion-comet/pkgs/container/datafusion-comet
-
Reply to the vote thread to close the vote and announce the release. The announcement email should include:
- The release version
diff --git a/_sources/user-guide/1.0/s3-credential-providers.md.txt b/_sources/user-guide/1.0/s3-credential-providers.md.txt
index 33777c497a..df3d76005b 100644
--- a/_sources/user-guide/1.0/s3-credential-providers.md.txt
+++ b/_sources/user-guide/1.0/s3-credential-providers.md.txt
@@ -201,6 +201,35 @@ public CometS3Credentials getCredentialsForPath(CometS3CredentialContext ctx) th
}
```
+### Credentials per location
+
+On the Parquet path, a `CometS3CredentialProvider` gets one credential per bucket: Comet requests it with the path of the first file it reads from the bucket and uses it for every file there. If your policies differ by location within a bucket, for example one policy for `warehouse/sales` and another for `warehouse/finance`, implement `CometS3LocationScopedCredentialProvider` and tell Comet where those locations are:
+
+```java
+public final class MyLocationProvider implements CometS3LocationScopedCredentialProvider {
+ @Override
+ public List<String> getPolicyLocations(String bucket) throws Exception {
+ return policyService.locationsWithPolicies(bucket); // ["warehouse/sales", "warehouse/finance"]
+ }
+
+ @Override
+ public CometS3Credentials getCredentialsForPath(CometS3CredentialContext ctx) throws Exception {
+ // ctx.getPath() is a returned location with a leading slash, or "/" for the bucket root.
+ return sessionForLocation(ctx.getBucket(), ctx.getPath(), ctx.getMode());
+ }
+}
+```
+
+Comet serves each request with the credential of the longest location that covers its path. A location covers a path when the path is the location itself or lies below it, compared one `/`-separated segment at a time, so `warehouse/sales` covers `warehouse/sales/part-0.parquet` but not `warehouse/sales_eu/part-0.parquet`. The bucket root covers every path that no returned location covers, and an empty list serves the whole bucket with the root's credential. Write locations the way `Comet [...]
+
+Comet requests a location's credential by calling `getCredentialsForPath` with the location as the path, as you returned it but with a leading slash. Every request under a location shares that credential, so it must authorize every path the location is the longest match for, and your cache can key on the location. Locations apply to Comet's native Parquet reads only; Iceberg reads call `getCredentialsForPath` as they do for any provider.
+
+**When Comet asks.** Comet calls `getPolicyLocations` when it creates the store for a bucket on an executor and keeps the answer for later reads of that bucket with the same S3 configuration. Reads that start at the same moment may each create a store and call it. If a read then fails with 403, Comet asks again, once for all the reads that failed on the same answer, and retries each read once if its path now falls under a different location, so a location added while a job runs is picked [...]
+
+**Failures.** If `getPolicyLocations` throws or returns `null`, or returns a location that is `null` or invalid, the read fails. A location is invalid if, once decoded, it is not valid UTF-8 or has a segment that is empty, `.`, `..`, or contains a control character, so a URI such as `s3://bucket/a` is invalid too. Comet does not fall back to a broader credential.
+
+**Backward compatibility.** Providers that implement only `CometS3CredentialProvider` are unaffected. Comet calls nothing new on them and keeps one credential per bucket, as before.
+
### Composing multiple credential backends
A single configured provider class is the dispatcher. If a vendor needs to route across several credential backends (per bucket, per path prefix, per tenant), the dispatch lives inside the vendor's class:
diff --git a/_sources/user-guide/latest/compatibility/expressions/index.md.txt b/_sources/user-guide/latest/compatibility/expressions/index.md.txt
index 3574699195..8d28eee9f5 100644
--- a/_sources/user-guide/latest/compatibility/expressions/index.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/index.md.txt
@@ -32,6 +32,9 @@ Spark 4.0 <spark-4.0/index>
Spark 4.1 <spark-4.1/index>
```
+Spark 3.4 support is deprecated and will be removed in a future release; see
+[Spark Version Compatibility](../spark-versions.md#spark-34).
+
Expressions that are not 100% Spark-compatible fall back to Spark by default, except those
with a JVM codegen-dispatch path, which stay in Comet's native pipeline and match Spark
exactly. Set `spark.comet.expression.EXPRNAME.allowIncompatible=true`, where `EXPRNAME` is
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-3.4/cast.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-3.4/cast.md.txt
index 17b9ac592b..54241e629d 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-3.4/cast.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-3.4/cast.md.txt
@@ -22,17 +22,23 @@ under the License.
Cast operations in Comet fall into three levels of support:
- **C (Compatible)**: The results match Apache Spark
-- **I (Incompatible)**: The results may match Apache Spark for some inputs, but there are known issues where some inputs
- will result in incorrect results or exceptions. The query stage will fall back to Spark by default. Setting
- `spark.comet.expression.Cast.allowIncompatible=true` will allow all incompatible casts to run natively in Comet, but this is not
- recommended for production use.
-- **U (Unsupported)**: Comet does not provide a native version of this cast expression and the query stage will fall back to
- Spark.
+- **I (Incompatible)**: Comet's native cast may produce different results from Apache Spark for some inputs. By
+ default, Comet instead runs Spark's own code-generated cast inside the Comet pipeline, so the results match Spark.
+ Setting `spark.comet.expression.Cast.allowIncompatible=true` selects the native cast, but this is not recommended
+ for production use.
+- **U (Unsupported)**: Comet does not provide a native version of this cast, so it runs Spark's own code-generated
+ cast inside the Comet pipeline.
- **N/A**: Spark does not support this cast.
-## ANSI Mode Fallback
+Incompatible and unsupported casts fall back to Spark only when Comet's codegen dispatcher cannot handle them (for
+example, casts involving `VariantType`) or when `spark.comet.exec.scalaUDF.codegen.enabled=false`.
-Cast will fall back to Spark in some cases when ANSI mode is enabled. This can be enabled by setting `spark.comet.expression.Cast.allowIncompatible=true`. See the [Comet Supported Expressions Guide](../../../expressions.md) for more information on this configuration setting.
+## ANSI Mode Support
+
+Enabling ANSI mode does not by itself make a cast fall back to Spark or require
+`spark.comet.expression.Cast.allowIncompatible=true`: every cast that Spark permits in ANSI mode has the same support
+level as in legacy mode. See the [ANSI Mode](#ansi-mode) table below for per-pair support, and the
+[Comet Supported Expressions Guide](../../../expressions.md) for more information on the `allowIncompatible` setting.
## Whitespace Trimming in Casts from String
@@ -143,20 +149,12 @@ Comet's native `CAST(date AS STRING)` is compatible with Spark. Years below 1000
zero-padded to four digits (e.g. year 999 renders as `0999-01-01`). Years above 9999 are
rendered without truncation. The cast is timezone-independent.
-## String to TimestampNTZ
-
-Comet's native `CAST(string AS TIMESTAMP_NTZ)` implementation matches Apache Spark's behavior.
-Unlike `CAST(string AS TIMESTAMP)`, this cast is timezone-independent: any timezone offset in
-the input string (e.g. `+08:00`, `Z`, `UTC`) is silently discarded, and the local date-time
-components are preserved as-is. Time-only strings (e.g. `T12:34:56`, `12:34`) produce `NULL`.
-The result is always a wall-clock timestamp with no timezone conversion or DST adjustment.
-
## Decimal with Negative Scale to String
Casting a `DecimalType` with a negative scale to `StringType` is marked as incompatible when
`spark.sql.legacy.allowNegativeScaleOfDecimal` is `false` (the default). When that config is
-disabled, Spark cannot create negative-scale decimals, so Comet falls back to avoid running
-native execution on unexpected inputs.
+disabled, Spark cannot create negative-scale decimals, so by default Comet runs Spark's
+code-generated cast instead of the native one.
When `spark.sql.legacy.allowNegativeScaleOfDecimal=true`, the cast is compatible. Comet matches
Spark's behavior of using Java `BigDecimal.toString()` semantics, which produces scientific
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-3.4/index.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-3.4/index.md.txt
index 40fe54c4ea..0550dfb0c6 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-3.4/index.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-3.4/index.md.txt
@@ -27,6 +27,9 @@ expression class name, to run Comet's native implementation despite its differen
from Spark. See the [Comet Supported Expressions Guide](../../../expressions.md)
for more information on this configuration setting.
+Spark 3.4 support is deprecated and will be removed in a future release; see
+[Spark Version Compatibility](../../spark-versions.md#spark-34).
+
```{toctree}
:maxdepth: 1
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-3.4/map.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-3.4/map.md.txt
index ddbe8e5d18..3f2b190fa7 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-3.4/map.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-3.4/map.md.txt
@@ -26,7 +26,7 @@ keys, in `try_element_at`, and in other contexts where map ordering must be dete
runs `MapSort` natively, so map shuffle and group-by-on-map stay on Comet under Spark 4.0.
When `spark.comet.exec.strictFloatingPoint=true`, `MapSort` falls back to Spark for maps whose
-keys contain `Float` or `Double` (consistent with `SortOrder` and `SortArray`). Arrow's sort uses
+keys contain `Float` or `Double`. Arrow's sort uses
IEEE total ordering for floating-point, which differs from Spark's `Double.compare` semantics for
`NaN` and `-0.0`.
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-3.5/cast.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-3.5/cast.md.txt
index 17b9ac592b..54241e629d 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-3.5/cast.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-3.5/cast.md.txt
@@ -22,17 +22,23 @@ under the License.
Cast operations in Comet fall into three levels of support:
- **C (Compatible)**: The results match Apache Spark
-- **I (Incompatible)**: The results may match Apache Spark for some inputs, but there are known issues where some inputs
- will result in incorrect results or exceptions. The query stage will fall back to Spark by default. Setting
- `spark.comet.expression.Cast.allowIncompatible=true` will allow all incompatible casts to run natively in Comet, but this is not
- recommended for production use.
-- **U (Unsupported)**: Comet does not provide a native version of this cast expression and the query stage will fall back to
- Spark.
+- **I (Incompatible)**: Comet's native cast may produce different results from Apache Spark for some inputs. By
+ default, Comet instead runs Spark's own code-generated cast inside the Comet pipeline, so the results match Spark.
+ Setting `spark.comet.expression.Cast.allowIncompatible=true` selects the native cast, but this is not recommended
+ for production use.
+- **U (Unsupported)**: Comet does not provide a native version of this cast, so it runs Spark's own code-generated
+ cast inside the Comet pipeline.
- **N/A**: Spark does not support this cast.
-## ANSI Mode Fallback
+Incompatible and unsupported casts fall back to Spark only when Comet's codegen dispatcher cannot handle them (for
+example, casts involving `VariantType`) or when `spark.comet.exec.scalaUDF.codegen.enabled=false`.
-Cast will fall back to Spark in some cases when ANSI mode is enabled. This can be enabled by setting `spark.comet.expression.Cast.allowIncompatible=true`. See the [Comet Supported Expressions Guide](../../../expressions.md) for more information on this configuration setting.
+## ANSI Mode Support
+
+Enabling ANSI mode does not by itself make a cast fall back to Spark or require
+`spark.comet.expression.Cast.allowIncompatible=true`: every cast that Spark permits in ANSI mode has the same support
+level as in legacy mode. See the [ANSI Mode](#ansi-mode) table below for per-pair support, and the
+[Comet Supported Expressions Guide](../../../expressions.md) for more information on the `allowIncompatible` setting.
## Whitespace Trimming in Casts from String
@@ -143,20 +149,12 @@ Comet's native `CAST(date AS STRING)` is compatible with Spark. Years below 1000
zero-padded to four digits (e.g. year 999 renders as `0999-01-01`). Years above 9999 are
rendered without truncation. The cast is timezone-independent.
-## String to TimestampNTZ
-
-Comet's native `CAST(string AS TIMESTAMP_NTZ)` implementation matches Apache Spark's behavior.
-Unlike `CAST(string AS TIMESTAMP)`, this cast is timezone-independent: any timezone offset in
-the input string (e.g. `+08:00`, `Z`, `UTC`) is silently discarded, and the local date-time
-components are preserved as-is. Time-only strings (e.g. `T12:34:56`, `12:34`) produce `NULL`.
-The result is always a wall-clock timestamp with no timezone conversion or DST adjustment.
-
## Decimal with Negative Scale to String
Casting a `DecimalType` with a negative scale to `StringType` is marked as incompatible when
`spark.sql.legacy.allowNegativeScaleOfDecimal` is `false` (the default). When that config is
-disabled, Spark cannot create negative-scale decimals, so Comet falls back to avoid running
-native execution on unexpected inputs.
+disabled, Spark cannot create negative-scale decimals, so by default Comet runs Spark's
+code-generated cast instead of the native one.
When `spark.sql.legacy.allowNegativeScaleOfDecimal=true`, the cast is compatible. Comet matches
Spark's behavior of using Java `BigDecimal.toString()` semantics, which produces scientific
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-3.5/map.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-3.5/map.md.txt
index ddbe8e5d18..3f2b190fa7 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-3.5/map.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-3.5/map.md.txt
@@ -26,7 +26,7 @@ keys, in `try_element_at`, and in other contexts where map ordering must be dete
runs `MapSort` natively, so map shuffle and group-by-on-map stay on Comet under Spark 4.0.
When `spark.comet.exec.strictFloatingPoint=true`, `MapSort` falls back to Spark for maps whose
-keys contain `Float` or `Double` (consistent with `SortOrder` and `SortArray`). Arrow's sort uses
+keys contain `Float` or `Double`. Arrow's sort uses
IEEE total ordering for floating-point, which differs from Spark's `Double.compare` semantics for
`NaN` and `-0.0`.
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-4.0/cast.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-4.0/cast.md.txt
index 17b9ac592b..54241e629d 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-4.0/cast.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-4.0/cast.md.txt
@@ -22,17 +22,23 @@ under the License.
Cast operations in Comet fall into three levels of support:
- **C (Compatible)**: The results match Apache Spark
-- **I (Incompatible)**: The results may match Apache Spark for some inputs, but there are known issues where some inputs
- will result in incorrect results or exceptions. The query stage will fall back to Spark by default. Setting
- `spark.comet.expression.Cast.allowIncompatible=true` will allow all incompatible casts to run natively in Comet, but this is not
- recommended for production use.
-- **U (Unsupported)**: Comet does not provide a native version of this cast expression and the query stage will fall back to
- Spark.
+- **I (Incompatible)**: Comet's native cast may produce different results from Apache Spark for some inputs. By
+ default, Comet instead runs Spark's own code-generated cast inside the Comet pipeline, so the results match Spark.
+ Setting `spark.comet.expression.Cast.allowIncompatible=true` selects the native cast, but this is not recommended
+ for production use.
+- **U (Unsupported)**: Comet does not provide a native version of this cast, so it runs Spark's own code-generated
+ cast inside the Comet pipeline.
- **N/A**: Spark does not support this cast.
-## ANSI Mode Fallback
+Incompatible and unsupported casts fall back to Spark only when Comet's codegen dispatcher cannot handle them (for
+example, casts involving `VariantType`) or when `spark.comet.exec.scalaUDF.codegen.enabled=false`.
-Cast will fall back to Spark in some cases when ANSI mode is enabled. This can be enabled by setting `spark.comet.expression.Cast.allowIncompatible=true`. See the [Comet Supported Expressions Guide](../../../expressions.md) for more information on this configuration setting.
+## ANSI Mode Support
+
+Enabling ANSI mode does not by itself make a cast fall back to Spark or require
+`spark.comet.expression.Cast.allowIncompatible=true`: every cast that Spark permits in ANSI mode has the same support
+level as in legacy mode. See the [ANSI Mode](#ansi-mode) table below for per-pair support, and the
+[Comet Supported Expressions Guide](../../../expressions.md) for more information on the `allowIncompatible` setting.
## Whitespace Trimming in Casts from String
@@ -143,20 +149,12 @@ Comet's native `CAST(date AS STRING)` is compatible with Spark. Years below 1000
zero-padded to four digits (e.g. year 999 renders as `0999-01-01`). Years above 9999 are
rendered without truncation. The cast is timezone-independent.
-## String to TimestampNTZ
-
-Comet's native `CAST(string AS TIMESTAMP_NTZ)` implementation matches Apache Spark's behavior.
-Unlike `CAST(string AS TIMESTAMP)`, this cast is timezone-independent: any timezone offset in
-the input string (e.g. `+08:00`, `Z`, `UTC`) is silently discarded, and the local date-time
-components are preserved as-is. Time-only strings (e.g. `T12:34:56`, `12:34`) produce `NULL`.
-The result is always a wall-clock timestamp with no timezone conversion or DST adjustment.
-
## Decimal with Negative Scale to String
Casting a `DecimalType` with a negative scale to `StringType` is marked as incompatible when
`spark.sql.legacy.allowNegativeScaleOfDecimal` is `false` (the default). When that config is
-disabled, Spark cannot create negative-scale decimals, so Comet falls back to avoid running
-native execution on unexpected inputs.
+disabled, Spark cannot create negative-scale decimals, so by default Comet runs Spark's
+code-generated cast instead of the native one.
When `spark.sql.legacy.allowNegativeScaleOfDecimal=true`, the cast is compatible. Comet matches
Spark's behavior of using Java `BigDecimal.toString()` semantics, which produces scientific
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-4.0/map.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-4.0/map.md.txt
index 85ce8af197..98a501a42a 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-4.0/map.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-4.0/map.md.txt
@@ -26,7 +26,7 @@ keys, in `try_element_at`, and in other contexts where map ordering must be dete
runs `MapSort` natively, so map shuffle and group-by-on-map stay on Comet under Spark 4.0.
When `spark.comet.exec.strictFloatingPoint=true`, `MapSort` falls back to Spark for maps whose
-keys contain `Float` or `Double` (consistent with `SortOrder` and `SortArray`). Arrow's sort uses
+keys contain `Float` or `Double`. Arrow's sort uses
IEEE total ordering for floating-point, which differs from Spark's `Double.compare` semantics for
`NaN` and `-0.0`.
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-4.1/cast.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-4.1/cast.md.txt
index 17b9ac592b..54241e629d 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-4.1/cast.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-4.1/cast.md.txt
@@ -22,17 +22,23 @@ under the License.
Cast operations in Comet fall into three levels of support:
- **C (Compatible)**: The results match Apache Spark
-- **I (Incompatible)**: The results may match Apache Spark for some inputs, but there are known issues where some inputs
- will result in incorrect results or exceptions. The query stage will fall back to Spark by default. Setting
- `spark.comet.expression.Cast.allowIncompatible=true` will allow all incompatible casts to run natively in Comet, but this is not
- recommended for production use.
-- **U (Unsupported)**: Comet does not provide a native version of this cast expression and the query stage will fall back to
- Spark.
+- **I (Incompatible)**: Comet's native cast may produce different results from Apache Spark for some inputs. By
+ default, Comet instead runs Spark's own code-generated cast inside the Comet pipeline, so the results match Spark.
+ Setting `spark.comet.expression.Cast.allowIncompatible=true` selects the native cast, but this is not recommended
+ for production use.
+- **U (Unsupported)**: Comet does not provide a native version of this cast, so it runs Spark's own code-generated
+ cast inside the Comet pipeline.
- **N/A**: Spark does not support this cast.
-## ANSI Mode Fallback
+Incompatible and unsupported casts fall back to Spark only when Comet's codegen dispatcher cannot handle them (for
+example, casts involving `VariantType`) or when `spark.comet.exec.scalaUDF.codegen.enabled=false`.
-Cast will fall back to Spark in some cases when ANSI mode is enabled. This can be enabled by setting `spark.comet.expression.Cast.allowIncompatible=true`. See the [Comet Supported Expressions Guide](../../../expressions.md) for more information on this configuration setting.
+## ANSI Mode Support
+
+Enabling ANSI mode does not by itself make a cast fall back to Spark or require
+`spark.comet.expression.Cast.allowIncompatible=true`: every cast that Spark permits in ANSI mode has the same support
+level as in legacy mode. See the [ANSI Mode](#ansi-mode) table below for per-pair support, and the
+[Comet Supported Expressions Guide](../../../expressions.md) for more information on the `allowIncompatible` setting.
## Whitespace Trimming in Casts from String
@@ -143,20 +149,12 @@ Comet's native `CAST(date AS STRING)` is compatible with Spark. Years below 1000
zero-padded to four digits (e.g. year 999 renders as `0999-01-01`). Years above 9999 are
rendered without truncation. The cast is timezone-independent.
-## String to TimestampNTZ
-
-Comet's native `CAST(string AS TIMESTAMP_NTZ)` implementation matches Apache Spark's behavior.
-Unlike `CAST(string AS TIMESTAMP)`, this cast is timezone-independent: any timezone offset in
-the input string (e.g. `+08:00`, `Z`, `UTC`) is silently discarded, and the local date-time
-components are preserved as-is. Time-only strings (e.g. `T12:34:56`, `12:34`) produce `NULL`.
-The result is always a wall-clock timestamp with no timezone conversion or DST adjustment.
-
## Decimal with Negative Scale to String
Casting a `DecimalType` with a negative scale to `StringType` is marked as incompatible when
`spark.sql.legacy.allowNegativeScaleOfDecimal` is `false` (the default). When that config is
-disabled, Spark cannot create negative-scale decimals, so Comet falls back to avoid running
-native execution on unexpected inputs.
+disabled, Spark cannot create negative-scale decimals, so by default Comet runs Spark's
+code-generated cast instead of the native one.
When `spark.sql.legacy.allowNegativeScaleOfDecimal=true`, the cast is compatible. Comet matches
Spark's behavior of using Java `BigDecimal.toString()` semantics, which produces scientific
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-4.1/map.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-4.1/map.md.txt
index 85ce8af197..98a501a42a 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-4.1/map.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-4.1/map.md.txt
@@ -26,7 +26,7 @@ keys, in `try_element_at`, and in other contexts where map ordering must be dete
runs `MapSort` natively, so map shuffle and group-by-on-map stay on Comet under Spark 4.0.
When `spark.comet.exec.strictFloatingPoint=true`, `MapSort` falls back to Spark for maps whose
-keys contain `Float` or `Double` (consistent with `SortOrder` and `SortArray`). Arrow's sort uses
+keys contain `Float` or `Double`. Arrow's sort uses
IEEE total ordering for floating-point, which differs from Spark's `Double.compare` semantics for
`NaN` and `-0.0`.
diff --git a/_sources/user-guide/latest/compatibility/index.md.txt b/_sources/user-guide/latest/compatibility/index.md.txt
index b1e3216161..36cd8cf2b8 100644
--- a/_sources/user-guide/latest/compatibility/index.md.txt
+++ b/_sources/user-guide/latest/compatibility/index.md.txt
@@ -68,7 +68,8 @@ which document their per-expression configs and the specific differences to expe
This is distinct from expressions that have **no** codegen-dispatch path: there, the
incompatible cases fall back to Spark by default, and `allowIncompatible=true` runs the native
-(incompatible) path instead. `cast` is the main example; see the
+(incompatible) path instead. Aggregate functions such as `mode` are the main example, because the
+codegen dispatcher covers only scalar expressions; see the
[expression reference](../expressions.md) for which expressions have incompatible cases.
## Strings with non-UTF-8 bytes
@@ -111,12 +112,10 @@ Code that catches `SparkException` and only asserts on message substrings is una
inspects the exception class, `getCondition()`, or the parameterised error class will observe
divergence:
-- Byte / Short `Add`, `Subtract`, and `Multiply` overflow raises `ARITHMETIC_OVERFLOW` where Spark
- 4.1 raises `BINARY_ARITHMETIC_OVERFLOW`, and Long overflow surfaces as `"integer overflow"`
- rather than `"long overflow"`. `Abs` uses Rust type names (`Int8`, `Int64`, ...) in the message
- instead of Spark's SQL type names, and the scalar path of `UnaryMinus` on Byte / Short emits a
- malformed message. The `try_` suggestion is omitted from all of these
- ([#5071](https://github.com/apache/datafusion-comet/issues/5071)).
+- Byte / Short `Add`, `Subtract`, and `Multiply` overflow raises `ARITHMETIC_OVERFLOW` (for
+ example `byte overflow`) where Spark raises `BINARY_ARITHMETIC_OVERFLOW`, and integral
+ `ARITHMETIC_OVERFLOW` messages omit Spark's `try_` suggestion
+ ([#6217](https://github.com/apache/datafusion-comet/issues/6217)).
- Wide-decimal arithmetic overflow, decimal divide-by-zero, and decimal-to-decimal cast overflow
raise raw Arrow errors that bypass `SparkErrorConverter` and surface as `CometNativeException`
rather than `SparkArithmeticException` with the proper error class and query context
@@ -130,18 +129,11 @@ The following native paths silently return values that differ from Spark for edg
Most also have entries in the per-category expression pages linked above; they are collected here
so users hunting an unexpected value have a single place to check:
-- `CAST(boolean AS DECIMAL(p, s))` where `10^s` exceeds the target precision (e.g.
- `DECIMAL(1, 1)`) throws `NUMERIC_VALUE_OUT_OF_RANGE` regardless of the eval mode. Spark returns
- `NULL` under legacy and try mode, and only throws under ANSI
- ([#5068](https://github.com/apache/datafusion-comet/issues/5068)).
- `CAST(string AS timestamp)` and `CAST(string AS timestamp_ntz)` trim Unicode whitespace.
Spark trims only the bytes `0x00`-`0x20` and `0x7F`, so a value padded with an ASCII control byte
parses in Spark and returns `NULL` in Comet, while a value padded with non-ASCII whitespace such
as `U+3000` returns `NULL` in Spark and parses in Comet
([#5149](https://github.com/apache/datafusion-comet/issues/5149)).
-- **Explicit positive timestamp years:** Spark accepts strings such as `+7528` as the start
- of that year, while Comet's native string-to-timestamp cast returns NULL in non-ANSI mode
- ([#5716](https://github.com/apache/datafusion-comet/issues/5716)).
- Native `RANGE` window frames with an explicit `PRECEDING` / `FOLLOWING` offset diverge from
Spark when the boundary arithmetic overflows for `DATE` or `DECIMAL` `ORDER BY` columns
([#5022](https://github.com/apache/datafusion-comet/issues/5022)).
@@ -156,13 +148,3 @@ so users hunting an unexpected value have a single place to check:
`spark.sql.codegen.fallback` or when the generated code exceeds
`spark.sql.codegen.hugeMethodLimit`, where an intermediate overflow that later cancels out
returns `NULL` (or raises under ANSI) in Spark but the recovered value in Comet.
-
-## Object store cache
-
-When Comet's native scan reads Parquet files, it caches one object store instance per
-`(scheme + host + port, hadoop-config-hash)` key. For `abfss://container@account.dfs.core.windows.net/...`
-URLs, the container lives in the URL userinfo, not the host, so two containers in one storage
-account currently collide on the same cache entry. Within a single executor process, reading from
-a second container after a first can be served by the first container's store instance and return
-its data. S3, GCS, and HDFS are unaffected because their bucket / host lives in the URL host
-component. Tracked by [#4993](https://github.com/apache/datafusion-comet/issues/4993).
diff --git a/_sources/user-guide/latest/compatibility/operators.md.txt b/_sources/user-guide/latest/compatibility/operators.md.txt
index 1e1103955b..cd779dc081 100644
--- a/_sources/user-guide/latest/compatibility/operators.md.txt
+++ b/_sources/user-guide/latest/compatibility/operators.md.txt
@@ -32,6 +32,18 @@ operator restrictions and aggregate buffer compatibility checks still apply.
Parquet writes whose input plans contain an empty relation use Spark's writer to preserve
readable empty output files and their schema metadata.
+## In-Memory Cache
+
+Comet can store cached relations (`df.cache()`, `CACHE TABLE`) in Arrow format and scan them
+natively. This is experimental and disabled by default; see [In-Memory Cache](../in-memory-cache.md)
+for how to enable it. Comet does not replace a `spark.sql.cache.serializer` that the application
+has already set. Relations whose schema Comet's Arrow writer does not support are cached in
+Spark's default format, and their scans fall back to Spark. Reads that feed Spark operators rather
+than Comet operators can be slower than Spark's cache.
+
+With Kryo and `spark.kryo.registrationRequired=true`, Comet needs its Kryo registrator whether or
+not the cache is enabled; see [Kryo serialization](../installation.md#kryo-serialization).
+
## Sampling
Comet runs `SampleExec` natively when sampling is performed without replacement, which covers
@@ -95,8 +107,11 @@ runs natively; it is controlled by `spark.comet.exec.windowGroupLimit.enabled` (
**Known incompatibilities:**
-- Signed-zero ordering (`-0.0` vs `+0.0`) diverges from Spark's `RankLimitIterator`; see
- [floating-point ordering](./floating-point.md#ordering-signed-zero-00-vs-00).
+- Floating-point values nested in array or struct `ORDER BY` keys are compared with Arrow's raw
+ total ordering, so ranks can differ from Spark when the data mixes `-0.0` and `+0.0` or more
+ than one NaN representation ([#5507](https://github.com/apache/datafusion-comet/issues/5507)).
+ Scalar `FLOAT` and `DOUBLE` keys are normalized and match Spark; see
+ [floating-point ordering](./floating-point.md).
## Round-Robin Partitioning
diff --git a/_sources/user-guide/latest/compatibility/scans.md.txt b/_sources/user-guide/latest/compatibility/scans.md.txt
index d479e75ffd..5736312533 100644
--- a/_sources/user-guide/latest/compatibility/scans.md.txt
+++ b/_sources/user-guide/latest/compatibility/scans.md.txt
@@ -48,6 +48,8 @@ The following features are not supported and cause Comet to fall back to Spark:
does not replicate. By default Comet falls back to Spark in this case. Set
`spark.comet.scan.allowDisabledParquetVectorizedReader=true` to opt in to running the
Comet Parquet scan regardless.
+- A read schema that repeats a Parquet field id, at the top level or within a struct, when
+ `spark.sql.parquet.fieldId.read.enabled=true`.
The following limitation may produce incorrect results without falling back to Spark:
diff --git a/_sources/user-guide/latest/compatibility/spark-versions.md.txt b/_sources/user-guide/latest/compatibility/spark-versions.md.txt
index 9992dc4f1f..516d08b899 100644
--- a/_sources/user-guide/latest/compatibility/spark-versions.md.txt
+++ b/_sources/user-guide/latest/compatibility/spark-versions.md.txt
@@ -33,21 +33,17 @@ Spark 3.4.3 is supported with Java 17 and Scala 2.12/2.13.
```{warning}
Spark 3.4 support is deprecated as of the 1.0.0 release and will be removed in a future release.
Comet continues to build and publish Spark 3.4 binaries in the meantime, but Apache Spark's own
-SQL test suite is no longer run against Spark 3.4 on every change: it runs only when a contributor
+SQL test suite no longer runs against Spark 3.4 automatically: it runs only when a contributor
opts a pull request into it. Regressions specific to Spark 3.4 are therefore more likely to reach a
release than on the other supported versions. We recommend moving to Spark 3.5 or later.
```
### Known Limitations
-- **Reading `TimestampLTZ` as `TimestampNTZ`**: Spark 3.4 raises an error for this operation
- (SPARK-36182), but Comet's Parquet scan silently returns the raw UTC value instead.
- See [Parquet Compatibility](scans.md#parquet-scan-limitations) for details.
-
-- **Unsupported Parquet type conversions**: Spark 3.4 raises schema incompatibility errors for
- certain type mismatches (e.g., reading INT32 as BIGINT, decimal precision changes), but Comet's
- Comet's Parquet scan may not detect these and could return unexpected values.
- See [Parquet Compatibility](scans.md#parquet-scan-limitations) for details.
+- **Extra `SparkException` layer in Parquet schema mismatch errors**: when a Parquet read is
+ rejected because a file's type cannot be converted to the requested type, the error's cause chain
+ has one more `SparkException` layer than Spark's own reader produces.
+ See [Parquet Compatibility](scans.md#schema-mismatch-handling) for details.
## Spark 3.5
@@ -55,18 +51,14 @@ Spark 3.5.9 is supported with Java 17 and Scala 2.12/2.13.
### Known Limitations
-- **Reading `TimestampLTZ` as `TimestampNTZ`**: Spark 3.5 raises an error for this operation
- (SPARK-36182), but Comet's Parquet scan silently returns the raw UTC value instead.
- See [Parquet Compatibility](scans.md#parquet-scan-limitations) for details.
-
-- **Unsupported Parquet type conversions**: Spark 3.5 raises schema incompatibility errors for
- certain type mismatches (e.g., reading INT32 as BIGINT, decimal precision changes), but Comet's
- Comet's Parquet scan may not detect these and could return unexpected values.
- See [Parquet Compatibility](scans.md#parquet-scan-limitations) for details.
+- **Extra `SparkException` layer in Parquet schema mismatch errors**: when a Parquet read is
+ rejected because a file's type cannot be converted to the requested type, the error's cause chain
+ has one more `SparkException` layer than Spark's own reader produces.
+ See [Parquet Compatibility](scans.md#schema-mismatch-handling) for details.
## Spark 4.0
-Spark 4.0.4 is supported with Java 17 and Scala 2.13.
+Spark 4.0.4 is supported with Java 17/21 and Scala 2.13.
### Known Limitations
@@ -93,6 +85,7 @@ Spark 4.1.3 is supported with Java 17/21 and Scala 2.13.
Spark 4.2.0 is provided as experimental support with Java 17 and Scala 2.13.
```{warning}
-Spark 4.2 support is experimental. Comet tests run in CI for this version, but the Spark SQL tests
-do not yet. It is intended for early evaluation only and should not be used in production.
+Spark 4.2 support is experimental. Comet's own test suites run nightly for this version, but
+Apache Spark's SQL test suite does not run against it yet. It is intended for early evaluation only
+and should not be used in production.
```
diff --git a/_sources/user-guide/latest/datasources.md.txt b/_sources/user-guide/latest/datasources.md.txt
index a2964517e9..2f454cba4e 100644
--- a/_sources/user-guide/latest/datasources.md.txt
+++ b/_sources/user-guide/latest/datasources.md.txt
@@ -29,7 +29,8 @@ Arrow format, allowing the Comet pipeline to take over after that, but the proce
### Apache Iceberg
-Comet accelerates Iceberg scans of Parquet files. See the [Iceberg Guide] for more information.
+Comet accelerates Iceberg scans of Parquet files and has an experimental, opt-in native Iceberg writer.
+See the [Iceberg Guide] and [Iceberg Writes](iceberg-writes.md) for more information.
[Iceberg Guide]: iceberg.md
@@ -37,7 +38,8 @@ Comet accelerates Iceberg scans of Parquet files. See the [Iceberg Guide] for mo
Comet provides experimental Rust-based CSV scan support. When `spark.comet.scan.csv.v2.enabled` is enabled, CSV files
are read in Rust for improved performance. This feature is experimental and performance benefits are
-workload-dependent.
+workload-dependent. Only Spark's DataSource V2 CSV scan is accelerated, and Spark reads CSV through the V1 API by
+default, so also remove `csv` from `spark.sql.sources.useV1SourceList`.
Alternatively, when `spark.comet.convert.csv.enabled` is enabled, data from Spark's CSV reader is immediately
converted into Arrow format, allowing the Comet pipeline to take over after that.
@@ -308,6 +310,10 @@ credential providers and options documented above also apply to alias-scheme URL
translation feeds the native Iceberg scan; see
[Object store configuration (S3)](iceberg.md#object-store-configuration-s3) in the Iceberg guide.
+A native Parquet scan whose alias-scheme paths span more than one bucket falls back to Spark. Alias
+schemes apply to native scans only: a native Iceberg write to an alias-scheme location falls back to
+iceberg-java.
+
### Examples
The following examples demonstrate how to configure S3 access using different authentication methods.
@@ -344,7 +350,7 @@ Comet's S3 support has the following limitations:
1. **Partial Hadoop S3A configuration support**: Not all Hadoop S3A configurations are currently supported. Only the configurations listed in the tables above are translated and applied to the underlying `object_store` crate.
-2. **Custom credential providers**: Custom implementations of AWS credential providers are not supported. The implementation only supports the standard credential providers listed in the table above. We are planning to add support for custom credential providers through a JNI-based adapter that will allow calling Java credential providers from Rust code. See [#1829](https://github.com/apache/datafusion-comet/issues/1829) for more details.
+2. **Custom credential providers**: Custom credential provider classes named in `fs.s3a.aws.credentials.provider` are not supported; only the standard providers listed in the table above are. To route credential requests through your own Java code, implement Comet's `CometS3CredentialProvider` SPI; see [S3 Credential Providers](s3-credential-providers.md). Broader Hadoop S3A integration is tracked in [#1829](https://github.com/apache/datafusion-comet/issues/1829).
## Azure
diff --git a/_sources/user-guide/latest/datatypes.md.txt b/_sources/user-guide/latest/datatypes.md.txt
index 5d3e6e736c..1b619e67ca 100644
--- a/_sources/user-guide/latest/datatypes.md.txt
+++ b/_sources/user-guide/latest/datatypes.md.txt
@@ -71,12 +71,12 @@ the tables below and may be reconsidered based on demand:
## Datetime
-| Type | Status | Notes |
-| ------------------ | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `DateType` | ✅ | |
-| `TimestampType` | ✅ | |
-| `TimestampNTZType` | ✅ | |
-| `TimeType` | ⚠️ | Spark 4.1+. Native serialization is in place; some operators (sort, shuffle, min/max) are still being wired up ([#4288](https://github.com/apache/datafusion-comet/issues/4288)). |
+| Type | Status | Notes |
+| ------------------ | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| `DateType` | ✅ | |
+| `TimestampType` | ✅ | |
+| `TimestampNTZType` | ✅ | |
+| `TimeType` | ⚠️ | Spark 4.1+. Native serialization is in place; some operators (sort, min/max) are still being wired up ([#4288](https://github.com/apache/datafusion-comet/issues/4288)). |
## Interval
@@ -97,7 +97,7 @@ functions. Remaining work is tracked by
| Type | Status | Notes |
| ------------ | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `StructType` | ✅ | Empty structs (no fields) fall back. |
+| `StructType` | ✅ | Empty structs (no fields) and structs with duplicate field names fall back. |
| `ArrayType` | ✅ | |
| `MapType` | ✅ | Hash aggregate group keys cannot contain a `MapType` (transitively): Arrow's row format used by DataFusion's grouped hash aggregate does not support `Map`, so such groupings fall back. |
@@ -113,8 +113,9 @@ Direct projection requires explicit configuration on every supported Spark versi
Parquet timestamp inference settings. Support for Spark's whole-value pushdown rewrite is tracked
by [#5519](https://github.com/apache/datafusion-comet/issues/5519). Nested Variant columns, pushed-down
Variant field extraction, expressions, writes, shuffle and spill, Python operators, encrypted
-files, and Iceberg scans fall back to Spark. Spark also handles columnar-to-row conversion of
-the native scan output and strict reads with `allowReadingShredded=false`. Broader
+files, and Iceberg scans that read a Variant column fall back to Spark. Iceberg scans of tables
+whose Variant columns the query does not read run natively. Spark also handles columnar-to-row
+conversion of the native scan output and strict reads with `allowReadingShredded=false`. Broader
support is tracked by [#4295](https://github.com/apache/datafusion-comet/issues/4295) and
[#3983](https://github.com/apache/datafusion-comet/issues/3983).
diff --git a/_sources/user-guide/latest/expressions.md.txt b/_sources/user-guide/latest/expressions.md.txt
index 85ae0cc509..c60948d5df 100644
--- a/_sources/user-guide/latest/expressions.md.txt
+++ b/_sources/user-guide/latest/expressions.md.txt
@@ -30,9 +30,9 @@ Some ✅ Supported expressions have specific incompatible cases that are not run
Those cases must be opted into per expression with
`spark.comet.expression.EXPRNAME.allowIncompatible=true` (where `EXPRNAME` is the Spark
expression class name, for example `Cast`). There is no global opt-in. By default such a case
-either falls back to Spark (for example `cast`) or, when the expression has a Spark-compatible
-codegen-dispatch implementation, runs through that instead (for example the regex and JSON
-families). See [Native and codegen-dispatch implementations](compatibility/index.md#native-and-codegen-dispatch-implementations)
+either falls back to Spark (for example the `mode` aggregate) or, when the expression has a
+Spark-compatible codegen-dispatch implementation, runs through that instead (for example `cast`
+and the regex and JSON families). See [Native and codegen-dispatch implementations](compatibility/index.md#native-and-codegen-dispatch-implementations)
for how Comet chooses.
Most expressions can also be disabled with `spark.comet.expression.EXPRNAME.enabled=false`, where
@@ -62,18 +62,18 @@ The Implementation column is auto-generated from the serde definitions in `Query
## Not currently planned
Comet focuses acceleration on mainstream relational, string, datetime, math, and collection
-expressions. The following function families are **not currently planned** for native acceleration (they are not on the 1.0 roadmap): specialized functionality with narrow real-world analytics use and high implementation cost. They fall back to Spark and may be reconsidered based on demand:
+expressions. The following function families are **not currently planned** for native acceleration (they are not on the current roadmap): specialized functionality with narrow real-world analytics use and high implementation cost. They fall back to Spark and may be reconsidered based on demand:
- **Probabilistic sketches and approximate top-k** (`kll_sketch_*`, `hll_*`, `theta_*`, `count_min_sketch`, `bitmap_*`, `approx_top_k*`): specialized data structures with exact-correctness traps.
- **Geospatial** (`st_*`): brand-new Spark 4.1 functionality, specialized.
- **Avro / Protobuf codecs** (`from_avro`, `to_avro`, `from_protobuf`, `to_protobuf`, `schema_of_avro`): format conversion belongs at the IO layer, not expression evaluation.
-- **JVM reflection** (`java_method`, `reflect`): niche, and they invoke arbitrary JVM methods (a security concern).
+- **JVM reflection** (`java_method`, `reflect`, `try_reflect`): niche, and they invoke arbitrary JVM methods (a security concern).
- **UTF-8 validation** (`is_valid_utf8`, `make_valid_utf8`, `validate_utf8`, `try_validate_utf8`): niche Spark 4.x string-validation helpers.
- **Miscellaneous niche** (`histogram_numeric`, `version`, `sentences`, `quote`): low-value or specialized functions with little benefit from native acceleration.
The file-metadata functions `input_file_name`, `input_file_block_start`, and `input_file_block_length` depend on scan-internal per-row file information rather than the expression layer; their support status is covered in the [scan compatibility guide](compatibility/scans.md).
-Note that `median` and `mode` are planned: they are mainstream exact aggregates. `approx_count_distinct` is supported because Comet ports Spark's `HyperLogLogPlusPlus` exactly, so its result is bit-identical to Spark.
+Note that `median` and `mode` are supported: they are mainstream exact aggregates. `mode` runs natively only with `spark.comet.expression.Mode.allowIncompatible=true`. `approx_count_distinct` is supported because Comet ports Spark's `HyperLogLogPlusPlus` exactly, so its result is bit-identical to Spark.
The tables below list every Spark built-in expression with its current status.
@@ -114,8 +114,9 @@ The tables below list every Spark built-in expression with its current status.
| `median` | ✅ | — | Rewrites to `percentile(col, 0.5)` and runs natively for supported percentile inputs |
| `min` | ✅ | Native | |
| `min_by` | ✅ | Native | Value and ordering must be fixed-length types |
-| `mode` | ✅ | — | `mode(col)` only; Spark breaks ties non-deterministically, so Comet returns the smallest tied value and falls back by default, opt-in via allowIncompatible ([#3970](https://github.com/apache/datafusion-comet/issues/3970)) |
+| `mode` | ✅ | — | `mode(col)` only; Spark breaks ties non-deterministically, so Comet returns the smallest tied value and falls back by default, opt-in via allowIncompatible |
| `percentile` | ✅ | Native | Single literal percentage on numeric input runs natively; array of percentages and a frequency argument fall back to Spark |
+| `percentile_approx` | ✅ | Native | Alias of `approx_percentile`; same restrictions apply |
| `percentile_cont` | ✅ | — | Spark 4.0+ `WITHIN GROUP (ORDER BY ...)`; ascending only runs natively, `DESC` falls back to Spark |
| `percentile_disc` | 🔜 | — | Percentile aggregate |
| `regr_avgx` | ✅ | — | Native: Spark rewrites to `Average` (tests in [#4551](https://github.com/apache/datafusion-comet/pull/4551)) |
@@ -150,14 +151,14 @@ The tables below list every Spark built-in expression with its current status.
| `array` | ✅ | Native | |
| `array_append` | ✅ | Native | |
| `array_compact` | ✅ | — | |
-| `array_contains` | ✅ | Hybrid | NaN/signed-zero handling may differ ([details](compatibility/floating-point.md)) |
+| `array_contains` | ✅ | Hybrid | Float/double element arrays route through the JVM codegen dispatcher by default; the native path is opt-in via allowIncompatible |
| `array_distinct` | ✅ | Native | NaN/signed-zero handling may differ ([details](compatibility/floating-point.md)) |
| `array_except` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default; the incompatible native path is opt-in via allowIncompatible ([details](compatibility/expressions/array.md)) |
| `array_insert` | ✅ | Native | |
| `array_intersect` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default; the incompatible native path is opt-in via allowIncompatible ([details](compatibility/expressions/array.md)) |
| `array_join` | ✅ | Hybrid | Native for literal or column delimiter and null replacement; other cases and non-UTF8_BINARY collations use the JVM codegen dispatcher ([details](compatibility/expressions/array.md)) |
| `array_max` | ✅ | Native | NaN ordering may differ ([details](compatibility/floating-point.md)) |
-| `array_min` | ✅ | Native | NaN ordering may differ ([details](compatibility/floating-point.md)) |
+| `array_min` | ✅ | Native | May return `-0.0` where Spark returns `0.0` when both zeros are present |
| `array_position` | ✅ | Native | Binary/struct/map/null elements fall back |
| `array_prepend` | ✅ | — | |
| `array_remove` | ✅ | Native | |
@@ -171,7 +172,7 @@ The tables below list every Spark built-in expression with its current status.
| `sequence` | ✅ | Hybrid | Integral types run natively; date/timestamp sequences use codegen dispatch |
| `shuffle` | ✅ | Native | Binary/struct/map elements fall back |
| `slice` | ✅ | Native | Native ([#4149](https://github.com/apache/datafusion-comet/pull/4149)) |
-| `sort_array` | ✅ | Hybrid | Nested struct/null arrays fall back |
+| `sort_array` | ✅ | Hybrid | Struct, nested-array, and null elements run natively; other element types (for example intervals), and floating-point elements when `spark.comet.exec.strictFloatingPoint=true`, route through the JVM codegen dispatcher |
---
@@ -200,8 +201,8 @@ The tables below list every Spark built-in expression with its current status.
| --- | --- | --- | --- |
| `array_size` | ✅ | — | |
| `cardinality` | ✅ | Native | |
-| `concat` | ✅ | Hybrid | Binary/array children fall back |
-| `reverse` | ✅ | Hybrid | Binary-element arrays fall back (Incompatible) ([details](compatibility/expressions/array.md)) |
+| `concat` | ✅ | Hybrid | Binary/array children and non-UTF8_BINARY collations route through the JVM codegen dispatcher |
+| `reverse` | ✅ | Hybrid | Arrays with binary, struct, or map elements, and collated strings, route through the JVM codegen dispatcher ([details](compatibility/expressions/array.md)) |
| `size` | ✅ | Native | |
---
@@ -229,7 +230,7 @@ The type-name conversion functions (`bigint`, `binary`, `boolean`, `date`, `deci
| Function | Status | Implementation | Notes |
| --- | --- | --- | --- |
-| `cast` | ✅ | Hybrid | Some casts fall back; float-to-decimal is opt-in ([details](compatibility/expressions/cast.md)) |
+| `cast` | ✅ | Hybrid | Casts without a native path (for example boolean to decimal) route through the JVM codegen dispatcher ([details](compatibility/expressions/cast.md)) |
---
@@ -278,7 +279,7 @@ The type-name conversion functions (`bigint`, `binary`, `boolean`, `date`, `deci
| `make_date` | ✅ | Native | |
| `make_dt_interval` | ✅ | Codegen dispatch | |
| `make_interval` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default; intervals outside Arrow's nanosecond range are tracked by [#5279](https://github.com/apache/datafusion-comet/issues/5279); the native path is opt-in via allowIncompatible ([details](compatibility/expressions/datetime.md)) |
-| `make_time` | 🔜 | — | Spark 4.1 TIME type; tracked by [#4288](https://github.com/apache/datafusion-comet/issues/4288) |
+| `make_time` | ✅ | — | Spark 4.1+; requires `spark.sql.timeType.enabled=true`, which Spark leaves off by default. Runs natively; remaining TIME type work is tracked by [#4288](https://github.com/apache/datafusion-comet/issues/4288) |
| `make_timestamp` | ✅ | — | |
| `make_timestamp_ltz` | ✅ | — | 2-arg TIME form falls back |
| `make_timestamp_ntz` | ✅ | — | 2-arg TIME form falls back |
@@ -301,7 +302,7 @@ The type-name conversion functions (`bigint`, `binary`, `boolean`, `date`, `deci
| `timestampadd` | ✅ | — | Reached through the grammar rather than the function registry; runs through codegen dispatch |
| `timestampdiff` | ✅ | — | Reached through the grammar rather than the function registry; runs through codegen dispatch |
| `to_date` | ✅ | — | Rewrites to `Cast` (or `Cast(GetTimestamp)` with a format) before Comet sees the plan |
-| `to_time` | 🔜 | — | Spark 4.1 TIME type; tracked by [#4288](https://github.com/apache/datafusion-comet/issues/4288) |
+| `to_time` | ✅ | — | Spark 4.1+; requires `spark.sql.timeType.enabled=true`, which Spark leaves off by default. The one-argument form runs natively |
| `to_timestamp` | ✅ | — | Rewrites to `Cast` (or `GetTimestamp` with a format) before Comet sees the plan |
| `to_timestamp_ltz` | ✅ | — | Rewrites to `to_timestamp` (`TimestampType`) |
| `to_timestamp_ntz` | ✅ | — | Rewrites to `to_timestamp` (`TimestampNTZType`) |
@@ -310,8 +311,10 @@ The type-name conversion functions (`bigint`, `binary`, `boolean`, `date`, `deci
| `trunc` | ✅ | Hybrid | |
| `try_make_interval` | ✅ | — | Rewrites to `MakeInterval`; same support as `make_interval` (Spark 4.0+) |
| `try_make_timestamp` | ✅ | — | |
+| `try_make_timestamp_ltz` | ✅ | — | Same support as `try_make_timestamp` (Spark 4.0+) |
+| `try_make_timestamp_ntz` | ✅ | — | Same support as `try_make_timestamp` (Spark 4.0+) |
| `try_to_date` | ✅ | — | Rewrites to `Cast`/`GetTimestamp` before Comet sees the plan; same support as `to_date` |
-| `try_to_time` | 🔜 | — | Spark 4.1 TIME type; tracked by [#4288](https://github.com/apache/datafusion-comet/issues/4288) |
+| `try_to_time` | ✅ | — | Same support as `to_time` |
| `try_to_timestamp` | ✅ | — | Rewrites to `Cast`/`GetTimestamp` before Comet sees the plan; same support as `to_timestamp` |
| `unix_date` | ✅ | Native | |
| `unix_micros` | ✅ | Codegen dispatch | |
@@ -329,7 +332,8 @@ The type-name conversion functions (`bigint`, `binary`, `boolean`, `date`, `deci
## generator_funcs
`explode`, `explode_outer`, `posexplode`, and `posexplode_outer` are supported via
-`CometExplodeExec` (operator-level, not expression-level). Enabled by default via
+`CometExplodeExec` (operator-level, not expression-level) for array input; map input falls back
+to Spark ([#2837](https://github.com/apache/datafusion-comet/issues/2837)). Enabled by default via
`spark.comet.exec.explode.enabled`.
| Function | Status | Implementation | Notes |
@@ -362,13 +366,13 @@ The type-name conversion functions (`bigint`, `binary`, `boolean`, `date`, `deci
| Function | Status | Implementation | Notes |
| --- | --- | --- | --- |
-| `from_json` | ✅ | Hybrid | Falls back by default; opt-in via allowIncompatible ([audit](../../contributor-guide/expression-audits/json_funcs.md#from_json)) |
-| `get_json_object` | ✅ | Hybrid | Some inputs need allowIncompatible ([audit](../../contributor-guide/expression-audits/json_funcs.md#get_json_object)) |
-| `json_array_length` | ✅ | Hybrid | Single-quoted/trailing JSON needs allowIncompatible ([audit](../../contributor-guide/expression-audits/json_funcs.md#json_array_length)) |
+| `from_json` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default; the native path (supported schemas only) is opt-in via allowIncompatible ([audit](../../contributor-guide/expression-audits/json_funcs.md#from_json)) |
+| `get_json_object` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default; the native path is opt-in via allowIncompatible ([audit](../../contributor-guide/expression-audits/json_funcs.md#get_json_object)) |
+| `json_array_length` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default; the native path, which differs for single-quoted JSON, unescaped control characters, and trailing content, is opt-in via allowIncompatible ([audit](../../contributor-guide/expression-audits/json_funcs.md#json_array_length)) |
| `json_object_keys` | ✅ | Codegen dispatch | |
| `json_tuple` | 🔜 | — | [#3160](https://github.com/apache/datafusion-comet/issues/3160) |
| `schema_of_json` | ✅ | Codegen dispatch | |
-| `to_json` | ✅ | Hybrid | Options and map/array inputs fall back ([audit](../../contributor-guide/expression-audits/json_funcs.md#to_json)) |
+| `to_json` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default, including options and map/array inputs; the native path is opt-in via allowIncompatible ([audit](../../contributor-guide/expression-audits/json_funcs.md#to_json)) |
---
@@ -401,7 +405,7 @@ The type-name conversion functions (`bigint`, `binary`, `boolean`, `date`, `deci
| `map_contains_key` | ✅ | — | |
| `map_entries` | ✅ | Native | |
| `map_from_arrays` | ✅ | Native | |
-| `map_from_entries` | ✅ | Hybrid | BinaryType key/value falls back (Incompatible) ([details](compatibility/expressions/map.md)) |
+| `map_from_entries` | ✅ | Hybrid | BinaryType keys/values and `spark.sql.mapKeyDedupPolicy=LAST_WIN` route through the JVM codegen dispatcher ([details](compatibility/expressions/map.md)) |
| `map_keys` | ✅ | Native | |
| `map_values` | ✅ | Native | |
| `str_to_map` | ✅ | Hybrid | |
@@ -462,7 +466,7 @@ The type-name conversion functions (`bigint`, `binary`, `boolean`, `date`, `deci
| `radians` | ✅ | Native | |
| `rand` | ✅ | Native | |
| `randn` | ✅ | Native | |
-| `random` | ✅ | Native | Alias for `rand` (Spark 4.0+); seed must be a literal |
+| `random` | ✅ | Native | Alias for `rand`; seed must be a literal |
| `randstr` | ✅ | Native | Random string (Spark 4.0+); length and seed must be literals |
| `rint` | ✅ | Native | |
| `round` | ✅ | Hybrid | Float/double inputs route through the JVM codegen dispatcher; other types run natively |
@@ -541,9 +545,9 @@ The type-name conversion functions (`bigint`, `binary`, `boolean`, `date`, `deci
| `like` | ✅ | Hybrid | |
| `not` | ✅ | Native | |
| `or` | ✅ | Native | |
-| `regexp` | ✅ | Hybrid | In-subset literals run natively; others fall back by default ([details](compatibility/regex.md)) |
-| `regexp_like` | ✅ | Hybrid | In-subset literals run natively; others fall back by default ([details](compatibility/regex.md)) |
-| `rlike` | ✅ | Hybrid | In-subset literals run natively; others fall back by default ([details](compatibility/regex.md)) |
+| `regexp` | ✅ | Hybrid | In-subset literal patterns run natively; other patterns route through the JVM codegen dispatcher ([details](compatibility/regex.md)) |
+| `regexp_like` | ✅ | Hybrid | In-subset literal patterns run natively; other patterns route through the JVM codegen dispatcher ([details](compatibility/regex.md)) |
+| `rlike` | ✅ | Hybrid | In-subset literal patterns run natively; other patterns route through the JVM codegen dispatcher ([details](compatibility/regex.md)) |
---
@@ -587,12 +591,12 @@ The type-name conversion functions (`bigint`, `binary`, `boolean`, `date`, `deci
| `overlay` | ✅ | Codegen dispatch | |
| `position` | ✅ | Codegen dispatch | |
| `printf` | ✅ | Codegen dispatch | |
-| `regexp_count` | ✅ | — | Runs natively (rewrites to `size(regexp_extract_all(...))`) |
+| `regexp_count` | ✅ | — | Rewrites to `size(regexp_extract_all(...))`; `regexp_extract_all` routes through the JVM codegen dispatcher by default |
| `regexp_extract` | ✅ | Native | |
| `regexp_extract_all` | ✅ | Native | |
| `regexp_instr` | ✅ | Codegen dispatch | Routed through the JVM codegen dispatcher |
| `regexp_replace` | ✅ | Hybrid | |
-| `regexp_substr` | ✅ | — | Runs natively (rewrites to `nullif(regexp_extract(...), '')`) |
+| `regexp_substr` | ✅ | — | Rewrites to `nullif(regexp_extract(...), '')`; `regexp_extract` routes through the JVM codegen dispatcher by default |
| `repeat` | ✅ | Native | |
| `replace` | ✅ | Hybrid | |
| `right` | ✅ | Native | |
@@ -606,16 +610,16 @@ The type-name conversion functions (`bigint`, `binary`, `boolean`, `date`, `deci
| `substr` | ✅ | Native | |
| `substring` | ✅ | Native | |
| `substring_index` | ✅ | Native | |
-| `to_binary` | ✅ | — | Hex and base64 forms accelerated natively; the `utf-8` form rewrites to `encode` and runs via codegen dispatch on Spark 4.0+ |
+| `to_binary` | ✅ | — | The hex form runs natively; the base64 and `utf-8` forms route through the JVM codegen dispatcher |
| `to_char` | ✅ | — | |
| `to_number` | ✅ | Codegen dispatch | |
| `to_varchar` | ✅ | — | |
| `translate` | ✅ | Hybrid | Codegen dispatch by default: DataFusion's `translate` iterates over Unicode graphemes (Spark uses code points) and substitutes U+0000 instead of treating it as a deletion sentinel, so the native path is opt-in via allowIncompatible |
| `trim` | ✅ | Native | |
-| `try_to_binary` | ✅ | — | Runs natively (rewrites to `try_eval(to_binary(...))`) |
+| `try_to_binary` | ✅ | — | Rewrites to `try_eval(to_binary(...))`, which routes through the JVM codegen dispatcher |
| `try_to_number` | ✅ | Codegen dispatch | Routed through the JVM codegen dispatcher |
| `ucase` | ✅ | Hybrid | |
-| `unbase64` | ✅ | Hybrid | |
+| `unbase64` | ✅ | Hybrid | Column or literal input runs natively; other child expressions, and the strict form used by `to_binary(str, 'base64')`, route through the JVM codegen dispatcher |
| `upper` | ✅ | Hybrid | |
---
@@ -634,6 +638,7 @@ The type-name conversion functions (`bigint`, `binary`, `boolean`, `date`, `deci
| Function | Status | Implementation | Notes |
| --- | --- | --- | --- |
| `parse_url` | ✅ | Native | |
+| `try_parse_url` | ✅ | — | Spark 4.0+; runs natively and returns NULL for an invalid URL |
| `try_url_decode` | ✅ | — | |
| `url_decode` | ✅ | — | |
| `url_encode` | ✅ | — | |
@@ -694,6 +699,8 @@ Comet also accelerates a number of Catalyst expressions that have no Spark SQL f
- **Accessor expressions (subscript and field access, not functions):** struct field access (`col.field`), array element access (`arr[i]`), and map value access (`map[key]`).
- **Internal decimal arithmetic:** `CheckOverflow`, `MakeDecimal`, and `UnscaledValue`, which the analyzer inserts around decimal operations.
- **User-defined functions:** Scala UDFs registered through the DataFrame or SQL API.
+- **DataSource V2 catalog functions:** Iceberg's system functions `bucket`, `truncate`, `years`, `months`, `days`, and `hours` (for example `system.bucket(16, id)`) run natively; see [Iceberg system functions](iceberg.md#iceberg-system-functions).
+- **Lowered built-ins:** Spark lowers some built-in functions to `StaticInvoke` or `Invoke` calls. Those without a native mapping run through the JVM codegen dispatcher when their input and output types are supported.
- **Structural expressions:** aliases, attribute references, literals, sort orders, and `CASE WHEN`.
This list is illustrative, not exhaustive: the per-function tables are not the complete set of expressions Comet can accelerate.
diff --git a/_sources/user-guide/latest/iceberg-writes.md.txt b/_sources/user-guide/latest/iceberg-writes.md.txt
index 74712b7314..c66ba42264 100644
--- a/_sources/user-guide/latest/iceberg-writes.md.txt
+++ b/_sources/user-guide/latest/iceberg-writes.md.txt
@@ -44,7 +44,7 @@ and it is the foundation for the second toggle: when
`spark.comet.iceberg.write.enabled=true` and the write passes the eligibility check below, the
`IcebergWrite` operator's per-task Parquet write is delegated to
[iceberg-rust](https://github.com/apache/iceberg-rust) via Comet's native execution pipeline
-([#5308](https://github.com/apache/datafusion-comet/issues/5308)).
+([#5361](https://github.com/apache/datafusion-comet/pull/5361)).
## How the native write works
@@ -127,7 +127,7 @@ trade-off, only no plan change.
## Native Parquet write eligibility
When `spark.comet.iceberg.write.enabled=true`
-([#5308](https://github.com/apache/datafusion-comet/issues/5308)), the `IcebergWrite` operator's
+([#5361](https://github.com/apache/datafusion-comet/pull/5361)), the `IcebergWrite` operator's
per-task Parquet write is delegated to [iceberg-rust](https://github.com/apache/iceberg-rust).
The native writer must produce the same outcome as iceberg-java — the same Parquet features,
statistics, and manifest metadata — so a write is only eligible when every table property it
@@ -136,7 +136,9 @@ feeding the write is fully Comet-native. For a partitioned table that plan inclu
distribution and local sort Iceberg requests on its partition transforms; those stay native
because the transforms themselves have native implementations (see
[Iceberg system functions](iceberg.md)). Ineligible writes run through iceberg-java unchanged,
-with the reason reported as a fall-back reason in Comet's extended EXPLAIN output.
+with the reason reported as a fall-back reason in Comet's extended EXPLAIN output. A write that
+runs natively shows `CometIcebergWrite` under `IcebergCommit` in the physical plan; an ineligible
+write keeps `IcebergWrite`.
The native writer reads its input as Arrow batches from a Comet operator, so the write's input
must itself run in Comet. A write whose input is a local relation, such as `INSERT ... VALUES` or
@@ -177,7 +179,8 @@ Within the namespaces that shape data-file bytes — `write.parquet.*` and `parq
everything not listed above must be absent: unvetted `write.parquet.*` keys (e.g.
`bloom-filter-max-bytes`, `stats-enabled.column.*`, keys added by future Iceberg versions),
any `parquet.*` table property (including `parquet.enable.dictionary`), and any `parquet.*`
-key in the session Hadoop configuration (with `HadoopFileIO`-backed output those reach
+key in the session Hadoop configuration other than the reader-only
+`parquet.hadoop.vectored.io.enabled` (with `HadoopFileIO`-backed output those reach
iceberg-java's writer but not the native one). Also gated explicitly: any `encryption.*` key,
`write.object-storage.enabled=true`, `write.location-provider.impl`, and `io-impl`.
diff --git a/_sources/user-guide/latest/iceberg.md.txt b/_sources/user-guide/latest/iceberg.md.txt
index 8fc076ab85..0c9ae8f108 100644
--- a/_sources/user-guide/latest/iceberg.md.txt
+++ b/_sources/user-guide/latest/iceberg.md.txt
@@ -26,7 +26,7 @@ then serialized to Comet's native execution engine (see
[PR #2528](https://github.com/apache/datafusion-comet/pull/2528)).
The example below uses Spark's package downloader to retrieve Comet 1.1.0-SNAPSHOT and Iceberg
-1.8.1, but Comet has been tested with Iceberg 1.5, 1.7, 1.8, 1.9, 1.10, and 1.11. The native Iceberg
+1.8.1, but Comet has been tested with Iceberg 1.5, 1.8, 1.9, 1.10, and 1.11. The native Iceberg
reader is enabled by default. To disable it, set `spark.comet.scan.icebergNative.enabled=false`.
The example uses the Spark 3.5 / Scala 2.12 build of Comet; substitute the Comet artifact
@@ -97,7 +97,7 @@ The native Iceberg reader supports the following features:
- Equality and comparison predicates (`=`, `!=`, `>`, `>=`, `<`, `<=`)
- Logical operators (`AND`, `OR`)
- NULL checks (`IS NULL`, `IS NOT NULL`) on primitive columns
-- `IN` and `NOT IN` list operations
+- `IN` list operations (`NOT IN` is applied after the scan)
- `BETWEEN` operations
NULL checks on struct, array, and map columns still use native scans and return correct
@@ -156,6 +156,8 @@ scala> spark.sql("SELECT * FROM rest_cat.db.test_table").show()
The native reader has its own Rust object store client and does not go through Iceberg's JVM FileIO, neither `S3FileIO` nor the older Hadoop S3A filesystem. It configures that client from the catalog's `s3.*` properties (the same keys `S3FileIO` reads), from `spark.hadoop.fs.s3a.*` settings, or, for a scheme opted into `spark.hadoop.fs.comet.s3Compliant.schemes`, from vendor-style `fs.<scheme>.<authority>.*` keys (see [S3-Compliant Filesystem Schemes](datasources.md#s3-compliant-filesyst [...]
+Per-bucket `fs.s3a.bucket.<bucket>.*` settings apply to the bucket that holds the table's data and delete files. The native reader uses one object-store configuration per scan, so a scan whose data or delete files span more than one S3 bucket falls back to Spark.
+
For a custom S3-compatible endpoint, configure the catalog with the endpoint, path-style access, region, and credentials (Hive shown):
```shell
@@ -170,7 +172,7 @@ For a custom S3-compatible endpoint, configure the catalog with the endpoint, pa
--conf spark.sql.catalog.s3_cat.s3.secret-access-key=...
```
-These `s3.*` storage properties are not specific to the Hive catalog shown here. When `s3.access-key-id` / `s3.secret-access-key` are omitted, credentials come from the standard AWS chain (environment variables, instance profiles, and so on). `client.region` is auto-detected for AWS but should be set for non-AWS endpoints. If your REST catalog vends temporary credentials, the native reader does not consume them automatically, and wiring that requires the credential provider bridge. See I [...]
+These `s3.*` storage properties are not specific to the Hive catalog shown here. When `s3.access-key-id` / `s3.secret-access-key` are omitted, credentials come from the standard AWS chain (environment variables, instance profiles, and so on). The region is not auto-detected: when neither the catalog (`client.region` or `s3.region`) nor the executor environment (`AWS_REGION` or `AWS_DEFAULT_REGION`) supplies one, Comet uses `us-east-1`, so set it for AWS buckets in any other region. If yo [...]
### Current limitations
@@ -178,10 +180,14 @@ The following scenarios will fall back to the JVM Iceberg reader:
- Iceberg table spec v4 or newer
- v3 tables with columns that declare an initial default value
-- v3 column types the native reader cannot read (`variant`, `geometry`, `geography`, `unknown`)
+- v3 column types the native reader cannot read (`geometry`, `geography`, `unknown`), and
+ `variant` columns the query reads (on Spark 4.0+, a table whose `variant` columns are not
+ projected is read natively)
- Encrypted tables with 192-bit data keys (no AES-192-GCM in the underlying crypto)
- Delete files in a format other than Parquet or Puffin (Avro or ORC positional/equality deletes)
- Tables backed by Avro or ORC data files (only Parquet is accelerated)
+- Scans whose data or delete files span more than one S3 bucket (the native reader uses one
+ object-store configuration per scan)
- Tables partitioned on `BINARY` or `DECIMAL` (with precision >28) columns
- Scans with residual filters using `truncate`, `bucket`, `year`, `month`, `day`, or `hour`
transform functions (partition pruning still works, but row-level filtering of these
@@ -245,6 +251,6 @@ the expression fall back to Spark.
### Task input metrics
-The native Iceberg reader populates Spark's task-level `inputMetrics.bytesRead` (visible in the Spark UI Stages tab) using the `bytes_read` counter from iceberg-rust's `ScanMetrics`. This counter includes bytes read from both data files and delete files.
+The native Iceberg reader populates Spark's task-level `inputMetrics.bytesRead` (visible in the Spark UI Stages tab) using the `bytes_read` counter from iceberg-rust's `ScanMetrics`. This counter includes bytes read from both data files and delete files. The scan's SQL metrics, including Iceberg's planning counters, are listed in the [Metrics Guide](metrics.md#cometicebergnativescan).
Iceberg Java does not explicitly report `bytesRead` to Spark's task input metrics. On the iceberg Java path, any `bytesRead` value comes from Hadoop's filesystem-level I/O counters, not from Iceberg itself. Because Comet's native reader and the Hadoop filesystem use different counting mechanisms, the exact byte counts will differ between the two paths.
diff --git a/_sources/user-guide/latest/in-memory-cache.md.txt b/_sources/user-guide/latest/in-memory-cache.md.txt
index c37fa2dc32..8e39f00fe5 100644
--- a/_sources/user-guide/latest/in-memory-cache.md.txt
+++ b/_sources/user-guide/latest/in-memory-cache.md.txt
@@ -159,7 +159,7 @@ Spark serializes a cached batch with `spark.serializer` whenever the block leave
`_SER` storage levels, replication, cross-executor fetches, and the disk half of the default
`MEMORY_AND_DISK`. So an ordinary `df.cache()` that spills is enough to reach it.
-If you run with `spark.kryo.registrationRequired=true`, register Comet's classes:
+If you run Kryo with `spark.kryo.registrationRequired=true`, register Comet's classes:
```
spark.serializer=org.apache.spark.serializer.KryoSerializer
@@ -171,7 +171,8 @@ Comet cannot set `spark.kryo.registrator` for you the way it sets `spark.sql.cac
`KryoSerializer` reads it when `SparkEnv` builds the serializer, which happens before any plugin
runs. Without it, caching fails with a "Class is not registered" error that does not name this
feature. Comet's driver plugin warns at startup when it sees Kryo, `registrationRequired`, and no
-registrator.
+registrator. Native broadcast needs the same registrator even when the cache is disabled; see
+[Kryo serialization](installation.md#kryo-serialization).
## Limitations
diff --git a/_sources/user-guide/latest/installation.md.txt b/_sources/user-guide/latest/installation.md.txt
index 978a6532bc..b1fd977dcf 100644
--- a/_sources/user-guide/latest/installation.md.txt
+++ b/_sources/user-guide/latest/installation.md.txt
@@ -49,29 +49,31 @@ Comet requires JDK 17 or later. JDK 11 is no longer supported as of the 1.1.0 re
```{warning}
Spark 3.4 support is deprecated as of the 1.0.0 release and will be removed in a future release.
-Apache Spark's own SQL test suite is no longer run against Spark 3.4 on every change; it runs only
-on demand. We recommend moving to Spark 3.5 or later.
+Apache Spark's own SQL test suite no longer runs against Spark 3.4 automatically; it runs only on
+demand. We recommend moving to Spark 3.5 or later.
```
| Spark Version | Java Version | Scala Version | Comet Tests in CI | Spark SQL Tests in CI |
| ------------- | ------------ | ------------- | ----------------- | --------------------- |
-| 3.4.3 | 17 | 2.12/2.13 | Yes | On demand |
-| 3.5.9 | 17 | 2.12/2.13 | Yes | Yes |
-| 4.0.4 | 17/21 | 2.13 | Yes | Yes |
-| 4.1.3 | 17/21 | 2.13 | Yes | Yes |
+| 3.4.3 | 17 | 2.12/2.13 | Nightly | On demand |
+| 3.5.9 | 17 | 2.12/2.13 | Nightly | Nightly |
+| 4.0.4 | 17/21 | 2.13 | Nightly | Nightly |
+| 4.1.3 | 17/21 | 2.13 | Before merge | Before merge |
Note that we do not test the full matrix of supported Java and Scala versions in CI for every Spark version.
-"On demand" in the table above means the suite is not run automatically before a change is merged.
-A contributor can still run it against an individual pull request, but Spark 3.4 is no longer
-covered by default.
+"Before merge" in the table above means the suite must pass before a change is merged. "Nightly" means
+the suite runs once a day against the `main` branch, so a regression it finds is caught after the change
+has been merged rather than before. "On demand" means the suite does not run automatically at all. A
+contributor can still run it against an individual pull request, but Spark 3.4 is no longer covered by
+default.
Experimental support is provided for the following versions of Apache Spark and is intended for development/testing
use only and should not be used in production yet.
| Spark Version | Java Version | Scala Version | Comet Tests in CI | Spark SQL Tests in CI |
| ------------- | ------------ | ------------- | ----------------- | --------------------- |
-| 4.2.0 | 17 | 2.13 | Yes | No |
+| 4.2.0 | 17 | 2.13 | Nightly | No |
Note that Comet may not fully work with proprietary forks of Apache Spark such as the Spark versions offered by
Cloud Service Providers.
@@ -236,3 +238,20 @@ Some cluster managers may require additional configuration, see <https://spark.a
In addition to Apache Spark memory configuration parameters, Comet introduces additional parameters to configure memory
allocation for native execution. See [Comet Memory Tuning](./tuning.md) for details.
+
+### Kryo serialization
+
+If the application uses Kryo (`spark.serializer=org.apache.spark.serializer.KryoSerializer`) with
+`spark.kryo.registrationRequired=true`, also register Comet's classes with Kryo:
+
+```shell
+--conf spark.kryo.registrator=org.apache.comet.CometKryoRegistrator
+```
+
+Without it, any query that uses Comet's native broadcast exchange, which is enabled by default,
+fails with Kryo's "Class is not registered" error, for example on the first broadcast hash join.
+The [in-memory cache](in-memory-cache.md#kryo) needs the same registrator. Set it before the
+`SparkContext` is created: `KryoSerializer` reads it before Comet's plugin runs, so Comet cannot
+add it for you. `spark.kryo.registrator` accepts a comma-separated list, so an application with
+its own registrator can list both. Comet logs a warning at startup when Kryo requires registration
+and this registrator is missing.
diff --git a/_sources/user-guide/latest/kubernetes.md.txt b/_sources/user-guide/latest/kubernetes.md.txt
index 0a82984559..1ef680e0ad 100644
--- a/_sources/user-guide/latest/kubernetes.md.txt
+++ b/_sources/user-guide/latest/kubernetes.md.txt
@@ -21,13 +21,16 @@
## Comet Docker Images
-Run the following command from the root of this repository to build the Comet Docker image, or use a [published
-Docker image](https://hub.docker.com/r/apache/datafusion-comet).
+Run the following command from the root of this repository to build the Comet Docker image. The image is based on
+`apache/spark:4.1.3` and adds the Comet JAR for Spark 4.1 and Scala 2.13 to `$SPARK_HOME/jars`.
```shell
docker build -t apache/datafusion-comet -f kube/Dockerfile .
```
+Push the image to a registry that your cluster can pull from, and use that image name in the `image` field of the
+examples below.
+
## Example Spark Submit
The exact syntax will vary depending on the Kubernetes distribution, but an example `spark-submit` command can be
@@ -69,14 +72,17 @@ metadata:
spec:
type: Scala
mode: cluster
- image: apache/datafusion-comet:1.1.0-SNAPSHOT-spark3.5.5-scala2.12-java11
+ image: apache/datafusion-comet
imagePullPolicy: IfNotPresent
mainClass: org.apache.spark.examples.SparkPi
mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.13-4.1.3.jar
sparkConf:
- "spark.executor.extraClassPath": "/opt/spark/jars/comet-spark-spark3.5_2.12-1.1.0-SNAPSHOT.jar"
- "spark.driver.extraClassPath": "/opt/spark/jars/comet-spark-spark3.5_2.12-1.1.0-SNAPSHOT.jar"
+ "spark.executor.extraClassPath": "/opt/spark/jars/comet-spark-spark4.1_2.13-1.1.0-SNAPSHOT.jar"
+ "spark.driver.extraClassPath": "/opt/spark/jars/comet-spark-spark4.1_2.13-1.1.0-SNAPSHOT.jar"
"spark.plugins": "org.apache.spark.CometPlugin"
+ "spark.memory.offHeap.enabled": "true"
+ "spark.memory.offHeap.size": "1g"
+ "spark.executor.memoryOverhead": "1g"
"spark.comet.enabled": "true"
"spark.comet.exec.enabled": "true"
"spark.comet.shuffle.enabled": "true"
diff --git a/_sources/user-guide/latest/metrics.md.txt b/_sources/user-guide/latest/metrics.md.txt
index 9e0aeeed46..5b5735954c 100644
--- a/_sources/user-guide/latest/metrics.md.txt
+++ b/_sources/user-guide/latest/metrics.md.txt
@@ -23,11 +23,14 @@ under the License.
Comet operators report the following metrics in the Spark SQL UI.
-### CometScanExec
+### CometBatchScan
-| Metric | Description |
-| ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `scan time` | Total time to scan a Parquet file. This is not comparable to the same metric in Spark because Comet's scan metric is more accurate. Although both Comet and Spark measure the time in nanoseconds, Spark rounds this time to the nearest millisecond per batch and Comet does not. |
+| Metric | Description |
+| ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `scan time` | Time spent reading batches from the wrapped DataSource V2 reader. Comet measures this time in nanoseconds and does not round it to the nearest millisecond per batch. |
+
+Parquet scans through the DataSource V1 API appear in plans as `CometNativeScan` and report the
+native metrics described under [Native Parquet scans](#native-parquet-scans) rather than `scan time`.
### CometIcebergNativeScan
@@ -36,12 +39,12 @@ during execution; the planning metrics are Iceberg's own scan-report counters, c
Iceberg's Java planner on the driver and surfaced here so they show in the UI as they do for a
plain Spark + Iceberg `BatchScan`.
-| Metric | Description |
-| --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
-| `number of output rows` | Rows produced by the scan. |
-| `number of bytes scanned` | Bytes read from storage, including data and delete files. |
-| `number of file splits processed` | File scan tasks (splits) read by this scan. |
-| `scan time` | Time spent in the native scan's record-batch polling, covering the iceberg-rust reader plus Comet's schema adaptation. It excludes time the stream spends waiting between polls, so it is decode/compute time, not end-to-end scan latency. This differs from the `scan time` under `CometScanExec`, which times Parquet file reads. |
+| Metric | Description |
+| --------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `number of output rows` | Rows produced by the scan. |
+| `number of bytes scanned` | Bytes read from storage, including data and delete files. |
+| `number of file splits processed` | File scan tasks (splits) read by this scan. |
+| `scan time` | Time spent in the native scan's record-batch polling, covering the iceberg-rust reader plus Comet's schema adaptation. It excludes time the stream spends waiting between polls, so it is decode/compute time, not end-to-end scan latency. This differs from the `scan time` under `CometBatchScan`, which times batch reads from a DataSource V2 reader. |
The planning metrics below mirror Iceberg's `ScanReport`. They are driver-side values known after
scan planning and do not change during execution.
@@ -69,6 +72,21 @@ Iceberg's `numDeletes` (deletes applied at read time) is not reported: it is a J
counter, and Comet reads natively through iceberg-rust, which exposes no deletes-applied count, so
the value would always be 0.
+### CometHashAggregate
+
+Native aggregates with grouping keys report these additional metrics:
+
+| Metric | Description |
+| ------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------- |
+| `rows bypassing partial aggregation` | Input rows passed through without partial aggregation. See [Adaptive Partial Aggregation](tuning.md#adaptive-partial-aggregation). |
+| `number of spills` | Number of times the aggregate spilled to disk. |
+| `total spilled bytes` | Bytes written to aggregate spill files. |
+| `number of spilled rows` | Rows written to aggregate spill files. |
+| `peak native aggregate memory` | Peak memory used by the native aggregate. |
+
+Spill bytes from native sorts, aggregates, and sort-merge joins are also added to Spark's task-level
+`diskBytesSpilled` metric in every stage, not only in shuffle stages.
+
### Hash Joins
With `spark.comet.exec.join.dynamicFilter.enabled=true`, native broadcast and shuffled hash joins
@@ -93,15 +111,20 @@ reader savings. Existing join, scan, and intervening filter metrics retain their
Comet adds some additional metrics:
-| Metric | Description |
-| ------------------------------- | --------------------------------------------------------------------------- |
-| `native shuffle time` | Total time in native code excluding any child operators. |
-| `repartition time` | Time to repartition batches. |
-| `partition interleaving time` | Time to interleave partitioned batches before writing them. |
-| `memory pool time` | Time interacting with memory pool. |
-| `encoding and compression time` | Time to encode batches in IPC format and compress using ZSTD. |
-| `disk spilled bytes` | Actual bytes written to native shuffle spill files on disk. |
-| `memory spilled bytes` | Uncompressed Arrow backing-buffer and partition-index data before spilling. |
+| Metric | Description |
+| --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
+| `native shuffle writer time` | Total time in the native shuffle writer, excluding any child operators. |
+| `repartition time` | Time to repartition batches. |
+| `partition interleaving time` | Time to interleave partitioned batches before writing them. |
+| `encoding and compression time` | Time to encode batches in Arrow IPC format and compress them with the configured codec (`spark.comet.shuffle.compression.codec`, default `lz4`). |
+| `decoding and decompression time` | Time to decompress and decode shuffle blocks when they are read. |
+| `number of spills` | Number of native shuffle spills. |
+| `disk spilled bytes` | Actual bytes written to native shuffle spill files on disk. |
+| `memory spilled bytes` | Uncompressed Arrow backing-buffer and partition-index data before spilling. |
+| `number of input batches` | Batches received by the native shuffle writer. |
+
+Comet exchanges also report Spark's standard shuffle read and write metrics, including when native
+operators read shuffle blocks directly.
Disk and memory spilled bytes measure different representations of the same native shuffle spill.
Disk spill bytes count the actual bytes written to disk: compressed when shuffle compression is
@@ -127,25 +150,24 @@ Here is a guide to some of the native metrics.
### ScanExec
-| Metric | Description |
-| ----------------- | --------------------------------------------------------------------------------------------------- |
-| `elapsed_compute` | Total time spent in this operator, fetching batches from a JVM iterator. |
-| `jvm_fetch_time` | Time spent in the JVM fetching input batches to be read by this `ScanExec` instance. |
-| `arrow_ffi_time` | Time spent using Arrow FFI to create Arrow batches from the memory addresses returned from the JVM. |
+| Metric | Description |
+| ----------------- | ------------------------------------------------------------------------ |
+| `elapsed_compute` | Total time spent in this operator, fetching batches from a JVM iterator. |
+| `cast_time` | Time spent casting columns to the requested data types during the scan. |
### ShuffleWriterExec
-| Metric | Description |
-| ---------------------- | --------------------------------------------------------------------- |
-| `elapsed_compute` | Total time excluding any child operators. |
-| `repart_time` | Time to repartition batches. |
-| `interleave_time` | Time to gather partitioned rows into output batches before writing. |
-| `ipc_time` | Time to encode batches in IPC format and compress using ZSTD. |
-| `mempool_time` | Time interacting with memory pool. |
-| `write_time` | Time spent writing bytes to disk. |
-| `spill_count` | Number of native shuffle spills. |
-| `spilled_bytes` | Actual bytes written to native shuffle spill files on disk. |
-| `memory_spilled_bytes` | Uncompressed Arrow backing-buffer and partition-index memory spilled. |
+| Metric | Description |
+| ---------------------- | --------------------------------------------------------------------------------------- |
+| `elapsed_compute` | Total time excluding any child operators. |
+| `repart_time` | Time to repartition batches. |
+| `interleave_time` | Time to gather partitioned rows into output batches before writing. |
+| `encode_time` | Time to encode batches in Arrow IPC format and compress them with the configured codec. |
+| `write_time` | Time spent writing encoded data to its destination. |
+| `input_batches` | Number of input batches. |
+| `spill_count` | Number of native shuffle spills. |
+| `spilled_bytes` | Actual bytes written to native shuffle spill files on disk. |
+| `memory_spilled_bytes` | Uncompressed Arrow backing-buffer and partition-index memory spilled. |
### Native Parquet scans
diff --git a/_sources/user-guide/latest/migration-guide.md.txt b/_sources/user-guide/latest/migration-guide.md.txt
index f16ad82606..1de5f88467 100644
--- a/_sources/user-guide/latest/migration-guide.md.txt
+++ b/_sources/user-guide/latest/migration-guide.md.txt
@@ -53,6 +53,13 @@ Treat setting one of these keys as a temporary measure. If you find you cannot s
legacy behavior, please open an issue describing your use case so it can be considered before the
key is removed.
+## Upgrading to Comet 1.1.0
+
+Comet `1.1.0` makes no behavior changes that need a `spark.comet.legacy.*` key.
+
+Comet `1.1.0` requires JDK 17 or later. JDK 11 is no longer supported. See
+[Installing Comet](installation.md) for the supported Java, Scala, and Spark versions.
+
## Upgrading to Comet 1.0.0
Comet `1.0.0` is the first release under the stable
diff --git a/_sources/user-guide/latest/operators.md.txt b/_sources/user-guide/latest/operators.md.txt
index 2182e7de58..899865d4ee 100644
--- a/_sources/user-guide/latest/operators.md.txt
+++ b/_sources/user-guide/latest/operators.md.txt
@@ -49,13 +49,13 @@ omitted from the tables below and may be reconsidered based on demand:
## Scans
-| Operator | Status | Notes |
-| ----------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
-| `FileSourceScanExec` | ✅ | Parquet only. Some types and configurations fall back. See [Parquet Scan Compatibility](compatibility/scans.md). |
-| `BatchScanExec` | ✅ | Parquet, Apache Iceberg Parquet, and CSV (native) scans. See [Parquet Scan Compatibility](compatibility/scans.md) and the [Iceberg Guide](iceberg.md). |
-| `LocalTableScanExec` | ⚠️ | Disabled by default; there is no acceleration advantage and this operator is typically only used in test code. Can be opted into via config ([#4393](https://github.com/apache/datafusion-comet/pull/4393)). |
-| `EmptyRelationExec` | ✅ | Spark 4.0 and later. See [Empty Relations](compatibility/operators.md#empty-relations) for native-input support and writer fallback. |
-| `InMemoryTableScanExec` | 🔜 | Cached / in-memory table scans fall back today. |
+| Operator | Status | Notes |
+| ----------------------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `FileSourceScanExec` | ✅ | Parquet only. Some types and configurations fall back. See [Parquet Scan Compatibility](compatibility/scans.md). |
+| `BatchScanExec` | ✅ | Apache Iceberg Parquet scans run natively. Native CSV scans are experimental and disabled by default. DataSource V2 Parquet scans are not accelerated. See [Parquet Scan Compatibility](compatibility/scans.md) and the [Iceberg Guide](iceberg.md). |
+| `LocalTableScanExec` | ⚠️ | Disabled by default; there is no acceleration advantage and this operator is typically only used in test code. Can be opted into via config ([#4393](https://github.com/apache/datafusion-comet/pull/4393)). |
+| `EmptyRelationExec` | ✅ | Spark 4.0 and later. See [Empty Relations](compatibility/operators.md#empty-relations) for native-input support and writer fallback. |
+| `InMemoryTableScanExec` | ⚠️ | Experimental, disabled by default. Set `spark.comet.exec.inMemoryCache.enabled=true` before the application starts so Comet installs its Arrow cache serializer. Relations with unsupported column types stay in Spark's cache format and fall back. See [In-Memory Cache](in-memory-cache.md). |
## Projection and filtering
@@ -77,11 +77,11 @@ omitted from the tables below and may be reconsidered based on demand:
## Aggregation
-| Operator | Status | Notes |
-| ------------------------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `HashAggregateExec` | ✅ | |
-| `ObjectHashAggregateExec` | ✅ | Supports a limited set of aggregates, such as `bloom_filter_agg`. Falls back when Comet shuffle is disabled, which would otherwise split the aggregate across Comet and Spark. See the [Tuning Guide](tuning.md). |
-| `SortAggregateExec` | 🔜 | Falls back today; Comet currently accelerates hash aggregates. |
+| Operator | Status | Notes |
+| ------------------------- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `HashAggregateExec` | ✅ | |
+| `ObjectHashAggregateExec` | ✅ | Runs the object-buffer aggregates Comet supports, such as `collect_list`, `collect_set`, `percentile`, `approx_percentile`, `mode`, `bloom_filter_agg`, and (Spark 4.0+) `listagg`. Falls back when Comet shuffle is disabled, which would otherwise split the aggregate across Comet and Spark. See [Shuffle](tuning.md#shuffle). |
+| `SortAggregateExec` | 🔜 | Falls back today; Comet currently accelerates hash aggregates. |
## Joins
@@ -89,7 +89,7 @@ omitted from the tables below and may be reconsidered based on demand:
| ----------------------------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `BroadcastHashJoinExec` | ✅ | |
| `ShuffledHashJoinExec` | ✅ | |
-| `SortMergeJoinExec` | ✅ | |
+| `SortMergeJoinExec` | ✅ | Supports `BINARY` join keys. Nested-type (struct, array, map) and collated-string join keys fall back to Spark. |
| `BroadcastNestedLoopJoinExec` | ✅ | Falls back to Spark when the preserved side is broadcast (for example LEFT OUTER with BROADCAST on the left) ([#4429](https://github.com/apache/datafusion-comet/pull/4429)). |
## Exchanges
@@ -104,7 +104,7 @@ omitted from the tables below and may be reconsidered based on demand:
| Operator | Status | Notes |
| ---------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `WindowExec` | ⚠️ | Runs natively and is enabled by default. A broad set of window functions is accelerated; unsupported shapes fall back to Spark. See [window function compatibility](compatibility/operators.md). |
-| `WindowGroupLimitExec` | ✅ | Streaming per-partition top-K pushdown for `ROW_NUMBER`, `RANK`, and `DENSE_RANK`. |
+| `WindowGroupLimitExec` | ✅ | Spark 3.5 and later. Streaming per-partition top-K pushdown for `ROW_NUMBER`, `RANK`, and `DENSE_RANK`. |
## Generators and set operations
@@ -117,16 +117,18 @@ omitted from the tables below and may be reconsidered based on demand:
## Writes
-| Operator | Status | Notes |
-| ------------------------ | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `WriteFilesExec` | ⚠️ | Spark 4.0+. Experimental native Parquet writes, disabled by default (opt-in). Non-partitioned, non-bucketed writes only, and not when `spark.sql.files.maxRecordsPerFile` is set. |
-| `DataWritingCommandExec` | ⚠️ | Spark 3.4/3.5 only. Experimental native Parquet writes, disabled by default (opt-in). Replaced by `WriteFilesExec` on Spark 4.0+ and removed with Spark 3.x support. |
+| Operator | Status | Notes |
+| -------------------------------------------------------------------------------------------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `WriteFilesExec` | ⚠️ | Spark 4.0+. Experimental native Parquet writes, disabled by default (opt-in). Non-partitioned, non-bucketed writes only, and not when `spark.sql.files.maxRecordsPerFile` is set. |
+| `DataWritingCommandExec` | ⚠️ | Spark 3.4/3.5 only. Experimental native Parquet writes, disabled by default (opt-in). Replaced by `WriteFilesExec` on Spark 4.0+ and removed with Spark 3.x support. |
+| `AppendDataExec`, `OverwriteByExpressionExec`, `OverwritePartitionsDynamicExec`, `ReplaceDataExec` | ⚠️ | Apache Iceberg tables only. Experimental, disabled by default. See [Iceberg Writes](iceberg-writes.md). |
## Python and UDF
-| Operator | Status | Notes |
-| --------------------------------------------------------------------------------------- | ------ | ---------------------------------------------------------------------------------------------------------------------------- |
-| `ArrowEvalPythonExec`, `MapInArrowExec`, `MapInPandasExec`, `FlatMapGroupsInPandasExec` | 🔜 | Experimental accelerated PyArrow UDF support is in progress ([#4234](https://github.com/apache/datafusion-comet/pull/4234)). |
+| Operator | Status | Notes |
+| -------------------------------------------------- | ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `MapInArrowExec`, `MapInPandasExec` | ⚠️ | Spark 4.0 and later. Experimental, disabled by default (`spark.comet.exec.pyarrowUDF.enabled`). See [PyArrow UDF Acceleration](pyarrow-udfs.md). |
+| `ArrowEvalPythonExec`, `FlatMapGroupsInPandasExec` | 🔜 | Scalar `@pandas_udf` ([#5386](https://github.com/apache/datafusion-comet/issues/5386)) and grouped `applyInPandas` ([#5123](https://github.com/apache/datafusion-comet/issues/5123)) fall back to Spark. |
## See also
diff --git a/_sources/user-guide/latest/pyarrow-udfs.md.txt b/_sources/user-guide/latest/pyarrow-udfs.md.txt
index 5aab9a61b4..eb704deb6f 100644
--- a/_sources/user-guide/latest/pyarrow-udfs.md.txt
+++ b/_sources/user-guide/latest/pyarrow-udfs.md.txt
@@ -30,7 +30,7 @@ using the Arrow IPC format.
Without Comet, the execution path for these UDFs involves unnecessary data conversions:
-1. Comet reads data in Arrow columnar format (via CometScan)
+1. Comet reads data in Arrow columnar format (via `CometNativeScan`)
2. Spark inserts a ColumnarToRow transition (converts Arrow to UnsafeRow)
3. The Python runner converts those rows back to Arrow to send to Python
4. Python executes the UDF on Arrow batches
@@ -40,8 +40,8 @@ Steps 2 and 3 are redundant since the data starts and ends in Arrow format.
## How Comet Optimizes This
-When enabled, Comet detects `PythonMapInArrowExec` / `MapInArrowExec` and `MapInPandasExec`
-operators in the physical plan and replaces them with `CometMapInBatchExec`, which:
+When enabled, Comet detects `MapInArrowExec` and `MapInPandasExec` operators in the physical plan
+and replaces them with `CometMapInBatchExec` (shown as `CometMapInBatch` in plans), which:
- Reads Arrow columnar batches directly from the upstream Comet operator
- Feeds them to the Python runner without the expensive UnsafeProjection copy
@@ -58,18 +58,16 @@ copies that remain.
Without Comet's optimization:
```
-PythonMapInArrow / MapInArrow / MapInPandas
-+- ColumnarToRow <- Arrow -> Row copy
- +- CometNativeExec <- Arrow batch
- +- CometScan
+MapInArrow / MapInPandas
++- CometColumnarToRow <- Arrow -> Row copy
+ +- CometNativeScan <- Arrow batch
```
With the optimization enabled:
```
CometMapInBatch <- Arrow batch in/out, Python runner attached
-+- CometNativeExec
- +- CometScan
++- CometNativeScan
```
## Configuration
@@ -95,7 +93,7 @@ worker. Both confs can be set independently.
| PySpark API | Spark Plan Node | Supported |
| -------------------------------- | --------------------------- | --------- |
-| `df.mapInArrow(func, schema)` | `PythonMapInArrowExec` | Yes |
+| `df.mapInArrow(func, schema)` | `MapInArrowExec` | Yes |
| `df.mapInPandas(func, schema)` | `MapInPandasExec` | Yes |
| `@pandas_udf` (scalar) | `ArrowEvalPythonExec` | Not yet |
| `df.applyInPandas(func, schema)` | `FlatMapGroupsInPandasExec` | Not yet |
@@ -144,17 +142,15 @@ You should see:
```
CometMapInBatch ...
-+- CometNativeExec ...
- +- CometScan ...
++- CometNativeScan parquet ...
```
Instead of the unoptimized plan:
```
-PythonMapInArrow ...
-+- ColumnarToRow
- +- CometNativeExec ...
- +- CometScan ...
+MapInArrow ...
++- CometColumnarToRow
+ +- CometNativeScan parquet ...
```
When AQE is enabled (the Spark default) and the query contains a shuffle, the
@@ -163,7 +159,7 @@ running an action will show the unoptimized plan:
```
AdaptiveSparkPlan isFinalPlan=false
-+- PythonMapInArrow ...
++- MapInArrow ...
+- CometExchange ...
```
diff --git a/_sources/user-guide/latest/s3-credential-providers.md.txt b/_sources/user-guide/latest/s3-credential-providers.md.txt
index df3d76005b..e97e22c4eb 100644
--- a/_sources/user-guide/latest/s3-credential-providers.md.txt
+++ b/_sources/user-guide/latest/s3-credential-providers.md.txt
@@ -19,7 +19,7 @@ under the License.
# S3 Credential Providers
-Comet's native S3 readers normally fetch credentials from the standard AWS credential chain (static keys, instance profiles, environment variables, etc.). Some clusters use a vendor-managed mechanism instead, where credentials are issued per request based on a JWT or per S3 path. For those clusters, Comet supports loading a vendor-supplied bridge class that routes every native credential request through the vendor's Java code.
+Comet's native S3 readers and native Iceberg writer normally fetch credentials from the standard AWS credential chain (static keys, instance profiles, environment variables, etc.). Some clusters use a vendor-managed mechanism instead, where credentials are issued per request based on a JWT or per S3 path. For those clusters, Comet supports loading a vendor-supplied bridge class that routes every native credential request through the vendor's Java code.
## Do I need this?
@@ -102,7 +102,7 @@ Without the config set, no credential-related log lines appear at startup; nativ
## Iceberg: explicit S3 region required
-With the bridge configured, Comet wires a custom credential loader into `iceberg-storage-opendal`. `opendal`'s built-in S3 region auto-detection only runs when no custom loader is configured, so on the bridge path the region (and endpoint for non-AWS) must be set explicitly on the Spark catalog:
+`iceberg-storage-opendal` does not auto-detect a bucket's region, with or without the bridge. When neither the catalog (`s3.region` or `client.region`) nor the executor environment (`AWS_REGION` / `AWS_DEFAULT_REGION`) supplies a region, Comet uses `us-east-1`. That suits most non-AWS S3-compatible services but fails for AWS buckets in other regions, so set the region (and the endpoint for non-AWS) explicitly on the Spark catalog:
```
spark.sql.catalog.<catalog>.s3.region = us-east-1
@@ -110,8 +110,6 @@ spark.sql.catalog.<catalog>.s3.endpoint = https://... (non-AWS only)
spark.sql.catalog.<catalog>.s3.path-style-access = true (path-style endpoints only)
```
-If you hit `region is missing. Please find it by S3::detect_region() or set them in env`, this is the missing config.
-
## Writing a bridge
Comet's native scan paths (`object_store` for raw Parquet, `opendal` via `iceberg-rust` for Iceberg) bypass Hadoop S3A entirely. The standard `AWSCredentialsProvider.getCredentials()` has no path argument, so vendors that issue per-path STS credentials cannot expose them through it. The `CometS3CredentialProvider` SPI fills that gap.
@@ -296,10 +294,10 @@ public final class IcebergRESTVendedS3Provider implements CometS3CredentialProvi
### Access mode
-| Value | Used for |
-| ------- | -------------------------------------------------------------------------- |
-| `READ` | All native scan paths (raw Parquet, Iceberg). Comet today only sends READ. |
-| `WRITE` | Reserved for future native write paths. |
+| Value | Used for |
+| ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `READ` | All native scan paths (raw Parquet, Iceberg). |
+| `WRITE` | The native Iceberg writer (see [Iceberg Writes](iceberg-writes.md)). If the configured provider fails to initialize, the write fails rather than falling back to the default credential chain. |
A `WRITE` credential is not implicitly read-capable. Vendors that need read-during-write workflows include the required read permissions in the IAM policy attached to their `WRITE` credentials.
diff --git a/_sources/user-guide/latest/scala_java_udfs.md.txt b/_sources/user-guide/latest/scala_java_udfs.md.txt
index 3a55982bbc..43a3037457 100644
--- a/_sources/user-guide/latest/scala_java_udfs.md.txt
+++ b/_sources/user-guide/latest/scala_java_udfs.md.txt
@@ -34,7 +34,7 @@ This feature is enabled by default. Set `spark.comet.exec.scalaUDF.codegen.enabl
## Supported
- User functions registered via `udf(...)`, `spark.udf.register(...)` (Scala or Java functional interfaces), or SQL `CREATE FUNCTION ... AS 'com.example.MyUDF'`.
-- Scalar input/output types: `Boolean`, `Byte`, `Short`, `Int`, `Long`, `Float`, `Double`, `Decimal`, `String`, `Binary`, `Date`, `Timestamp`, `TimestampNTZ`.
+- Scalar input/output types: `Boolean`, `Byte`, `Short`, `Int`, `Long`, `Float`, `Double`, `Decimal`, `String`, `Binary`, `Date`, `Timestamp`, `TimestampNTZ`, `YearMonthInterval`, `DayTimeInterval`, `CalendarInterval`.
- Complex input/output types with arbitrary nesting: `ArrayType`, `StructType`, `MapType`.
- Composition with other Catalyst expressions inside the argument tree (e.g. `myUdf(upper(s))` runs as one unit in the Comet pipeline).
- Higher-order functions (`transform`, `filter`, `exists`, `aggregate`, `zip_with`, `map_filter`, `map_zip_with`, etc.) inside the argument tree.
@@ -45,7 +45,7 @@ This feature is enabled by default. Set `spark.comet.exec.scalaUDF.codegen.enabl
- Table UDFs and generators.
- Python `@udf` and Pandas `@pandas_udf`.
- Hive `GenericUDF` and `SimpleUDF`.
-- `CalendarIntervalType`, `NullType`, and `UserDefinedType` arguments and return types. UDT-typed columns fall back to Spark; to keep execution in the Comet pipeline, store and read the underlying representation directly (e.g. write MLlib `Vector` outputs as `Struct<type: Byte, size: Int, indices: Array<Int>, values: Array<Double>>` rather than `VectorUDT`).
+- `NullType` and `UserDefinedType` arguments and return types. UDT-typed columns fall back to Spark; to keep execution in the Comet pipeline, store and read the underlying representation directly (e.g. write MLlib `Vector` outputs as `Struct<type: Byte, size: Int, indices: Array<Int>, values: Array<Double>>` rather than `VectorUDT`).
- Trees whose total nested-field count (output plus all input columns the UDF tree references) exceeds `spark.sql.codegen.maxFields` (default 100). Comet refuses these at plan time and the operator falls back to Spark.
When a UDF is rejected, the reason surfaces through Comet's standard fallback diagnostics; the query still runs on Spark.
diff --git a/_sources/user-guide/latest/source.md.txt b/_sources/user-guide/latest/source.md.txt
index e629ede4dd..95aad9efd1 100644
--- a/_sources/user-guide/latest/source.md.txt
+++ b/_sources/user-guide/latest/source.md.txt
@@ -25,7 +25,7 @@ It is sometimes preferable to build from source for a specific platform.
This documentation is for the current development version of Comet. Published source releases are only available for released versions.
-To use this version of Comet, see the following section on building from the GitHubn repository.
+To use this version of Comet, see the following section on building from the GitHub repository.
diff --git a/_sources/user-guide/latest/tuning.md.txt b/_sources/user-guide/latest/tuning.md.txt
index 429befc648..7fcb8ec610 100644
--- a/_sources/user-guide/latest/tuning.md.txt
+++ b/_sources/user-guide/latest/tuning.md.txt
@@ -23,13 +23,11 @@ Comet provides some tuning options to help you get the best performance from you
## Configuring Tokio Runtime
-Comet uses a global tokio runtime per executor process using tokio's defaults of one worker thread per core and a
-maximum of 512 blocking threads. These values can be overridden using the environment variables `COMET_WORKER_THREADS`
-and `COMET_MAX_BLOCKING_THREADS`.
-
-It is recommended that `COMET_WORKER_THREADS` be set to the number of executor cores. This may not be necessary
-in some environments, such as Kubernetes, where the number of cores allocated to a pod will already be equal to the
-number of executor cores.
+Comet uses a global tokio runtime per executor process. By default it starts one worker thread per executor core
+(`spark.executor.cores`, or the thread count of `local[N]` and `local[*]` masters) and allows up to 512 blocking
+threads, which is tokio's default. If `spark.executor.cores` is not set outside local mode, Comet starts a single
+worker thread. These values can be overridden using the environment variables `COMET_WORKER_THREADS` and
+`COMET_MAX_BLOCKING_THREADS`.
## Adaptive Partial Aggregation
@@ -432,13 +430,17 @@ back to Spark for shuffle operations.
#### Native Shuffle
Comet provides a fully native shuffle implementation, which generally provides the best performance. Native shuffle
-supports `HashPartitioning`, `RangePartitioning` and `SinglePartitioning` but currently only supports primitive type
-partitioning keys. Columns that are not partitioning keys may contain complex types like maps, structs, and arrays.
+supports `HashPartitioning`, `RangePartitioning`, and `SinglePartition`, plus `RoundRobinPartitioning` when enabled
+(see [Round-Robin Partitioning](compatibility/operators.md#round-robin-partitioning)). Range partitioning keys must be
+scalar types. Hash partitioning keys must be scalar types unless
+`spark.comet.shuffle.native.partitioning.hash.nested.enabled=true`, which also admits struct, array, and (Spark 4.0
+and later) map keys. That setting is disabled by default until the performance of the nested hashing paths has been
+measured. Columns that are not partitioning keys may contain complex types like maps, structs, and arrays.
#### Columnar (JVM) Shuffle
Comet Columnar shuffle is JVM-based and supports `HashPartitioning`, `RoundRobinPartitioning`, `RangePartitioning`, and
-`SinglePartitioning`. This shuffle implementation supports complex data types as partitioning keys.
+`SinglePartition`. This shuffle implementation supports complex data types as partitioning keys.
By default, Comet will convert a Spark `ShuffleExchangeExec` to columnar shuffle even when the shuffle's child is a
non-Comet (Spark) plan. The benefit is that the next query stage can start as native Comet execution, since the
@@ -450,10 +452,11 @@ on Spark.
#### Automatic Revert to Spark Shuffle
-When a Comet columnar shuffle ends up between two non-Comet operators (for example, a partial/final hash aggregate
-pair that Comet could not convert), Comet reverts it to Spark's built-in shuffle. Keeping columnar shuffle between
-two row-based operators would add `row -> Arrow -> shuffle -> Arrow -> row` conversions with no Comet consumer on
-either side to benefit from columnar output.
+When a Comet columnar shuffle ends up between a partial and a final aggregate that Comet could not convert (both
+remain Spark `HashAggregateExec` or `ObjectHashAggregateExec` operators), Comet reverts it to Spark's built-in shuffle.
+Keeping columnar shuffle between the two row-based aggregates would add `row -> Arrow -> shuffle -> Arrow -> row`
+conversions with no Comet consumer on either side to benefit from columnar output. Other shuffles between non-Comet
+operators are not reverted.
This shifts the affected shuffles from Comet's off-heap memory pool back to the JVM execution memory pool. Clusters
tuned for a small JVM heap may see `ExternalSorter` spills on queries where this revert fires. Shuffle I/O may also
@@ -464,7 +467,7 @@ Each revert is logged at `INFO` level on the driver as `Reverting Comet columnar
This optimization is enabled by default and can be disabled by setting
`spark.comet.shuffle.revertRedundantColumnar.enabled=false`, in which case Comet will keep the columnar shuffle
-even when both its parent and child are non-Comet operators.
+even when both of those aggregates run on Spark.
### Remote Shuffle with Celeborn
@@ -562,26 +565,28 @@ repeated columnar-to-row and row-to-columnar conversions that dominate stage run
`spark.comet.exec.transitionRevert.enabled=true` to have Comet revert the entire stage to Spark row execution
when the number of columnar-to-row transitions exceeds
`spark.comet.exec.transitionRevert.maxTransitions` (default `2`). This trades native execution of a small
-subset of operators for eliminating conversion overhead across the stage.
+subset of operators for eliminating conversion overhead across the stage. A stage is not reverted when it holds a
+native aggregate whose intermediate buffer Spark cannot exchange with Comet across a stage boundary, because
+reverting it would split that aggregate between the two engines.
-### Entire-Plan Fallback for Wide or Deeply Nested Schemas
+### Wide or Deeply Nested Schemas
The cost of each conversion also grows sharply with schema shape: for wide or deeply nested schemas,
columnar-to-row conversion is especially expensive because the conversion work scales with the number of
columns and nested fields. If profiling shows these conversions dominating a query over such a schema, set
`spark.comet.exec.transitionRevert.enabled=true` and lower
-`spark.comet.exec.transitionRevert.maxTransitions` (default `2`) to `1`. Note that this causes the entire
-plan to fall back to Spark row-based execution — Comet removes its native operators rather than running a
+`spark.comet.exec.transitionRevert.maxTransitions` (default `2`) to `1`. Note that this reverts every stage that
+exceeds the threshold to Spark row-based execution — Comet removes the stage's native operators rather than running a
mix of native and fallback operators joined by repeated conversions — which can be cheaper than paying the
expensive conversions again and again.
## Metrics Overhead
-Comet exposes rich native operator metrics for observability (see [Metrics](metrics.md)), but they are
-disabled by default because traversing the Spark plan on every task adds measurable overhead, and metrics
-require an external sink (for example Prometheus) to be useful. Enable them with
-`spark.comet.metrics.enabled=true` when you have a metrics sink configured. This setting must be applied
-before the `SparkSession` is created.
+The SQL metrics described in [Metrics](metrics.md) are always collected. Setting `spark.comet.metrics.enabled=true`
+additionally publishes plan-coverage counters (`operators.native`, `operators.spark`, `queries.planned`,
+`transitions`, and `acceleration.ratio`) through Spark's metrics system under the `comet` source. It is disabled by
+default because it walks every executed plan on the driver after each query, and the counters are only useful with an
+external sink (for example Prometheus) configured. This setting must be applied before the `SparkSession` is created.
## Explain Plan
diff --git a/_sources/user-guide/latest/understanding-comet-plans.md.txt b/_sources/user-guide/latest/understanding-comet-plans.md.txt
index 6d2b361510..beb5c6ebd1 100644
--- a/_sources/user-guide/latest/understanding-comet-plans.md.txt
+++ b/_sources/user-guide/latest/understanding-comet-plans.md.txt
@@ -301,6 +301,9 @@ by role. Names match what is shown in the plan output.
| `CometNativeScan` | Parquet scan that runs entirely in Rust via DataFusion. |
| `CometIcebergNativeScan` | Iceberg Parquet scan that runs entirely in Rust via DataFusion. |
| `CometCsvNativeScan` | CSV scan that runs entirely in Rust via DataFusion (experimental). |
+| `CometInMemoryTableScan` | JVM-side scan of a table cached in Comet's Arrow format (experimental, disabled by default). |
+| `CometLocalTableScan` | JVM-side scan of a `LocalTableScanExec` (disabled by default). |
+| `CometEmptyRelation` | Empty native input that replaces `EmptyRelationExec` (Spark 4.0 and later). |
### Native Rust Operators
@@ -321,20 +324,28 @@ consecutively in a plan, they execute as a single fused block.
| `CometBroadcastHashJoin` | `BroadcastHashJoinExec` |
| `CometBroadcastNestedLoopJoin` | `BroadcastNestedLoopJoinExec` |
| `CometSortMergeJoin` | `SortMergeJoinExec` |
-| `CometWindow` | `WindowExec` |
+| `CometWindowExec` | `WindowExec` |
+| `CometWindowGroupLimitExec` | `WindowGroupLimitExec` (Spark 3.5 and later) |
+| `CometSample` | `SampleExec` (sampling without replacement) |
| `CometTakeOrderedAndProject` | `TakeOrderedAndProjectExec` |
+| `CometWriteFiles` | `WriteFilesExec` (Spark 4.0 and later, experimental native Parquet writes) |
+| `CometNativeWrite` | `DataWritingCommandExec` (Spark 3.x, experimental native Parquet writes) |
+| `CometIcebergWrite` | `IcebergWrite` (experimental native Iceberg data-file writes) |
### JVM-Side Operators
These keep their data on the JVM but participate in the Comet pipeline.
-| Node | Notes |
-| ------------------------ | ----------------------------------------------------------------------------------- |
-| `CometUnion` | JVM-side union of Comet inputs. The Rust side reads each branch as a separate scan. |
-| `CometCoalesce` | JVM-side partition coalesce. |
-| `CometCollectLimit` | JVM-side collect limit, equivalent to `CollectLimitExec`. |
-| `CometBroadcastExchange` | Broadcast exchange producing serialized Arrow batches that the consumer can decode. |
-| `CometSubqueryBroadcast` | Companion to `CometBroadcastExchange` for dynamic partition pruning subqueries. |
+| Node | Notes |
+| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------- |
+| `CometUnion` | JVM-side union of Comet inputs. The Rust side reads each branch as a separate scan. |
+| `CometCoalesce` | JVM-side partition coalesce. |
+| `CometCollectLimit` | JVM-side collect limit, equivalent to `CollectLimitExec`. |
+| `CometBroadcastExchange` | Broadcast exchange producing serialized Arrow batches that the consumer can decode. |
+| `CometSubqueryBroadcast` | Companion to `CometBroadcastExchange` for dynamic partition pruning subqueries. |
+| `CometMapInBatch` | Runs `mapInArrow` / `mapInPandas` Python UDFs on Comet's Arrow batches (experimental). See [PyArrow UDF Acceleration](pyarrow-udfs.md). |
+| `IcebergWrite` | Executor-side Iceberg data-file write in Comet's split-operator Iceberg write plan (experimental). See [Iceberg Writes](iceberg-writes.md). |
+| `IcebergCommit` | Driver-side commit for Comet's split-operator Iceberg write plan (experimental). |
### Shuffle Operators
@@ -344,21 +355,26 @@ use:
- **`CometExchange`** is the **native shuffle** path. The child must already
be a Comet operator producing columnar Arrow batches; the node calls
`executeColumnar()` on its child and the partition, encode, and compress
- steps run in Rust. Hash and range partitioning **keys** must be
- primitive types because the Rust hashing and ordering do not support complex
- types, but the data columns themselves can include `StructType`,
- `ArrayType`, and `MapType` since batches are serialized via the Arrow IPC
- writer.
+ steps run in Rust. Range partitioning **keys** must be scalar types. Hash
+ partitioning keys must also be scalar types unless
+ `spark.comet.shuffle.native.partitioning.hash.nested.enabled=true`, which
+ admits struct, array, and (Spark 4.0 and later) map keys. The data columns
+ themselves can include `StructType`, `ArrayType`, and `MapType` since
+ batches are serialized via the Arrow IPC writer.
- **`CometColumnarExchange`** is the **JVM columnar shuffle** path. It accepts
either Spark row-based input or Comet columnar input, which makes it the
fallback when the child is not a Comet operator or when a hash/range key
- type is not supported by native shuffle (for example, collated strings). It
- is still preferred over Spark's native shuffle when Comet shuffle is
- enabled.
+ type is not supported by native shuffle (for example, a struct or array
+ hash key while nested hash keys are disabled). It is still preferred over
+ Spark's native shuffle when Comet shuffle is enabled. Keys with a
+ non-default string collation are supported by neither path and use Spark's
+ shuffle.
Both paths support the same set of partitioning schemes
(`HashPartitioning`, `RangePartitioning`, `RoundRobinPartitioning`,
-`SinglePartition`) and both can carry complex types in data columns.
+`SinglePartition`) and both can carry complex types in data columns. Native
+round-robin partitioning is disabled by default; see
+[Round-Robin Partitioning](compatibility/operators.md#round-robin-partitioning).
The choice between the two is automatic. See the
[Tuning Guide shuffle section](tuning.md#shuffle) for how to enable Comet
diff --git a/contributor-guide/release_process.html b/contributor-guide/release_process.html
index 45a10607d2..24c44b607e 100644
--- a/contributor-guide/release_process.html
+++ b/contributor-guide/release_process.html
@@ -796,8 +796,6 @@ git<span class="w"> </span>tag<span class="w"> </span><span class="m">0</span>.1
git<span class="w"> </span>push<span class="w"> </span>apache<span class="w"> </span><span class="m">0</span>.13.0-rc1
</pre></div>
</div>
-<p>Note that pushing a release candidate tag will trigger a GitHub workflow that will build a Docker image and publish
-it to GitHub Container Registry at https://github.com/apache/datafusion-comet/pkgs/container/datafusion-comet</p>
</section>
<section id="publishing-documentation">
<h3>Publishing Documentation<a class="headerlink" href="#publishing-documentation" title="Link to this heading">#</a></h3>
@@ -959,8 +957,6 @@ git<span class="w"> </span>tag<span class="w"> </span><span class="m">0</span>.1
git<span class="w"> </span>push<span class="w"> </span>apache<span class="w"> </span><span class="m">0</span>.13.0
</pre></div>
</div>
-<p>Note that pushing a release tag will trigger a GitHub workflow that will build a Docker image and publish
-it to GitHub Container Registry at https://github.com/apache/datafusion-comet/pkgs/container/datafusion-comet</p>
<p>Reply to the vote thread to close the vote and announce the release. The announcement email should include:</p>
<ul class="simple">
<li><p>The release version</p></li>
diff --git a/searchindex.js b/searchindex.js
index af8193e669..ca27ecfa1d 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[56, "id1"]], "%": [[54, "id1"]], "&": [[44, "id1"]], "*": [[54, "id2"]], "+": [[54, "id3"]], "-": [[54, "id4"]], "/": [[54, "id5"]], "1. Format Your Code": [[41, "format-your-code"]], "1. Install Comet": [[62, "install-comet"], [74, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[41, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[62, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[56, "id1"]], "%": [[54, "id1"]], "&": [[44, "id1"]], "*": [[54, "id2"]], "+": [[54, "id3"]], "-": [[54, "id4"]], "/": [[54, "id5"]], "1. Format Your Code": [[41, "format-your-code"]], "1. Install Comet": [[62, "install-comet"], [74, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[41, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[62, "clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/1.0/s3-credential-providers.html b/user-guide/1.0/s3-credential-providers.html
index ed408128a0..e1c1a25ab6 100644
--- a/user-guide/1.0/s3-credential-providers.html
+++ b/user-guide/1.0/s3-credential-providers.html
@@ -945,6 +945,29 @@ under the License.
</pre></div>
</div>
</section>
+<section id="credentials-per-location">
+<h3>Credentials per location<a class="headerlink" href="#credentials-per-location" title="Link to this heading">#</a></h3>
+<p>On the Parquet path, a <code class="docutils literal notranslate"><span class="pre">CometS3CredentialProvider</span></code> gets one credential per bucket: Comet requests it with the path of the first file it reads from the bucket and uses it for every file there. If your policies differ by location within a bucket, for example one policy for <code class="docutils literal notranslate"><span class="pre">warehouse/sales</span></code> and another for <code class="docutils literal notrans [...]
+<div class="highlight-java notranslate"><div class="highlight"><pre><span></span><span class="kd">public</span><span class="w"> </span><span class="kd">final</span><span class="w"> </span><span class="kd">class</span> <span class="nc">MyLocationProvider</span><span class="w"> </span><span class="kd">implements</span><span class="w"> </span><span class="n">CometS3LocationScopedCredentialProvider</span><span class="w"> </span><span class="p">{</span>
+<span class="w"> </span><span class="nd">@Override</span>
+<span class="w"> </span><span class="kd">public</span><span class="w"> </span><span class="n">List</span><span class="o"><</span><span class="n">String</span><span class="o">></span><span class="w"> </span><span class="nf">getPolicyLocations</span><span class="p">(</span><span class="n">String</span><span class="w"> </span><span class="n">bucket</span><span class="p">)</span><span class="w"> </span><span class="kd">throws</span><span class="w"> </span><span class="n">Exception</ [...]
+<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">policyService</span><span class="p">.</span><span class="na">locationsWithPolicies</span><span class="p">(</span><span class="n">bucket</span><span class="p">);</span><span class="w"> </span><span class="c1">// ["warehouse/sales", "warehouse/finance"]</span>
+<span class="w"> </span><span class="p">}</span>
+
+<span class="w"> </span><span class="nd">@Override</span>
+<span class="w"> </span><span class="kd">public</span><span class="w"> </span><span class="n">CometS3Credentials</span><span class="w"> </span><span class="nf">getCredentialsForPath</span><span class="p">(</span><span class="n">CometS3CredentialContext</span><span class="w"> </span><span class="n">ctx</span><span class="p">)</span><span class="w"> </span><span class="kd">throws</span><span class="w"> </span><span class="n">Exception</span><span class="w"> </span><span class="p">{</span>
+<span class="w"> </span><span class="c1">// ctx.getPath() is a returned location with a leading slash, or "/" for the bucket root.</span>
+<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">sessionForLocation</span><span class="p">(</span><span class="n">ctx</span><span class="p">.</span><span class="na">getBucket</span><span class="p">(),</span><span class="w"> </span><span class="n">ctx</span><span class="p">.</span><span class="na">getPath</span><span class="p">(),</span><span class="w"> </span><span class="n">ctx</span><span class="p">.</span><span class="na">getMode</spa [...]
+<span class="w"> </span><span class="p">}</span>
+<span class="p">}</span>
+</pre></div>
+</div>
+<p>Comet serves each request with the credential of the longest location that covers its path. A location covers a path when the path is the location itself or lies below it, compared one <code class="docutils literal notranslate"><span class="pre">/</span></code>-separated segment at a time, so <code class="docutils literal notranslate"><span class="pre">warehouse/sales</span></code> covers <code class="docutils literal notranslate"><span class="pre">warehouse/sales/part-0.parquet</span [...]
+<p>Comet requests a location’s credential by calling <code class="docutils literal notranslate"><span class="pre">getCredentialsForPath</span></code> with the location as the path, as you returned it but with a leading slash. Every request under a location shares that credential, so it must authorize every path the location is the longest match for, and your cache can key on the location. Locations apply to Comet’s native Parquet reads only; Iceberg reads call <code class="docutils liter [...]
+<p><strong>When Comet asks.</strong> Comet calls <code class="docutils literal notranslate"><span class="pre">getPolicyLocations</span></code> when it creates the store for a bucket on an executor and keeps the answer for later reads of that bucket with the same S3 configuration. Reads that start at the same moment may each create a store and call it. If a read then fails with 403, Comet asks again, once for all the reads that failed on the same answer, and retries each read once if its [...]
+<p><strong>Failures.</strong> If <code class="docutils literal notranslate"><span class="pre">getPolicyLocations</span></code> throws or returns <code class="docutils literal notranslate"><span class="pre">null</span></code>, or returns a location that is <code class="docutils literal notranslate"><span class="pre">null</span></code> or invalid, the read fails. A location is invalid if, once decoded, it is not valid UTF-8 or has a segment that is empty, <code class="docutils literal notr [...]
+<p><strong>Backward compatibility.</strong> Providers that implement only <code class="docutils literal notranslate"><span class="pre">CometS3CredentialProvider</span></code> are unaffected. Comet calls nothing new on them and keeps one credential per bucket, as before.</p>
+</section>
<section id="composing-multiple-credential-backends">
<h3>Composing multiple credential backends<a class="headerlink" href="#composing-multiple-credential-backends" title="Link to this heading">#</a></h3>
<p>A single configured provider class is the dispatcher. If a vendor needs to route across several credential backends (per bucket, per path prefix, per tenant), the dispatch lives inside the vendor’s class:</p>
@@ -1100,6 +1123,7 @@ under the License.
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#caching-refresh-and-distribution">Caching, refresh, and distribution</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#returned-fields">Returned fields</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#returns-or-throws">Returns or throws</a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#credentials-per-location">Credentials per location</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#composing-multiple-credential-backends">Composing multiple credential backends</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#reference-implementation-iceberg-rest-vended-credentials">Reference implementation: Iceberg REST vended credentials</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#access-mode">Access mode</a></li>
diff --git a/user-guide/latest/compatibility/expressions/index.html b/user-guide/latest/compatibility/expressions/index.html
index 5e7a0c6479..9718d0d81c 100644
--- a/user-guide/latest/compatibility/expressions/index.html
+++ b/user-guide/latest/compatibility/expressions/index.html
@@ -832,6 +832,8 @@ version you are running:</p>
</li>
</ul>
</div>
+<p>Spark 3.4 support is deprecated and will be removed in a future release; see
+<a class="reference internal" href="../spark-versions.html#spark-3-4"><span class="std std-ref">Spark Version Compatibility</span></a>.</p>
<p>Expressions that are not 100% Spark-compatible fall back to Spark by default, except those
with a JVM codegen-dispatch path, which stay in Comet’s native pipeline and match Spark
exactly. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.EXPRNAME.allowIncompatible=true</span></code>, where <code class="docutils literal notranslate"><span class="pre">EXPRNAME</span></code> is
diff --git a/user-guide/latest/compatibility/expressions/spark-3.4/cast.html b/user-guide/latest/compatibility/expressions/spark-3.4/cast.html
index f8274538c9..ebc25d0ee4 100644
--- a/user-guide/latest/compatibility/expressions/spark-3.4/cast.html
+++ b/user-guide/latest/compatibility/expressions/spark-3.4/cast.html
@@ -786,17 +786,22 @@ under the License.
<p>Cast operations in Comet fall into three levels of support:</p>
<ul class="simple">
<li><p><strong>C (Compatible)</strong>: The results match Apache Spark</p></li>
-<li><p><strong>I (Incompatible)</strong>: The results may match Apache Spark for some inputs, but there are known issues where some inputs
-will result in incorrect results or exceptions. The query stage will fall back to Spark by default. Setting
-<code class="docutils literal notranslate"><span class="pre">spark.comet.expression.Cast.allowIncompatible=true</span></code> will allow all incompatible casts to run natively in Comet, but this is not
-recommended for production use.</p></li>
-<li><p><strong>U (Unsupported)</strong>: Comet does not provide a native version of this cast expression and the query stage will fall back to
-Spark.</p></li>
+<li><p><strong>I (Incompatible)</strong>: Comet’s native cast may produce different results from Apache Spark for some inputs. By
+default, Comet instead runs Spark’s own code-generated cast inside the Comet pipeline, so the results match Spark.
+Setting <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.Cast.allowIncompatible=true</span></code> selects the native cast, but this is not recommended
+for production use.</p></li>
+<li><p><strong>U (Unsupported)</strong>: Comet does not provide a native version of this cast, so it runs Spark’s own code-generated
+cast inside the Comet pipeline.</p></li>
<li><p><strong>N/A</strong>: Spark does not support this cast.</p></li>
</ul>
-<section id="ansi-mode-fallback">
-<h2>ANSI Mode Fallback<a class="headerlink" href="#ansi-mode-fallback" title="Link to this heading">#</a></h2>
-<p>Cast will fall back to Spark in some cases when ANSI mode is enabled. This can be enabled by setting <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.Cast.allowIncompatible=true</span></code>. See the <a class="reference internal" href="../../../expressions.html"><span class="std std-doc">Comet Supported Expressions Guide</span></a> for more information on this configuration setting.</p>
+<p>Incompatible and unsupported casts fall back to Spark only when Comet’s codegen dispatcher cannot handle them (for
+example, casts involving <code class="docutils literal notranslate"><span class="pre">VariantType</span></code>) or when <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.scalaUDF.codegen.enabled=false</span></code>.</p>
+<section id="ansi-mode-support">
+<h2>ANSI Mode Support<a class="headerlink" href="#ansi-mode-support" title="Link to this heading">#</a></h2>
+<p>Enabling ANSI mode does not by itself make a cast fall back to Spark or require
+<code class="docutils literal notranslate"><span class="pre">spark.comet.expression.Cast.allowIncompatible=true</span></code>: every cast that Spark permits in ANSI mode has the same support
+level as in legacy mode. See the <a class="reference internal" href="#ansi-mode">ANSI Mode</a> table below for per-pair support, and the
+<a class="reference internal" href="../../../expressions.html"><span class="std std-doc">Comet Supported Expressions Guide</span></a> for more information on the <code class="docutils literal notranslate"><span class="pre">allowIncompatible</span></code> setting.</p>
</section>
<section id="whitespace-trimming-in-casts-from-string">
<h2>Whitespace Trimming in Casts from String<a class="headerlink" href="#whitespace-trimming-in-casts-from-string" title="Link to this heading">#</a></h2>
@@ -942,20 +947,12 @@ the same result regardless of the session timezone.</p>
zero-padded to four digits (e.g. year 999 renders as <code class="docutils literal notranslate"><span class="pre">0999-01-01</span></code>). Years above 9999 are
rendered without truncation. The cast is timezone-independent.</p>
</section>
-<section id="id1">
-<h2>String to TimestampNTZ<a class="headerlink" href="#id1" title="Link to this heading">#</a></h2>
-<p>Comet’s native <code class="docutils literal notranslate"><span class="pre">CAST(string</span> <span class="pre">AS</span> <span class="pre">TIMESTAMP_NTZ)</span></code> implementation matches Apache Spark’s behavior.
-Unlike <code class="docutils literal notranslate"><span class="pre">CAST(string</span> <span class="pre">AS</span> <span class="pre">TIMESTAMP)</span></code>, this cast is timezone-independent: any timezone offset in
-the input string (e.g. <code class="docutils literal notranslate"><span class="pre">+08:00</span></code>, <code class="docutils literal notranslate"><span class="pre">Z</span></code>, <code class="docutils literal notranslate"><span class="pre">UTC</span></code>) is silently discarded, and the local date-time
-components are preserved as-is. Time-only strings (e.g. <code class="docutils literal notranslate"><span class="pre">T12:34:56</span></code>, <code class="docutils literal notranslate"><span class="pre">12:34</span></code>) produce <code class="docutils literal notranslate"><span class="pre">NULL</span></code>.
-The result is always a wall-clock timestamp with no timezone conversion or DST adjustment.</p>
-</section>
<section id="decimal-with-negative-scale-to-string">
<h2>Decimal with Negative Scale to String<a class="headerlink" href="#decimal-with-negative-scale-to-string" title="Link to this heading">#</a></h2>
<p>Casting a <code class="docutils literal notranslate"><span class="pre">DecimalType</span></code> with a negative scale to <code class="docutils literal notranslate"><span class="pre">StringType</span></code> is marked as incompatible when
<code class="docutils literal notranslate"><span class="pre">spark.sql.legacy.allowNegativeScaleOfDecimal</span></code> is <code class="docutils literal notranslate"><span class="pre">false</span></code> (the default). When that config is
-disabled, Spark cannot create negative-scale decimals, so Comet falls back to avoid running
-native execution on unexpected inputs.</p>
+disabled, Spark cannot create negative-scale decimals, so by default Comet runs Spark’s
+code-generated cast instead of the native one.</p>
<p>When <code class="docutils literal notranslate"><span class="pre">spark.sql.legacy.allowNegativeScaleOfDecimal=true</span></code>, the cast is compatible. Comet matches
Spark’s behavior of using Java <code class="docutils literal notranslate"><span class="pre">BigDecimal.toString()</span></code> semantics, which produces scientific
notation (e.g. a value of 12300 stored as <code class="docutils literal notranslate"><span class="pre">Decimal(7,-2)</span></code> with unscaled value 123 is rendered
@@ -1706,7 +1703,7 @@ as <code class="docutils literal notranslate"><span class="pre">"1.23E+4&qu
</div>
<nav class="bd-toc-nav page-toc" aria-labelledby="pst-page-navigation-heading-2">
<ul class="visible nav section-nav flex-column">
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#ansi-mode-fallback">ANSI Mode Fallback</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#ansi-mode-support">ANSI Mode Support</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#whitespace-trimming-in-casts-from-string">Whitespace Trimming in Casts from String</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#string-to-decimal">String to Decimal</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#string-to-date">String to Date</a></li>
@@ -1717,7 +1714,6 @@ as <code class="docutils literal notranslate"><span class="pre">"1.23E+4&qu
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#string-to-timestampntz">String to TimestampNTZ</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#timestampntz-casts">TimestampNTZ Casts</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#date-to-string">Date to String</a></li>
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#id1">String to TimestampNTZ</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#decimal-with-negative-scale-to-string">Decimal with Negative Scale to String</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#legacy-mode">Legacy Mode</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#try-mode">Try Mode</a></li>
diff --git a/user-guide/latest/compatibility/expressions/spark-3.4/index.html b/user-guide/latest/compatibility/expressions/spark-3.4/index.html
index 8962be9319..e05ff7abcb 100644
--- a/user-guide/latest/compatibility/expressions/spark-3.4/index.html
+++ b/user-guide/latest/compatibility/expressions/spark-3.4/index.html
@@ -783,6 +783,8 @@ path, which stay in Comet’s native pipeline and match Spark exactly. Set
expression class name, to run Comet’s native implementation despite its differences
from Spark. See the <a class="reference internal" href="../../../expressions.html"><span class="std std-doc">Comet Supported Expressions Guide</span></a>
for more information on this configuration setting.</p>
+<p>Spark 3.4 support is deprecated and will be removed in a future release; see
+<a class="reference internal" href="../../spark-versions.html#spark-3-4"><span class="std std-ref">Spark Version Compatibility</span></a>.</p>
<div class="toctree-wrapper compound">
<ul>
<li class="toctree-l1"><a class="reference internal" href="aggregate.html">Aggregate Expressions</a></li>
diff --git a/user-guide/latest/compatibility/expressions/spark-3.4/map.html b/user-guide/latest/compatibility/expressions/spark-3.4/map.html
index 1860eaf64a..fa5054846d 100644
--- a/user-guide/latest/compatibility/expressions/spark-3.4/map.html
+++ b/user-guide/latest/compatibility/expressions/spark-3.4/map.html
@@ -789,7 +789,7 @@ under the License.
keys, in <code class="docutils literal notranslate"><span class="pre">try_element_at</span></code>, and in other contexts where map ordering must be deterministic. Comet
runs <code class="docutils literal notranslate"><span class="pre">MapSort</span></code> natively, so map shuffle and group-by-on-map stay on Comet under Spark 4.0.</p>
<p>When <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.strictFloatingPoint=true</span></code>, <code class="docutils literal notranslate"><span class="pre">MapSort</span></code> falls back to Spark for maps whose
-keys contain <code class="docutils literal notranslate"><span class="pre">Float</span></code> or <code class="docutils literal notranslate"><span class="pre">Double</span></code> (consistent with <code class="docutils literal notranslate"><span class="pre">SortOrder</span></code> and <code class="docutils literal notranslate"><span class="pre">SortArray</span></code>). Arrow’s sort uses
+keys contain <code class="docutils literal notranslate"><span class="pre">Float</span></code> or <code class="docutils literal notranslate"><span class="pre">Double</span></code>. Arrow’s sort uses
IEEE total ordering for floating-point, which differs from Spark’s <code class="docutils literal notranslate"><span class="pre">Double.compare</span></code> semantics for
<code class="docutils literal notranslate"><span class="pre">NaN</span></code> and <code class="docutils literal notranslate"><span class="pre">-0.0</span></code>.</p>
<!--BEGIN:EXPR_COMPAT[map]-->
diff --git a/user-guide/latest/compatibility/expressions/spark-3.5/cast.html b/user-guide/latest/compatibility/expressions/spark-3.5/cast.html
index ab949de566..dbc3c7162a 100644
--- a/user-guide/latest/compatibility/expressions/spark-3.5/cast.html
+++ b/user-guide/latest/compatibility/expressions/spark-3.5/cast.html
@@ -786,17 +786,22 @@ under the License.
<p>Cast operations in Comet fall into three levels of support:</p>
<ul class="simple">
<li><p><strong>C (Compatible)</strong>: The results match Apache Spark</p></li>
-<li><p><strong>I (Incompatible)</strong>: The results may match Apache Spark for some inputs, but there are known issues where some inputs
-will result in incorrect results or exceptions. The query stage will fall back to Spark by default. Setting
-<code class="docutils literal notranslate"><span class="pre">spark.comet.expression.Cast.allowIncompatible=true</span></code> will allow all incompatible casts to run natively in Comet, but this is not
-recommended for production use.</p></li>
-<li><p><strong>U (Unsupported)</strong>: Comet does not provide a native version of this cast expression and the query stage will fall back to
-Spark.</p></li>
+<li><p><strong>I (Incompatible)</strong>: Comet’s native cast may produce different results from Apache Spark for some inputs. By
+default, Comet instead runs Spark’s own code-generated cast inside the Comet pipeline, so the results match Spark.
+Setting <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.Cast.allowIncompatible=true</span></code> selects the native cast, but this is not recommended
+for production use.</p></li>
+<li><p><strong>U (Unsupported)</strong>: Comet does not provide a native version of this cast, so it runs Spark’s own code-generated
+cast inside the Comet pipeline.</p></li>
<li><p><strong>N/A</strong>: Spark does not support this cast.</p></li>
</ul>
-<section id="ansi-mode-fallback">
-<h2>ANSI Mode Fallback<a class="headerlink" href="#ansi-mode-fallback" title="Link to this heading">#</a></h2>
-<p>Cast will fall back to Spark in some cases when ANSI mode is enabled. This can be enabled by setting <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.Cast.allowIncompatible=true</span></code>. See the <a class="reference internal" href="../../../expressions.html"><span class="std std-doc">Comet Supported Expressions Guide</span></a> for more information on this configuration setting.</p>
+<p>Incompatible and unsupported casts fall back to Spark only when Comet’s codegen dispatcher cannot handle them (for
+example, casts involving <code class="docutils literal notranslate"><span class="pre">VariantType</span></code>) or when <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.scalaUDF.codegen.enabled=false</span></code>.</p>
+<section id="ansi-mode-support">
+<h2>ANSI Mode Support<a class="headerlink" href="#ansi-mode-support" title="Link to this heading">#</a></h2>
+<p>Enabling ANSI mode does not by itself make a cast fall back to Spark or require
+<code class="docutils literal notranslate"><span class="pre">spark.comet.expression.Cast.allowIncompatible=true</span></code>: every cast that Spark permits in ANSI mode has the same support
+level as in legacy mode. See the <a class="reference internal" href="#ansi-mode">ANSI Mode</a> table below for per-pair support, and the
+<a class="reference internal" href="../../../expressions.html"><span class="std std-doc">Comet Supported Expressions Guide</span></a> for more information on the <code class="docutils literal notranslate"><span class="pre">allowIncompatible</span></code> setting.</p>
</section>
<section id="whitespace-trimming-in-casts-from-string">
<h2>Whitespace Trimming in Casts from String<a class="headerlink" href="#whitespace-trimming-in-casts-from-string" title="Link to this heading">#</a></h2>
@@ -942,20 +947,12 @@ the same result regardless of the session timezone.</p>
zero-padded to four digits (e.g. year 999 renders as <code class="docutils literal notranslate"><span class="pre">0999-01-01</span></code>). Years above 9999 are
rendered without truncation. The cast is timezone-independent.</p>
</section>
-<section id="id1">
-<h2>String to TimestampNTZ<a class="headerlink" href="#id1" title="Link to this heading">#</a></h2>
-<p>Comet’s native <code class="docutils literal notranslate"><span class="pre">CAST(string</span> <span class="pre">AS</span> <span class="pre">TIMESTAMP_NTZ)</span></code> implementation matches Apache Spark’s behavior.
-Unlike <code class="docutils literal notranslate"><span class="pre">CAST(string</span> <span class="pre">AS</span> <span class="pre">TIMESTAMP)</span></code>, this cast is timezone-independent: any timezone offset in
-the input string (e.g. <code class="docutils literal notranslate"><span class="pre">+08:00</span></code>, <code class="docutils literal notranslate"><span class="pre">Z</span></code>, <code class="docutils literal notranslate"><span class="pre">UTC</span></code>) is silently discarded, and the local date-time
-components are preserved as-is. Time-only strings (e.g. <code class="docutils literal notranslate"><span class="pre">T12:34:56</span></code>, <code class="docutils literal notranslate"><span class="pre">12:34</span></code>) produce <code class="docutils literal notranslate"><span class="pre">NULL</span></code>.
-The result is always a wall-clock timestamp with no timezone conversion or DST adjustment.</p>
-</section>
<section id="decimal-with-negative-scale-to-string">
<h2>Decimal with Negative Scale to String<a class="headerlink" href="#decimal-with-negative-scale-to-string" title="Link to this heading">#</a></h2>
<p>Casting a <code class="docutils literal notranslate"><span class="pre">DecimalType</span></code> with a negative scale to <code class="docutils literal notranslate"><span class="pre">StringType</span></code> is marked as incompatible when
<code class="docutils literal notranslate"><span class="pre">spark.sql.legacy.allowNegativeScaleOfDecimal</span></code> is <code class="docutils literal notranslate"><span class="pre">false</span></code> (the default). When that config is
-disabled, Spark cannot create negative-scale decimals, so Comet falls back to avoid running
-native execution on unexpected inputs.</p>
+disabled, Spark cannot create negative-scale decimals, so by default Comet runs Spark’s
+code-generated cast instead of the native one.</p>
<p>When <code class="docutils literal notranslate"><span class="pre">spark.sql.legacy.allowNegativeScaleOfDecimal=true</span></code>, the cast is compatible. Comet matches
Spark’s behavior of using Java <code class="docutils literal notranslate"><span class="pre">BigDecimal.toString()</span></code> semantics, which produces scientific
notation (e.g. a value of 12300 stored as <code class="docutils literal notranslate"><span class="pre">Decimal(7,-2)</span></code> with unscaled value 123 is rendered
@@ -1706,7 +1703,7 @@ as <code class="docutils literal notranslate"><span class="pre">"1.23E+4&qu
</div>
<nav class="bd-toc-nav page-toc" aria-labelledby="pst-page-navigation-heading-2">
<ul class="visible nav section-nav flex-column">
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#ansi-mode-fallback">ANSI Mode Fallback</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#ansi-mode-support">ANSI Mode Support</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#whitespace-trimming-in-casts-from-string">Whitespace Trimming in Casts from String</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#string-to-decimal">String to Decimal</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#string-to-date">String to Date</a></li>
@@ -1717,7 +1714,6 @@ as <code class="docutils literal notranslate"><span class="pre">"1.23E+4&qu
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#string-to-timestampntz">String to TimestampNTZ</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#timestampntz-casts">TimestampNTZ Casts</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#date-to-string">Date to String</a></li>
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#id1">String to TimestampNTZ</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#decimal-with-negative-scale-to-string">Decimal with Negative Scale to String</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#legacy-mode">Legacy Mode</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#try-mode">Try Mode</a></li>
diff --git a/user-guide/latest/compatibility/expressions/spark-3.5/map.html b/user-guide/latest/compatibility/expressions/spark-3.5/map.html
index f52cd73890..c037f5791f 100644
--- a/user-guide/latest/compatibility/expressions/spark-3.5/map.html
+++ b/user-guide/latest/compatibility/expressions/spark-3.5/map.html
@@ -789,7 +789,7 @@ under the License.
keys, in <code class="docutils literal notranslate"><span class="pre">try_element_at</span></code>, and in other contexts where map ordering must be deterministic. Comet
runs <code class="docutils literal notranslate"><span class="pre">MapSort</span></code> natively, so map shuffle and group-by-on-map stay on Comet under Spark 4.0.</p>
<p>When <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.strictFloatingPoint=true</span></code>, <code class="docutils literal notranslate"><span class="pre">MapSort</span></code> falls back to Spark for maps whose
-keys contain <code class="docutils literal notranslate"><span class="pre">Float</span></code> or <code class="docutils literal notranslate"><span class="pre">Double</span></code> (consistent with <code class="docutils literal notranslate"><span class="pre">SortOrder</span></code> and <code class="docutils literal notranslate"><span class="pre">SortArray</span></code>). Arrow’s sort uses
+keys contain <code class="docutils literal notranslate"><span class="pre">Float</span></code> or <code class="docutils literal notranslate"><span class="pre">Double</span></code>. Arrow’s sort uses
IEEE total ordering for floating-point, which differs from Spark’s <code class="docutils literal notranslate"><span class="pre">Double.compare</span></code> semantics for
<code class="docutils literal notranslate"><span class="pre">NaN</span></code> and <code class="docutils literal notranslate"><span class="pre">-0.0</span></code>.</p>
<!--BEGIN:EXPR_COMPAT[map]-->
diff --git a/user-guide/latest/compatibility/expressions/spark-4.0/cast.html b/user-guide/latest/compatibility/expressions/spark-4.0/cast.html
index c1d54dd2e4..5c8368b53e 100644
--- a/user-guide/latest/compatibility/expressions/spark-4.0/cast.html
+++ b/user-guide/latest/compatibility/expressions/spark-4.0/cast.html
@@ -786,17 +786,22 @@ under the License.
<p>Cast operations in Comet fall into three levels of support:</p>
<ul class="simple">
<li><p><strong>C (Compatible)</strong>: The results match Apache Spark</p></li>
-<li><p><strong>I (Incompatible)</strong>: The results may match Apache Spark for some inputs, but there are known issues where some inputs
-will result in incorrect results or exceptions. The query stage will fall back to Spark by default. Setting
-<code class="docutils literal notranslate"><span class="pre">spark.comet.expression.Cast.allowIncompatible=true</span></code> will allow all incompatible casts to run natively in Comet, but this is not
-recommended for production use.</p></li>
-<li><p><strong>U (Unsupported)</strong>: Comet does not provide a native version of this cast expression and the query stage will fall back to
-Spark.</p></li>
+<li><p><strong>I (Incompatible)</strong>: Comet’s native cast may produce different results from Apache Spark for some inputs. By
+default, Comet instead runs Spark’s own code-generated cast inside the Comet pipeline, so the results match Spark.
+Setting <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.Cast.allowIncompatible=true</span></code> selects the native cast, but this is not recommended
+for production use.</p></li>
+<li><p><strong>U (Unsupported)</strong>: Comet does not provide a native version of this cast, so it runs Spark’s own code-generated
+cast inside the Comet pipeline.</p></li>
<li><p><strong>N/A</strong>: Spark does not support this cast.</p></li>
</ul>
-<section id="ansi-mode-fallback">
-<h2>ANSI Mode Fallback<a class="headerlink" href="#ansi-mode-fallback" title="Link to this heading">#</a></h2>
-<p>Cast will fall back to Spark in some cases when ANSI mode is enabled. This can be enabled by setting <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.Cast.allowIncompatible=true</span></code>. See the <a class="reference internal" href="../../../expressions.html"><span class="std std-doc">Comet Supported Expressions Guide</span></a> for more information on this configuration setting.</p>
+<p>Incompatible and unsupported casts fall back to Spark only when Comet’s codegen dispatcher cannot handle them (for
+example, casts involving <code class="docutils literal notranslate"><span class="pre">VariantType</span></code>) or when <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.scalaUDF.codegen.enabled=false</span></code>.</p>
+<section id="ansi-mode-support">
+<h2>ANSI Mode Support<a class="headerlink" href="#ansi-mode-support" title="Link to this heading">#</a></h2>
+<p>Enabling ANSI mode does not by itself make a cast fall back to Spark or require
+<code class="docutils literal notranslate"><span class="pre">spark.comet.expression.Cast.allowIncompatible=true</span></code>: every cast that Spark permits in ANSI mode has the same support
+level as in legacy mode. See the <a class="reference internal" href="#ansi-mode">ANSI Mode</a> table below for per-pair support, and the
+<a class="reference internal" href="../../../expressions.html"><span class="std std-doc">Comet Supported Expressions Guide</span></a> for more information on the <code class="docutils literal notranslate"><span class="pre">allowIncompatible</span></code> setting.</p>
</section>
<section id="whitespace-trimming-in-casts-from-string">
<h2>Whitespace Trimming in Casts from String<a class="headerlink" href="#whitespace-trimming-in-casts-from-string" title="Link to this heading">#</a></h2>
@@ -942,20 +947,12 @@ the same result regardless of the session timezone.</p>
zero-padded to four digits (e.g. year 999 renders as <code class="docutils literal notranslate"><span class="pre">0999-01-01</span></code>). Years above 9999 are
rendered without truncation. The cast is timezone-independent.</p>
</section>
-<section id="id1">
-<h2>String to TimestampNTZ<a class="headerlink" href="#id1" title="Link to this heading">#</a></h2>
-<p>Comet’s native <code class="docutils literal notranslate"><span class="pre">CAST(string</span> <span class="pre">AS</span> <span class="pre">TIMESTAMP_NTZ)</span></code> implementation matches Apache Spark’s behavior.
-Unlike <code class="docutils literal notranslate"><span class="pre">CAST(string</span> <span class="pre">AS</span> <span class="pre">TIMESTAMP)</span></code>, this cast is timezone-independent: any timezone offset in
-the input string (e.g. <code class="docutils literal notranslate"><span class="pre">+08:00</span></code>, <code class="docutils literal notranslate"><span class="pre">Z</span></code>, <code class="docutils literal notranslate"><span class="pre">UTC</span></code>) is silently discarded, and the local date-time
-components are preserved as-is. Time-only strings (e.g. <code class="docutils literal notranslate"><span class="pre">T12:34:56</span></code>, <code class="docutils literal notranslate"><span class="pre">12:34</span></code>) produce <code class="docutils literal notranslate"><span class="pre">NULL</span></code>.
-The result is always a wall-clock timestamp with no timezone conversion or DST adjustment.</p>
-</section>
<section id="decimal-with-negative-scale-to-string">
<h2>Decimal with Negative Scale to String<a class="headerlink" href="#decimal-with-negative-scale-to-string" title="Link to this heading">#</a></h2>
<p>Casting a <code class="docutils literal notranslate"><span class="pre">DecimalType</span></code> with a negative scale to <code class="docutils literal notranslate"><span class="pre">StringType</span></code> is marked as incompatible when
<code class="docutils literal notranslate"><span class="pre">spark.sql.legacy.allowNegativeScaleOfDecimal</span></code> is <code class="docutils literal notranslate"><span class="pre">false</span></code> (the default). When that config is
-disabled, Spark cannot create negative-scale decimals, so Comet falls back to avoid running
-native execution on unexpected inputs.</p>
+disabled, Spark cannot create negative-scale decimals, so by default Comet runs Spark’s
+code-generated cast instead of the native one.</p>
<p>When <code class="docutils literal notranslate"><span class="pre">spark.sql.legacy.allowNegativeScaleOfDecimal=true</span></code>, the cast is compatible. Comet matches
Spark’s behavior of using Java <code class="docutils literal notranslate"><span class="pre">BigDecimal.toString()</span></code> semantics, which produces scientific
notation (e.g. a value of 12300 stored as <code class="docutils literal notranslate"><span class="pre">Decimal(7,-2)</span></code> with unscaled value 123 is rendered
@@ -1706,7 +1703,7 @@ as <code class="docutils literal notranslate"><span class="pre">"1.23E+4&qu
</div>
<nav class="bd-toc-nav page-toc" aria-labelledby="pst-page-navigation-heading-2">
<ul class="visible nav section-nav flex-column">
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#ansi-mode-fallback">ANSI Mode Fallback</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#ansi-mode-support">ANSI Mode Support</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#whitespace-trimming-in-casts-from-string">Whitespace Trimming in Casts from String</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#string-to-decimal">String to Decimal</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#string-to-date">String to Date</a></li>
@@ -1717,7 +1714,6 @@ as <code class="docutils literal notranslate"><span class="pre">"1.23E+4&qu
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#string-to-timestampntz">String to TimestampNTZ</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#timestampntz-casts">TimestampNTZ Casts</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#date-to-string">Date to String</a></li>
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#id1">String to TimestampNTZ</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#decimal-with-negative-scale-to-string">Decimal with Negative Scale to String</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#legacy-mode">Legacy Mode</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#try-mode">Try Mode</a></li>
diff --git a/user-guide/latest/compatibility/expressions/spark-4.0/map.html b/user-guide/latest/compatibility/expressions/spark-4.0/map.html
index 5e18803810..2e2b9864d8 100644
--- a/user-guide/latest/compatibility/expressions/spark-4.0/map.html
+++ b/user-guide/latest/compatibility/expressions/spark-4.0/map.html
@@ -789,7 +789,7 @@ under the License.
keys, in <code class="docutils literal notranslate"><span class="pre">try_element_at</span></code>, and in other contexts where map ordering must be deterministic. Comet
runs <code class="docutils literal notranslate"><span class="pre">MapSort</span></code> natively, so map shuffle and group-by-on-map stay on Comet under Spark 4.0.</p>
<p>When <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.strictFloatingPoint=true</span></code>, <code class="docutils literal notranslate"><span class="pre">MapSort</span></code> falls back to Spark for maps whose
-keys contain <code class="docutils literal notranslate"><span class="pre">Float</span></code> or <code class="docutils literal notranslate"><span class="pre">Double</span></code> (consistent with <code class="docutils literal notranslate"><span class="pre">SortOrder</span></code> and <code class="docutils literal notranslate"><span class="pre">SortArray</span></code>). Arrow’s sort uses
+keys contain <code class="docutils literal notranslate"><span class="pre">Float</span></code> or <code class="docutils literal notranslate"><span class="pre">Double</span></code>. Arrow’s sort uses
IEEE total ordering for floating-point, which differs from Spark’s <code class="docutils literal notranslate"><span class="pre">Double.compare</span></code> semantics for
<code class="docutils literal notranslate"><span class="pre">NaN</span></code> and <code class="docutils literal notranslate"><span class="pre">-0.0</span></code>.</p>
<!--BEGIN:EXPR_COMPAT[map]-->
diff --git a/user-guide/latest/compatibility/expressions/spark-4.1/cast.html b/user-guide/latest/compatibility/expressions/spark-4.1/cast.html
index 243811accd..cde0749617 100644
--- a/user-guide/latest/compatibility/expressions/spark-4.1/cast.html
+++ b/user-guide/latest/compatibility/expressions/spark-4.1/cast.html
@@ -786,17 +786,22 @@ under the License.
<p>Cast operations in Comet fall into three levels of support:</p>
<ul class="simple">
<li><p><strong>C (Compatible)</strong>: The results match Apache Spark</p></li>
-<li><p><strong>I (Incompatible)</strong>: The results may match Apache Spark for some inputs, but there are known issues where some inputs
-will result in incorrect results or exceptions. The query stage will fall back to Spark by default. Setting
-<code class="docutils literal notranslate"><span class="pre">spark.comet.expression.Cast.allowIncompatible=true</span></code> will allow all incompatible casts to run natively in Comet, but this is not
-recommended for production use.</p></li>
-<li><p><strong>U (Unsupported)</strong>: Comet does not provide a native version of this cast expression and the query stage will fall back to
-Spark.</p></li>
+<li><p><strong>I (Incompatible)</strong>: Comet’s native cast may produce different results from Apache Spark for some inputs. By
+default, Comet instead runs Spark’s own code-generated cast inside the Comet pipeline, so the results match Spark.
+Setting <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.Cast.allowIncompatible=true</span></code> selects the native cast, but this is not recommended
+for production use.</p></li>
+<li><p><strong>U (Unsupported)</strong>: Comet does not provide a native version of this cast, so it runs Spark’s own code-generated
+cast inside the Comet pipeline.</p></li>
<li><p><strong>N/A</strong>: Spark does not support this cast.</p></li>
</ul>
-<section id="ansi-mode-fallback">
-<h2>ANSI Mode Fallback<a class="headerlink" href="#ansi-mode-fallback" title="Link to this heading">#</a></h2>
-<p>Cast will fall back to Spark in some cases when ANSI mode is enabled. This can be enabled by setting <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.Cast.allowIncompatible=true</span></code>. See the <a class="reference internal" href="../../../expressions.html"><span class="std std-doc">Comet Supported Expressions Guide</span></a> for more information on this configuration setting.</p>
+<p>Incompatible and unsupported casts fall back to Spark only when Comet’s codegen dispatcher cannot handle them (for
+example, casts involving <code class="docutils literal notranslate"><span class="pre">VariantType</span></code>) or when <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.scalaUDF.codegen.enabled=false</span></code>.</p>
+<section id="ansi-mode-support">
+<h2>ANSI Mode Support<a class="headerlink" href="#ansi-mode-support" title="Link to this heading">#</a></h2>
+<p>Enabling ANSI mode does not by itself make a cast fall back to Spark or require
+<code class="docutils literal notranslate"><span class="pre">spark.comet.expression.Cast.allowIncompatible=true</span></code>: every cast that Spark permits in ANSI mode has the same support
+level as in legacy mode. See the <a class="reference internal" href="#ansi-mode">ANSI Mode</a> table below for per-pair support, and the
+<a class="reference internal" href="../../../expressions.html"><span class="std std-doc">Comet Supported Expressions Guide</span></a> for more information on the <code class="docutils literal notranslate"><span class="pre">allowIncompatible</span></code> setting.</p>
</section>
<section id="whitespace-trimming-in-casts-from-string">
<h2>Whitespace Trimming in Casts from String<a class="headerlink" href="#whitespace-trimming-in-casts-from-string" title="Link to this heading">#</a></h2>
@@ -942,20 +947,12 @@ the same result regardless of the session timezone.</p>
zero-padded to four digits (e.g. year 999 renders as <code class="docutils literal notranslate"><span class="pre">0999-01-01</span></code>). Years above 9999 are
rendered without truncation. The cast is timezone-independent.</p>
</section>
-<section id="id1">
-<h2>String to TimestampNTZ<a class="headerlink" href="#id1" title="Link to this heading">#</a></h2>
-<p>Comet’s native <code class="docutils literal notranslate"><span class="pre">CAST(string</span> <span class="pre">AS</span> <span class="pre">TIMESTAMP_NTZ)</span></code> implementation matches Apache Spark’s behavior.
-Unlike <code class="docutils literal notranslate"><span class="pre">CAST(string</span> <span class="pre">AS</span> <span class="pre">TIMESTAMP)</span></code>, this cast is timezone-independent: any timezone offset in
-the input string (e.g. <code class="docutils literal notranslate"><span class="pre">+08:00</span></code>, <code class="docutils literal notranslate"><span class="pre">Z</span></code>, <code class="docutils literal notranslate"><span class="pre">UTC</span></code>) is silently discarded, and the local date-time
-components are preserved as-is. Time-only strings (e.g. <code class="docutils literal notranslate"><span class="pre">T12:34:56</span></code>, <code class="docutils literal notranslate"><span class="pre">12:34</span></code>) produce <code class="docutils literal notranslate"><span class="pre">NULL</span></code>.
-The result is always a wall-clock timestamp with no timezone conversion or DST adjustment.</p>
-</section>
<section id="decimal-with-negative-scale-to-string">
<h2>Decimal with Negative Scale to String<a class="headerlink" href="#decimal-with-negative-scale-to-string" title="Link to this heading">#</a></h2>
<p>Casting a <code class="docutils literal notranslate"><span class="pre">DecimalType</span></code> with a negative scale to <code class="docutils literal notranslate"><span class="pre">StringType</span></code> is marked as incompatible when
<code class="docutils literal notranslate"><span class="pre">spark.sql.legacy.allowNegativeScaleOfDecimal</span></code> is <code class="docutils literal notranslate"><span class="pre">false</span></code> (the default). When that config is
-disabled, Spark cannot create negative-scale decimals, so Comet falls back to avoid running
-native execution on unexpected inputs.</p>
+disabled, Spark cannot create negative-scale decimals, so by default Comet runs Spark’s
+code-generated cast instead of the native one.</p>
<p>When <code class="docutils literal notranslate"><span class="pre">spark.sql.legacy.allowNegativeScaleOfDecimal=true</span></code>, the cast is compatible. Comet matches
Spark’s behavior of using Java <code class="docutils literal notranslate"><span class="pre">BigDecimal.toString()</span></code> semantics, which produces scientific
notation (e.g. a value of 12300 stored as <code class="docutils literal notranslate"><span class="pre">Decimal(7,-2)</span></code> with unscaled value 123 is rendered
@@ -1706,7 +1703,7 @@ as <code class="docutils literal notranslate"><span class="pre">"1.23E+4&qu
</div>
<nav class="bd-toc-nav page-toc" aria-labelledby="pst-page-navigation-heading-2">
<ul class="visible nav section-nav flex-column">
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#ansi-mode-fallback">ANSI Mode Fallback</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#ansi-mode-support">ANSI Mode Support</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#whitespace-trimming-in-casts-from-string">Whitespace Trimming in Casts from String</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#string-to-decimal">String to Decimal</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#string-to-date">String to Date</a></li>
@@ -1717,7 +1714,6 @@ as <code class="docutils literal notranslate"><span class="pre">"1.23E+4&qu
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#string-to-timestampntz">String to TimestampNTZ</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#timestampntz-casts">TimestampNTZ Casts</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#date-to-string">Date to String</a></li>
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#id1">String to TimestampNTZ</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#decimal-with-negative-scale-to-string">Decimal with Negative Scale to String</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#legacy-mode">Legacy Mode</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#try-mode">Try Mode</a></li>
diff --git a/user-guide/latest/compatibility/expressions/spark-4.1/map.html b/user-guide/latest/compatibility/expressions/spark-4.1/map.html
index ad06c9c774..7f01345c96 100644
--- a/user-guide/latest/compatibility/expressions/spark-4.1/map.html
+++ b/user-guide/latest/compatibility/expressions/spark-4.1/map.html
@@ -789,7 +789,7 @@ under the License.
keys, in <code class="docutils literal notranslate"><span class="pre">try_element_at</span></code>, and in other contexts where map ordering must be deterministic. Comet
runs <code class="docutils literal notranslate"><span class="pre">MapSort</span></code> natively, so map shuffle and group-by-on-map stay on Comet under Spark 4.0.</p>
<p>When <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.strictFloatingPoint=true</span></code>, <code class="docutils literal notranslate"><span class="pre">MapSort</span></code> falls back to Spark for maps whose
-keys contain <code class="docutils literal notranslate"><span class="pre">Float</span></code> or <code class="docutils literal notranslate"><span class="pre">Double</span></code> (consistent with <code class="docutils literal notranslate"><span class="pre">SortOrder</span></code> and <code class="docutils literal notranslate"><span class="pre">SortArray</span></code>). Arrow’s sort uses
+keys contain <code class="docutils literal notranslate"><span class="pre">Float</span></code> or <code class="docutils literal notranslate"><span class="pre">Double</span></code>. Arrow’s sort uses
IEEE total ordering for floating-point, which differs from Spark’s <code class="docutils literal notranslate"><span class="pre">Double.compare</span></code> semantics for
<code class="docutils literal notranslate"><span class="pre">NaN</span></code> and <code class="docutils literal notranslate"><span class="pre">-0.0</span></code>.</p>
<!--BEGIN:EXPR_COMPAT[map]-->
diff --git a/user-guide/latest/compatibility/index.html b/user-guide/latest/compatibility/index.html
index 0c4df4094e..e4fdd08d88 100644
--- a/user-guide/latest/compatibility/index.html
+++ b/user-guide/latest/compatibility/index.html
@@ -822,7 +822,8 @@ through the codegen dispatcher rather than running something incompatible.</p>
which document their per-expression configs and the specific differences to expect.</p>
<p>This is distinct from expressions that have <strong>no</strong> codegen-dispatch path: there, the
incompatible cases fall back to Spark by default, and <code class="docutils literal notranslate"><span class="pre">allowIncompatible=true</span></code> runs the native
-(incompatible) path instead. <code class="docutils literal notranslate"><span class="pre">cast</span></code> is the main example; see the
+(incompatible) path instead. Aggregate functions such as <code class="docutils literal notranslate"><span class="pre">mode</span></code> are the main example, because the
+codegen dispatcher covers only scalar expressions; see the
<a class="reference internal" href="../expressions.html"><span class="std std-doc">expression reference</span></a> for which expressions have incompatible cases.</p>
</section>
<section id="strings-with-non-utf-8-bytes">
@@ -863,12 +864,10 @@ Code that catches <code class="docutils literal notranslate"><span class="pre">S
inspects the exception class, <code class="docutils literal notranslate"><span class="pre">getCondition()</span></code>, or the parameterised error class will observe
divergence:</p>
<ul class="simple">
-<li><p>Byte / Short <code class="docutils literal notranslate"><span class="pre">Add</span></code>, <code class="docutils literal notranslate"><span class="pre">Subtract</span></code>, and <code class="docutils literal notranslate"><span class="pre">Multiply</span></code> overflow raises <code class="docutils literal notranslate"><span class="pre">ARITHMETIC_OVERFLOW</span></code> where Spark
-4.1 raises <code class="docutils literal notranslate"><span class="pre">BINARY_ARITHMETIC_OVERFLOW</span></code>, and Long overflow surfaces as <code class="docutils literal notranslate"><span class="pre">"integer</span> <span class="pre">overflow"</span></code>
-rather than <code class="docutils literal notranslate"><span class="pre">"long</span> <span class="pre">overflow"</span></code>. <code class="docutils literal notranslate"><span class="pre">Abs</span></code> uses Rust type names (<code class="docutils literal notranslate"><span class="pre">Int8</span></code>, <code class="docutils literal notranslate"><span class="pre">Int64</span></code>, …) in the message
-instead of Spark’s SQL type names, and the scalar path of <code class="docutils literal notranslate"><span class="pre">UnaryMinus</span></code> on Byte / Short emits a
-malformed message. The <code class="docutils literal notranslate"><span class="pre">try_</span></code> suggestion is omitted from all of these
-(<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/5071">#5071</a>).</p></li>
+<li><p>Byte / Short <code class="docutils literal notranslate"><span class="pre">Add</span></code>, <code class="docutils literal notranslate"><span class="pre">Subtract</span></code>, and <code class="docutils literal notranslate"><span class="pre">Multiply</span></code> overflow raises <code class="docutils literal notranslate"><span class="pre">ARITHMETIC_OVERFLOW</span></code> (for
+example <code class="docutils literal notranslate"><span class="pre">byte</span> <span class="pre">overflow</span></code>) where Spark raises <code class="docutils literal notranslate"><span class="pre">BINARY_ARITHMETIC_OVERFLOW</span></code>, and integral
+<code class="docutils literal notranslate"><span class="pre">ARITHMETIC_OVERFLOW</span></code> messages omit Spark’s <code class="docutils literal notranslate"><span class="pre">try_</span></code> suggestion
+(<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/6217">#6217</a>).</p></li>
<li><p>Wide-decimal arithmetic overflow, decimal divide-by-zero, and decimal-to-decimal cast overflow
raise raw Arrow errors that bypass <code class="docutils literal notranslate"><span class="pre">SparkErrorConverter</span></code> and surface as <code class="docutils literal notranslate"><span class="pre">CometNativeException</span></code>
rather than <code class="docutils literal notranslate"><span class="pre">SparkArithmeticException</span></code> with the proper error class and query context
@@ -883,18 +882,11 @@ not yet track (<a class="reference external" href="https://github.com/apache/dat
Most also have entries in the per-category expression pages linked above; they are collected here
so users hunting an unexpected value have a single place to check:</p>
<ul class="simple">
-<li><p><code class="docutils literal notranslate"><span class="pre">CAST(boolean</span> <span class="pre">AS</span> <span class="pre">DECIMAL(p,</span> <span class="pre">s))</span></code> where <code class="docutils literal notranslate"><span class="pre">10^s</span></code> exceeds the target precision (e.g.
-<code class="docutils literal notranslate"><span class="pre">DECIMAL(1,</span> <span class="pre">1)</span></code>) throws <code class="docutils literal notranslate"><span class="pre">NUMERIC_VALUE_OUT_OF_RANGE</span></code> regardless of the eval mode. Spark returns
-<code class="docutils literal notranslate"><span class="pre">NULL</span></code> under legacy and try mode, and only throws under ANSI
-(<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/5068">#5068</a>).</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">CAST(string</span> <span class="pre">AS</span> <span class="pre">timestamp)</span></code> and <code class="docutils literal notranslate"><span class="pre">CAST(string</span> <span class="pre">AS</span> <span class="pre">timestamp_ntz)</span></code> trim Unicode whitespace.
Spark trims only the bytes <code class="docutils literal notranslate"><span class="pre">0x00</span></code>-<code class="docutils literal notranslate"><span class="pre">0x20</span></code> and <code class="docutils literal notranslate"><span class="pre">0x7F</span></code>, so a value padded with an ASCII control byte
parses in Spark and returns <code class="docutils literal notranslate"><span class="pre">NULL</span></code> in Comet, while a value padded with non-ASCII whitespace such
as <code class="docutils literal notranslate"><span class="pre">U+3000</span></code> returns <code class="docutils literal notranslate"><span class="pre">NULL</span></code> in Spark and parses in Comet
(<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/5149">#5149</a>).</p></li>
-<li><p><strong>Explicit positive timestamp years:</strong> Spark accepts strings such as <code class="docutils literal notranslate"><span class="pre">+7528</span></code> as the start
-of that year, while Comet’s native string-to-timestamp cast returns NULL in non-ANSI mode
-(<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/5716">#5716</a>).</p></li>
<li><p>Native <code class="docutils literal notranslate"><span class="pre">RANGE</span></code> window frames with an explicit <code class="docutils literal notranslate"><span class="pre">PRECEDING</span></code> / <code class="docutils literal notranslate"><span class="pre">FOLLOWING</span></code> offset diverge from
Spark when the boundary arithmetic overflows for <code class="docutils literal notranslate"><span class="pre">DATE</span></code> or <code class="docutils literal notranslate"><span class="pre">DECIMAL</span></code> <code class="docutils literal notranslate"><span class="pre">ORDER</span> <span class="pre">BY</span></code> columns
(<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/5022">#5022</a>).</p></li>
@@ -911,16 +903,6 @@ when Spark abandons codegen at runtime, after a compile failure under
returns <code class="docutils literal notranslate"><span class="pre">NULL</span></code> (or raises under ANSI) in Spark but the recovered value in Comet.</p></li>
</ul>
</section>
-<section id="object-store-cache">
-<h2>Object store cache<a class="headerlink" href="#object-store-cache" title="Link to this heading">#</a></h2>
-<p>When Comet’s native scan reads Parquet files, it caches one object store instance per
-<code class="docutils literal notranslate"><span class="pre">(scheme</span> <span class="pre">+</span> <span class="pre">host</span> <span class="pre">+</span> <span class="pre">port,</span> <span class="pre">hadoop-config-hash)</span></code> key. For <code class="docutils literal notranslate"><span class="pre">abfss://container@account.dfs.core.windows.net/...</span></code>
-URLs, the container lives in the URL userinfo, not the host, so two containers in one storage
-account currently collide on the same cache entry. Within a single executor process, reading from
-a second container after a first can be served by the first container’s store instance and return
-its data. S3, GCS, and HDFS are unaffected because their bucket / host lives in the URL host
-component. Tracked by <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4993">#4993</a>.</p>
-</section>
</section>
@@ -975,7 +957,6 @@ component. Tracked by <a class="reference external" href="https://github.com/apa
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#strings-with-non-utf-8-bytes">Strings with non-UTF-8 bytes</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#ansi-mode-error-classes-and-messages">ANSI-mode error classes and messages</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#known-result-value-divergences">Known result-value divergences</a></li>
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#object-store-cache">Object store cache</a></li>
</ul>
</nav></div>
diff --git a/user-guide/latest/compatibility/operators.html b/user-guide/latest/compatibility/operators.html
index cfa5e196db..2ccb709f2d 100644
--- a/user-guide/latest/compatibility/operators.html
+++ b/user-guide/latest/compatibility/operators.html
@@ -789,6 +789,17 @@ operator restrictions and aggregate buffer compatibility checks still apply.
Parquet writes whose input plans contain an empty relation use Spark’s writer to preserve
readable empty output files and their schema metadata.</p>
</section>
+<section id="in-memory-cache">
+<h2>In-Memory Cache<a class="headerlink" href="#in-memory-cache" title="Link to this heading">#</a></h2>
+<p>Comet can store cached relations (<code class="docutils literal notranslate"><span class="pre">df.cache()</span></code>, <code class="docutils literal notranslate"><span class="pre">CACHE</span> <span class="pre">TABLE</span></code>) in Arrow format and scan them
+natively. This is experimental and disabled by default; see <a class="reference internal" href="../in-memory-cache.html"><span class="std std-doc">In-Memory Cache</span></a>
+for how to enable it. Comet does not replace a <code class="docutils literal notranslate"><span class="pre">spark.sql.cache.serializer</span></code> that the application
+has already set. Relations whose schema Comet’s Arrow writer does not support are cached in
+Spark’s default format, and their scans fall back to Spark. Reads that feed Spark operators rather
+than Comet operators can be slower than Spark’s cache.</p>
+<p>With Kryo and <code class="docutils literal notranslate"><span class="pre">spark.kryo.registrationRequired=true</span></code>, Comet needs its Kryo registrator whether or
+not the cache is enabled; see <a class="reference internal" href="../installation.html#kryo-serialization"><span class="std std-ref">Kryo serialization</span></a>.</p>
+</section>
<section id="sampling">
<h2>Sampling<a class="headerlink" href="#sampling" title="Link to this heading">#</a></h2>
<p>Comet runs <code class="docutils literal notranslate"><span class="pre">SampleExec</span></code> natively when sampling is performed without replacement, which covers
@@ -848,8 +859,11 @@ comparing Arrow row-encoded keys for byte equality, which splits peers that Spar
</ul>
<p><strong>Known incompatibilities:</strong></p>
<ul class="simple">
-<li><p>Signed-zero ordering (<code class="docutils literal notranslate"><span class="pre">-0.0</span></code> vs <code class="docutils literal notranslate"><span class="pre">+0.0</span></code>) diverges from Spark’s <code class="docutils literal notranslate"><span class="pre">RankLimitIterator</span></code>; see
-<a class="reference internal" href="floating-point.html#ordering-signed-zero-00-vs-00"><span class="std std-ref">floating-point ordering</span></a>.</p></li>
+<li><p>Floating-point values nested in array or struct <code class="docutils literal notranslate"><span class="pre">ORDER</span> <span class="pre">BY</span></code> keys are compared with Arrow’s raw
+total ordering, so ranks can differ from Spark when the data mixes <code class="docutils literal notranslate"><span class="pre">-0.0</span></code> and <code class="docutils literal notranslate"><span class="pre">+0.0</span></code> or more
+than one NaN representation (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/5507">#5507</a>).
+Scalar <code class="docutils literal notranslate"><span class="pre">FLOAT</span></code> and <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> keys are normalized and match Spark; see
+<a class="reference internal" href="floating-point.html"><span class="std std-doc">floating-point ordering</span></a>.</p></li>
</ul>
</section>
<section id="round-robin-partitioning">
@@ -924,6 +938,7 @@ they will be identical to Spark. Unsorted results may have different row orderin
<nav class="bd-toc-nav page-toc" aria-labelledby="pst-page-navigation-heading-2">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#empty-relations">Empty Relations</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#in-memory-cache">In-Memory Cache</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#sampling">Sampling</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#window-functions">Window Functions</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#round-robin-partitioning">Round-Robin Partitioning</a></li>
diff --git a/user-guide/latest/compatibility/scans.html b/user-guide/latest/compatibility/scans.html
index d326b576d5..b0973ebc7e 100644
--- a/user-guide/latest/compatibility/scans.html
+++ b/user-guide/latest/compatibility/scans.html
@@ -805,6 +805,8 @@ Spark’s parquet-mr semantics (silent overflow, null-on-narrowing), which Comet
does not replicate. By default Comet falls back to Spark in this case. Set
<code class="docutils literal notranslate"><span class="pre">spark.comet.scan.allowDisabledParquetVectorizedReader=true</span></code> to opt in to running the
Comet Parquet scan regardless.</p></li>
+<li><p>A read schema that repeats a Parquet field id, at the top level or within a struct, when
+<code class="docutils literal notranslate"><span class="pre">spark.sql.parquet.fieldId.read.enabled=true</span></code>.</p></li>
</ul>
<p>The following limitation may produce incorrect results without falling back to Spark:</p>
<ul class="simple">
diff --git a/user-guide/latest/compatibility/spark-versions.html b/user-guide/latest/compatibility/spark-versions.html
index 9d5756cb24..bb03ce23e3 100644
--- a/user-guide/latest/compatibility/spark-versions.html
+++ b/user-guide/latest/compatibility/spark-versions.html
@@ -789,20 +789,17 @@ of the versioning policy.</p>
<p class="admonition-title">Warning</p>
<p>Spark 3.4 support is deprecated as of the 1.0.0 release and will be removed in a future release.
Comet continues to build and publish Spark 3.4 binaries in the meantime, but Apache Spark’s own
-SQL test suite is no longer run against Spark 3.4 on every change: it runs only when a contributor
+SQL test suite no longer runs against Spark 3.4 automatically: it runs only when a contributor
opts a pull request into it. Regressions specific to Spark 3.4 are therefore more likely to reach a
release than on the other supported versions. We recommend moving to Spark 3.5 or later.</p>
</div>
<section id="known-limitations">
<h3>Known Limitations<a class="headerlink" href="#known-limitations" title="Link to this heading">#</a></h3>
<ul class="simple">
-<li><p><strong>Reading <code class="docutils literal notranslate"><span class="pre">TimestampLTZ</span></code> as <code class="docutils literal notranslate"><span class="pre">TimestampNTZ</span></code></strong>: Spark 3.4 raises an error for this operation
-(SPARK-36182), but Comet’s Parquet scan silently returns the raw UTC value instead.
-See <a class="reference internal" href="scans.html#parquet-scan-limitations"><span class="std std-ref">Parquet Compatibility</span></a> for details.</p></li>
-<li><p><strong>Unsupported Parquet type conversions</strong>: Spark 3.4 raises schema incompatibility errors for
-certain type mismatches (e.g., reading INT32 as BIGINT, decimal precision changes), but Comet’s
-Comet’s Parquet scan may not detect these and could return unexpected values.
-See <a class="reference internal" href="scans.html#parquet-scan-limitations"><span class="std std-ref">Parquet Compatibility</span></a> for details.</p></li>
+<li><p><strong>Extra <code class="docutils literal notranslate"><span class="pre">SparkException</span></code> layer in Parquet schema mismatch errors</strong>: when a Parquet read is
+rejected because a file’s type cannot be converted to the requested type, the error’s cause chain
+has one more <code class="docutils literal notranslate"><span class="pre">SparkException</span></code> layer than Spark’s own reader produces.
+See <a class="reference internal" href="scans.html#schema-mismatch-handling"><span class="std std-ref">Parquet Compatibility</span></a> for details.</p></li>
</ul>
</section>
</section>
@@ -812,19 +809,16 @@ See <a class="reference internal" href="scans.html#parquet-scan-limitations"><sp
<section id="id1">
<h3>Known Limitations<a class="headerlink" href="#id1" title="Link to this heading">#</a></h3>
<ul class="simple">
-<li><p><strong>Reading <code class="docutils literal notranslate"><span class="pre">TimestampLTZ</span></code> as <code class="docutils literal notranslate"><span class="pre">TimestampNTZ</span></code></strong>: Spark 3.5 raises an error for this operation
-(SPARK-36182), but Comet’s Parquet scan silently returns the raw UTC value instead.
-See <a class="reference internal" href="scans.html#parquet-scan-limitations"><span class="std std-ref">Parquet Compatibility</span></a> for details.</p></li>
-<li><p><strong>Unsupported Parquet type conversions</strong>: Spark 3.5 raises schema incompatibility errors for
-certain type mismatches (e.g., reading INT32 as BIGINT, decimal precision changes), but Comet’s
-Comet’s Parquet scan may not detect these and could return unexpected values.
-See <a class="reference internal" href="scans.html#parquet-scan-limitations"><span class="std std-ref">Parquet Compatibility</span></a> for details.</p></li>
+<li><p><strong>Extra <code class="docutils literal notranslate"><span class="pre">SparkException</span></code> layer in Parquet schema mismatch errors</strong>: when a Parquet read is
+rejected because a file’s type cannot be converted to the requested type, the error’s cause chain
+has one more <code class="docutils literal notranslate"><span class="pre">SparkException</span></code> layer than Spark’s own reader produces.
+See <a class="reference internal" href="scans.html#schema-mismatch-handling"><span class="std std-ref">Parquet Compatibility</span></a> for details.</p></li>
</ul>
</section>
</section>
<section id="spark-4-0">
<h2>Spark 4.0<a class="headerlink" href="#spark-4-0" title="Link to this heading">#</a></h2>
-<p>Spark 4.0.4 is supported with Java 17 and Scala 2.13.</p>
+<p>Spark 4.0.4 is supported with Java 17/21 and Scala 2.13.</p>
<section id="id2">
<h3>Known Limitations<a class="headerlink" href="#id2" title="Link to this heading">#</a></h3>
<ul class="simple">
@@ -855,8 +849,9 @@ persisting, or read the file with Comet disabled for that query.</p></li>
<p>Spark 4.2.0 is provided as experimental support with Java 17 and Scala 2.13.</p>
<div class="admonition warning">
<p class="admonition-title">Warning</p>
-<p>Spark 4.2 support is experimental. Comet tests run in CI for this version, but the Spark SQL tests
-do not yet. It is intended for early evaluation only and should not be used in production.</p>
+<p>Spark 4.2 support is experimental. Comet’s own test suites run nightly for this version, but
+Apache Spark’s SQL test suite does not run against it yet. It is intended for early evaluation only
+and should not be used in production.</p>
</div>
</section>
</section>
diff --git a/user-guide/latest/datasources.html b/user-guide/latest/datasources.html
index af7e7b1593..23ff1a3bee 100644
--- a/user-guide/latest/datasources.html
+++ b/user-guide/latest/datasources.html
@@ -787,13 +787,15 @@ Arrow format, allowing the Comet pipeline to take over after that, but the proce
</section>
<section id="apache-iceberg">
<h3>Apache Iceberg<a class="headerlink" href="#apache-iceberg" title="Link to this heading">#</a></h3>
-<p>Comet accelerates Iceberg scans of Parquet files. See the <a class="reference internal" href="iceberg.html"><span class="std std-doc">Iceberg Guide</span></a> for more information.</p>
+<p>Comet accelerates Iceberg scans of Parquet files and has an experimental, opt-in native Iceberg writer.
+See the <a class="reference internal" href="iceberg.html"><span class="std std-doc">Iceberg Guide</span></a> and <a class="reference internal" href="iceberg-writes.html"><span class="std std-doc">Iceberg Writes</span></a> for more information.</p>
</section>
<section id="csv">
<h3>CSV<a class="headerlink" href="#csv" title="Link to this heading">#</a></h3>
<p>Comet provides experimental Rust-based CSV scan support. When <code class="docutils literal notranslate"><span class="pre">spark.comet.scan.csv.v2.enabled</span></code> is enabled, CSV files
are read in Rust for improved performance. This feature is experimental and performance benefits are
-workload-dependent.</p>
+workload-dependent. Only Spark’s DataSource V2 CSV scan is accelerated, and Spark reads CSV through the V1 API by
+default, so also remove <code class="docutils literal notranslate"><span class="pre">csv</span></code> from <code class="docutils literal notranslate"><span class="pre">spark.sql.sources.useV1SourceList</span></code>.</p>
<p>Alternatively, when <code class="docutils literal notranslate"><span class="pre">spark.comet.convert.csv.enabled</span></code> is enabled, data from Spark’s CSV reader is immediately
converted into Arrow format, allowing the Comet pipeline to take over after that.</p>
</section>
@@ -1108,6 +1110,9 @@ described in <a class="reference internal" href="#additional-s3-configuration-op
credential providers and options documented above also apply to alias-scheme URLs. The same
translation feeds the native Iceberg scan; see
<a class="reference internal" href="iceberg.html#object-store-configuration-s3"><span class="std std-ref">Object store configuration (S3)</span></a> in the Iceberg guide.</p>
+<p>A native Parquet scan whose alias-scheme paths span more than one bucket falls back to Spark. Alias
+schemes apply to native scans only: a native Iceberg write to an alias-scheme location falls back to
+iceberg-java.</p>
</section>
<section id="examples">
<h3>Examples<a class="headerlink" href="#examples" title="Link to this heading">#</a></h3>
@@ -1138,7 +1143,7 @@ translation feeds the native Iceberg scan; see
<p>Comet’s S3 support has the following limitations:</p>
<ol class="arabic simple">
<li><p><strong>Partial Hadoop S3A configuration support</strong>: Not all Hadoop S3A configurations are currently supported. Only the configurations listed in the tables above are translated and applied to the underlying <code class="docutils literal notranslate"><span class="pre">object_store</span></code> crate.</p></li>
-<li><p><strong>Custom credential providers</strong>: Custom implementations of AWS credential providers are not supported. The implementation only supports the standard credential providers listed in the table above. We are planning to add support for custom credential providers through a JNI-based adapter that will allow calling Java credential providers from Rust code. See <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/1829">#1829</a> for more det [...]
+<li><p><strong>Custom credential providers</strong>: Custom credential provider classes named in <code class="docutils literal notranslate"><span class="pre">fs.s3a.aws.credentials.provider</span></code> are not supported; only the standard providers listed in the table above are. To route credential requests through your own Java code, implement Comet’s <code class="docutils literal notranslate"><span class="pre">CometS3CredentialProvider</span></code> SPI; see <a class="reference inter [...]
</ol>
</section>
</section>
diff --git a/user-guide/latest/datatypes.html b/user-guide/latest/datatypes.html
index 73c8f061c5..a79d98d055 100644
--- a/user-guide/latest/datatypes.html
+++ b/user-guide/latest/datatypes.html
@@ -927,7 +927,7 @@ the tables below and may be reconsidered based on demand:</p>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">TimeType</span></code></p></td>
<td><p>⚠️</p></td>
-<td><p>Spark 4.1+. Native serialization is in place; some operators (sort, shuffle, min/max) are still being wired up (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4288">#4288</a>).</p></td>
+<td><p>Spark 4.1+. Native serialization is in place; some operators (sort, min/max) are still being wired up (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4288">#4288</a>).</p></td>
</tr>
</tbody>
</table>
@@ -977,7 +977,7 @@ functions. Remaining work is tracked by
<tbody>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">StructType</span></code></p></td>
<td><p>✅</p></td>
-<td><p>Empty structs (no fields) fall back.</p></td>
+<td><p>Empty structs (no fields) and structs with duplicate field names fall back.</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">ArrayType</span></code></p></td>
<td><p>✅</p></td>
@@ -1014,8 +1014,9 @@ functions. Remaining work is tracked by
Parquet timestamp inference settings. Support for Spark’s whole-value pushdown rewrite is tracked
by <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/5519">#5519</a>. Nested Variant columns, pushed-down
Variant field extraction, expressions, writes, shuffle and spill, Python operators, encrypted
-files, and Iceberg scans fall back to Spark. Spark also handles columnar-to-row conversion of
-the native scan output and strict reads with <code class="docutils literal notranslate"><span class="pre">allowReadingShredded=false</span></code>. Broader
+files, and Iceberg scans that read a Variant column fall back to Spark. Iceberg scans of tables
+whose Variant columns the query does not read run natively. Spark also handles columnar-to-row
+conversion of the native scan output and strict reads with <code class="docutils literal notranslate"><span class="pre">allowReadingShredded=false</span></code>. Broader
support is tracked by <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4295">#4295</a> and
<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/3983">#3983</a>.</p>
<p>Shredded reconstruction can be slower than Spark’s reader; see the
diff --git a/user-guide/latest/expressions.html b/user-guide/latest/expressions.html
index 411a937b64..1b98d77978 100644
--- a/user-guide/latest/expressions.html
+++ b/user-guide/latest/expressions.html
@@ -786,9 +786,9 @@ transparently falls back to Spark for that part of the plan; results are unaffec
Those cases must be opted into per expression with
<code class="docutils literal notranslate"><span class="pre">spark.comet.expression.EXPRNAME.allowIncompatible=true</span></code> (where <code class="docutils literal notranslate"><span class="pre">EXPRNAME</span></code> is the Spark
expression class name, for example <code class="docutils literal notranslate"><span class="pre">Cast</span></code>). There is no global opt-in. By default such a case
-either falls back to Spark (for example <code class="docutils literal notranslate"><span class="pre">cast</span></code>) or, when the expression has a Spark-compatible
-codegen-dispatch implementation, runs through that instead (for example the regex and JSON
-families). See <a class="reference internal" href="compatibility/index.html#native-and-codegen-dispatch-implementations"><span class="std std-ref">Native and codegen-dispatch implementations</span></a>
+either falls back to Spark (for example the <code class="docutils literal notranslate"><span class="pre">mode</span></code> aggregate) or, when the expression has a
+Spark-compatible codegen-dispatch implementation, runs through that instead (for example <code class="docutils literal notranslate"><span class="pre">cast</span></code>
+and the regex and JSON families). See <a class="reference internal" href="compatibility/index.html#native-and-codegen-dispatch-implementations"><span class="std std-ref">Native and codegen-dispatch implementations</span></a>
for how Comet chooses.</p>
<p>Most expressions can also be disabled with <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.EXPRNAME.enabled=false</span></code>, where
<code class="docutils literal notranslate"><span class="pre">EXPRNAME</span></code> is the Spark expression class name (for example <code class="docutils literal notranslate"><span class="pre">Length</span></code> or <code class="docutils literal notranslate"><span class="pre">StartsWith</span></code>). See the
@@ -842,17 +842,17 @@ for how Comet chooses.</p>
<section id="not-currently-planned">
<h2>Not currently planned<a class="headerlink" href="#not-currently-planned" title="Link to this heading">#</a></h2>
<p>Comet focuses acceleration on mainstream relational, string, datetime, math, and collection
-expressions. The following function families are <strong>not currently planned</strong> for native acceleration (they are not on the 1.0 roadmap): specialized functionality with narrow real-world analytics use and high implementation cost. They fall back to Spark and may be reconsidered based on demand:</p>
+expressions. The following function families are <strong>not currently planned</strong> for native acceleration (they are not on the current roadmap): specialized functionality with narrow real-world analytics use and high implementation cost. They fall back to Spark and may be reconsidered based on demand:</p>
<ul class="simple">
<li><p><strong>Probabilistic sketches and approximate top-k</strong> (<code class="docutils literal notranslate"><span class="pre">kll_sketch_*</span></code>, <code class="docutils literal notranslate"><span class="pre">hll_*</span></code>, <code class="docutils literal notranslate"><span class="pre">theta_*</span></code>, <code class="docutils literal notranslate"><span class="pre">count_min_sketch</span></code>, <code class="docutils literal notranslate"><span class="pre">bitmap_*</spa [...]
<li><p><strong>Geospatial</strong> (<code class="docutils literal notranslate"><span class="pre">st_*</span></code>): brand-new Spark 4.1 functionality, specialized.</p></li>
<li><p><strong>Avro / Protobuf codecs</strong> (<code class="docutils literal notranslate"><span class="pre">from_avro</span></code>, <code class="docutils literal notranslate"><span class="pre">to_avro</span></code>, <code class="docutils literal notranslate"><span class="pre">from_protobuf</span></code>, <code class="docutils literal notranslate"><span class="pre">to_protobuf</span></code>, <code class="docutils literal notranslate"><span class="pre">schema_of_avro</span></code>): form [...]
-<li><p><strong>JVM reflection</strong> (<code class="docutils literal notranslate"><span class="pre">java_method</span></code>, <code class="docutils literal notranslate"><span class="pre">reflect</span></code>): niche, and they invoke arbitrary JVM methods (a security concern).</p></li>
+<li><p><strong>JVM reflection</strong> (<code class="docutils literal notranslate"><span class="pre">java_method</span></code>, <code class="docutils literal notranslate"><span class="pre">reflect</span></code>, <code class="docutils literal notranslate"><span class="pre">try_reflect</span></code>): niche, and they invoke arbitrary JVM methods (a security concern).</p></li>
<li><p><strong>UTF-8 validation</strong> (<code class="docutils literal notranslate"><span class="pre">is_valid_utf8</span></code>, <code class="docutils literal notranslate"><span class="pre">make_valid_utf8</span></code>, <code class="docutils literal notranslate"><span class="pre">validate_utf8</span></code>, <code class="docutils literal notranslate"><span class="pre">try_validate_utf8</span></code>): niche Spark 4.x string-validation helpers.</p></li>
<li><p><strong>Miscellaneous niche</strong> (<code class="docutils literal notranslate"><span class="pre">histogram_numeric</span></code>, <code class="docutils literal notranslate"><span class="pre">version</span></code>, <code class="docutils literal notranslate"><span class="pre">sentences</span></code>, <code class="docutils literal notranslate"><span class="pre">quote</span></code>): low-value or specialized functions with little benefit from native acceleration.</p></li>
</ul>
<p>The file-metadata functions <code class="docutils literal notranslate"><span class="pre">input_file_name</span></code>, <code class="docutils literal notranslate"><span class="pre">input_file_block_start</span></code>, and <code class="docutils literal notranslate"><span class="pre">input_file_block_length</span></code> depend on scan-internal per-row file information rather than the expression layer; their support status is covered in the <a class="reference internal" href="compatibi [...]
-<p>Note that <code class="docutils literal notranslate"><span class="pre">median</span></code> and <code class="docutils literal notranslate"><span class="pre">mode</span></code> are planned: they are mainstream exact aggregates. <code class="docutils literal notranslate"><span class="pre">approx_count_distinct</span></code> is supported because Comet ports Spark’s <code class="docutils literal notranslate"><span class="pre">HyperLogLogPlusPlus</span></code> exactly, so its result is bit [...]
+<p>Note that <code class="docutils literal notranslate"><span class="pre">median</span></code> and <code class="docutils literal notranslate"><span class="pre">mode</span></code> are supported: they are mainstream exact aggregates. <code class="docutils literal notranslate"><span class="pre">mode</span></code> runs natively only with <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.Mode.allowIncompatible=true</span></code>. <code class="docutils literal [...]
<p>The tables below list every Spark built-in expression with its current status.</p>
</section>
<section id="agg-funcs">
@@ -1034,129 +1034,134 @@ expressions. The following function families are <strong>not currently planned</
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">mode</span></code></p></td>
<td><p>✅</p></td>
<td><p>—</p></td>
-<td><p><code class="docutils literal notranslate"><span class="pre">mode(col)</span></code> only; Spark breaks ties non-deterministically, so Comet returns the smallest tied value and falls back by default, opt-in via allowIncompatible (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/3970">#3970</a>)</p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">mode(col)</span></code> only; Spark breaks ties non-deterministically, so Comet returns the smallest tied value and falls back by default, opt-in via allowIncompatible</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">percentile</span></code></p></td>
<td><p>✅</p></td>
<td><p>Native</p></td>
<td><p>Single literal percentage on numeric input runs natively; array of percentages and a frequency argument fall back to Spark</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">percentile_cont</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">percentile_approx</span></code></p></td>
+<td><p>✅</p></td>
+<td><p>Native</p></td>
+<td><p>Alias of <code class="docutils literal notranslate"><span class="pre">approx_percentile</span></code>; same restrictions apply</p></td>
+</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">percentile_cont</span></code></p></td>
<td><p>✅</p></td>
<td><p>—</p></td>
<td><p>Spark 4.0+ <code class="docutils literal notranslate"><span class="pre">WITHIN</span> <span class="pre">GROUP</span> <span class="pre">(ORDER</span> <span class="pre">BY</span> <span class="pre">...)</span></code>; ascending only runs natively, <code class="docutils literal notranslate"><span class="pre">DESC</span></code> falls back to Spark</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">percentile_disc</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">percentile_disc</span></code></p></td>
<td><p>🔜</p></td>
<td><p>—</p></td>
<td><p>Percentile aggregate</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">regr_avgx</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">regr_avgx</span></code></p></td>
<td><p>✅</p></td>
<td><p>—</p></td>
<td><p>Native: Spark rewrites to <code class="docutils literal notranslate"><span class="pre">Average</span></code> (tests in <a class="reference external" href="https://github.com/apache/datafusion-comet/pull/4551">#4551</a>)</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">regr_avgy</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">regr_avgy</span></code></p></td>
<td><p>✅</p></td>
<td><p>—</p></td>
<td><p>Native: Spark rewrites to <code class="docutils literal notranslate"><span class="pre">Average</span></code> (tests in <a class="reference external" href="https://github.com/apache/datafusion-comet/pull/4551">#4551</a>)</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">regr_count</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">regr_count</span></code></p></td>
<td><p>✅</p></td>
<td><p>—</p></td>
<td><p>Native: Spark rewrites to <code class="docutils literal notranslate"><span class="pre">Count</span></code> (tests in <a class="reference external" href="https://github.com/apache/datafusion-comet/pull/4551">#4551</a>)</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">regr_intercept</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">regr_intercept</span></code></p></td>
<td><p>✅</p></td>
<td><p>Native</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">regr_r2</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">regr_r2</span></code></p></td>
<td><p>✅</p></td>
<td><p>Native</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">regr_slope</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">regr_slope</span></code></p></td>
<td><p>✅</p></td>
<td><p>Native</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">regr_sxx</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">regr_sxx</span></code></p></td>
<td><p>✅</p></td>
<td><p>—</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">regr_sxy</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">regr_sxy</span></code></p></td>
<td><p>✅</p></td>
<td><p>Native</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">regr_syy</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">regr_syy</span></code></p></td>
<td><p>✅</p></td>
<td><p>—</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">skewness</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">skewness</span></code></p></td>
<td><p>🔜</p></td>
<td><p>—</p></td>
<td><p>Not yet implemented natively</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">some</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">some</span></code></p></td>
<td><p>✅</p></td>
<td><p>—</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">std</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">std</span></code></p></td>
<td><p>✅</p></td>
<td><p>Native</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">stddev</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">stddev</span></code></p></td>
<td><p>✅</p></td>
<td><p>Native</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">stddev_pop</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">stddev_pop</span></code></p></td>
<td><p>✅</p></td>
<td><p>Native</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">stddev_samp</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">stddev_samp</span></code></p></td>
<td><p>✅</p></td>
<td><p>Native</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">string_agg</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">string_agg</span></code></p></td>
<td><p>✅</p></td>
<td><p>Native</p></td>
<td><p>Alias of <code class="docutils literal notranslate"><span class="pre">listagg</span></code>; same restrictions apply.</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">sum</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">sum</span></code></p></td>
<td><p>✅</p></td>
<td><p>Native</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">try_avg</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">try_avg</span></code></p></td>
<td><p>✅</p></td>
<td><p>—</p></td>
<td><p>Interval types fall back</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">try_sum</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">try_sum</span></code></p></td>
<td><p>✅</p></td>
<td><p>—</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">var_pop</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">var_pop</span></code></p></td>
<td><p>✅</p></td>
<td><p>Native</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">var_samp</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">var_samp</span></code></p></td>
<td><p>✅</p></td>
<td><p>Native</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">variance</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">variance</span></code></p></td>
<td><p>✅</p></td>
<td><p>Native</p></td>
<td><p></p></td>
@@ -1195,7 +1200,7 @@ expressions. The following function families are <strong>not currently planned</
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">array_contains</span></code></p></td>
<td><p>✅</p></td>
<td><p>Hybrid</p></td>
-<td><p>NaN/signed-zero handling may differ (<a class="reference internal" href="compatibility/floating-point.html"><span class="std std-doc">details</span></a>)</p></td>
+<td><p>Float/double element arrays route through the JVM codegen dispatcher by default; the native path is opt-in via allowIncompatible</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">array_distinct</span></code></p></td>
<td><p>✅</p></td>
@@ -1230,7 +1235,7 @@ expressions. The following function families are <strong>not currently planned</
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">array_min</span></code></p></td>
<td><p>✅</p></td>
<td><p>Native</p></td>
-<td><p>NaN ordering may differ (<a class="reference internal" href="compatibility/floating-point.html"><span class="std std-doc">details</span></a>)</p></td>
+<td><p>May return <code class="docutils literal notranslate"><span class="pre">-0.0</span></code> where Spark returns <code class="docutils literal notranslate"><span class="pre">0.0</span></code> when both zeros are present</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">array_position</span></code></p></td>
<td><p>✅</p></td>
@@ -1300,7 +1305,7 @@ expressions. The following function families are <strong>not currently planned</
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">sort_array</span></code></p></td>
<td><p>✅</p></td>
<td><p>Hybrid</p></td>
-<td><p>Nested struct/null arrays fall back</p></td>
+<td><p>Struct, nested-array, and null elements run natively; other element types (for example intervals), and floating-point elements when <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.strictFloatingPoint=true</span></code>, route through the JVM codegen dispatcher</p></td>
</tr>
</tbody>
</table>
@@ -1407,12 +1412,12 @@ expressions. The following function families are <strong>not currently planned</
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">concat</span></code></p></td>
<td><p>✅</p></td>
<td><p>Hybrid</p></td>
-<td><p>Binary/array children fall back</p></td>
+<td><p>Binary/array children and non-UTF8_BINARY collations route through the JVM codegen dispatcher</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">reverse</span></code></p></td>
<td><p>✅</p></td>
<td><p>Hybrid</p></td>
-<td><p>Binary-element arrays fall back (Incompatible) (<a class="reference internal" href="#compatibility/expressions/array.md"><span class="xref myst">details</span></a>)</p></td>
+<td><p>Arrays with binary, struct, or map elements, and collated strings, route through the JVM codegen dispatcher (<a class="reference internal" href="#compatibility/expressions/array.md"><span class="xref myst">details</span></a>)</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">size</span></code></p></td>
<td><p>✅</p></td>
@@ -1505,7 +1510,7 @@ expressions. The following function families are <strong>not currently planned</
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">cast</span></code></p></td>
<td><p>✅</p></td>
<td><p>Hybrid</p></td>
-<td><p>Some casts fall back; float-to-decimal is opt-in (<a class="reference internal" href="#compatibility/expressions/cast.md"><span class="xref myst">details</span></a>)</p></td>
+<td><p>Casts without a native path (for example boolean to decimal) route through the JVM codegen dispatcher (<a class="reference internal" href="#compatibility/expressions/cast.md"><span class="xref myst">details</span></a>)</p></td>
</tr>
</tbody>
</table>
@@ -1710,9 +1715,9 @@ expressions. The following function families are <strong>not currently planned</
<td><p>Routes through the JVM codegen dispatcher by default; intervals outside Arrow’s nanosecond range are tracked by <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/5279">#5279</a>; the native path is opt-in via allowIncompatible (<a class="reference internal" href="#compatibility/expressions/datetime.md"><span class="xref myst">details</span></a>)</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">make_time</span></code></p></td>
-<td><p>🔜</p></td>
+<td><p>✅</p></td>
<td><p>—</p></td>
-<td><p>Spark 4.1 TIME type; tracked by <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4288">#4288</a></p></td>
+<td><p>Spark 4.1+; requires <code class="docutils literal notranslate"><span class="pre">spark.sql.timeType.enabled=true</span></code>, which Spark leaves off by default. Runs natively; remaining TIME type work is tracked by <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4288">#4288</a></p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">make_timestamp</span></code></p></td>
<td><p>✅</p></td>
@@ -1825,9 +1830,9 @@ expressions. The following function families are <strong>not currently planned</
<td><p>Rewrites to <code class="docutils literal notranslate"><span class="pre">Cast</span></code> (or <code class="docutils literal notranslate"><span class="pre">Cast(GetTimestamp)</span></code> with a format) before Comet sees the plan</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">to_time</span></code></p></td>
-<td><p>🔜</p></td>
+<td><p>✅</p></td>
<td><p>—</p></td>
-<td><p>Spark 4.1 TIME type; tracked by <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4288">#4288</a></p></td>
+<td><p>Spark 4.1+; requires <code class="docutils literal notranslate"><span class="pre">spark.sql.timeType.enabled=true</span></code>, which Spark leaves off by default. The one-argument form runs natively</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">to_timestamp</span></code></p></td>
<td><p>✅</p></td>
@@ -1869,15 +1874,25 @@ expressions. The following function families are <strong>not currently planned</
<td><p>—</p></td>
<td><p></p></td>
</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">try_make_timestamp_ltz</span></code></p></td>
+<td><p>✅</p></td>
+<td><p>—</p></td>
+<td><p>Same support as <code class="docutils literal notranslate"><span class="pre">try_make_timestamp</span></code> (Spark 4.0+)</p></td>
+</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">try_make_timestamp_ntz</span></code></p></td>
+<td><p>✅</p></td>
+<td><p>—</p></td>
+<td><p>Same support as <code class="docutils literal notranslate"><span class="pre">try_make_timestamp</span></code> (Spark 4.0+)</p></td>
+</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">try_to_date</span></code></p></td>
<td><p>✅</p></td>
<td><p>—</p></td>
<td><p>Rewrites to <code class="docutils literal notranslate"><span class="pre">Cast</span></code>/<code class="docutils literal notranslate"><span class="pre">GetTimestamp</span></code> before Comet sees the plan; same support as <code class="docutils literal notranslate"><span class="pre">to_date</span></code></p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">try_to_time</span></code></p></td>
-<td><p>🔜</p></td>
+<td><p>✅</p></td>
<td><p>—</p></td>
-<td><p>Spark 4.1 TIME type; tracked by <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4288">#4288</a></p></td>
+<td><p>Same support as <code class="docutils literal notranslate"><span class="pre">to_time</span></code></p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">try_to_timestamp</span></code></p></td>
<td><p>✅</p></td>
@@ -1942,7 +1957,8 @@ expressions. The following function families are <strong>not currently planned</
<section id="generator-funcs">
<h2>generator_funcs<a class="headerlink" href="#generator-funcs" title="Link to this heading">#</a></h2>
<p><code class="docutils literal notranslate"><span class="pre">explode</span></code>, <code class="docutils literal notranslate"><span class="pre">explode_outer</span></code>, <code class="docutils literal notranslate"><span class="pre">posexplode</span></code>, and <code class="docutils literal notranslate"><span class="pre">posexplode_outer</span></code> are supported via
-<code class="docutils literal notranslate"><span class="pre">CometExplodeExec</span></code> (operator-level, not expression-level). Enabled by default via
+<code class="docutils literal notranslate"><span class="pre">CometExplodeExec</span></code> (operator-level, not expression-level) for array input; map input falls back
+to Spark (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/2837">#2837</a>). Enabled by default via
<code class="docutils literal notranslate"><span class="pre">spark.comet.exec.explode.enabled</span></code>.</p>
<div class="pst-scrollable-table-container"><table class="table">
<thead>
@@ -2058,17 +2074,17 @@ expressions. The following function families are <strong>not currently planned</
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">from_json</span></code></p></td>
<td><p>✅</p></td>
<td><p>Hybrid</p></td>
-<td><p>Falls back by default; opt-in via allowIncompatible (<a class="reference internal" href="../../contributor-guide/expression-audits/json_funcs.html#from-json"><span class="std std-ref">audit</span></a>)</p></td>
+<td><p>Routes through the JVM codegen dispatcher by default; the native path (supported schemas only) is opt-in via allowIncompatible (<a class="reference internal" href="../../contributor-guide/expression-audits/json_funcs.html#from-json"><span class="std std-ref">audit</span></a>)</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">get_json_object</span></code></p></td>
<td><p>✅</p></td>
<td><p>Hybrid</p></td>
-<td><p>Some inputs need allowIncompatible (<a class="reference internal" href="../../contributor-guide/expression-audits/json_funcs.html#get-json-object"><span class="std std-ref">audit</span></a>)</p></td>
+<td><p>Routes through the JVM codegen dispatcher by default; the native path is opt-in via allowIncompatible (<a class="reference internal" href="../../contributor-guide/expression-audits/json_funcs.html#get-json-object"><span class="std std-ref">audit</span></a>)</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">json_array_length</span></code></p></td>
<td><p>✅</p></td>
<td><p>Hybrid</p></td>
-<td><p>Single-quoted/trailing JSON needs allowIncompatible (<a class="reference internal" href="../../contributor-guide/expression-audits/json_funcs.html#json-array-length"><span class="std std-ref">audit</span></a>)</p></td>
+<td><p>Routes through the JVM codegen dispatcher by default; the native path, which differs for single-quoted JSON, unescaped control characters, and trailing content, is opt-in via allowIncompatible (<a class="reference internal" href="../../contributor-guide/expression-audits/json_funcs.html#json-array-length"><span class="std std-ref">audit</span></a>)</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">json_object_keys</span></code></p></td>
<td><p>✅</p></td>
@@ -2088,7 +2104,7 @@ expressions. The following function families are <strong>not currently planned</
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">to_json</span></code></p></td>
<td><p>✅</p></td>
<td><p>Hybrid</p></td>
-<td><p>Options and map/array inputs fall back (<a class="reference internal" href="../../contributor-guide/expression-audits/json_funcs.html#to-json"><span class="std std-ref">audit</span></a>)</p></td>
+<td><p>Routes through the JVM codegen dispatcher by default, including options and map/array inputs; the native path is opt-in via allowIncompatible (<a class="reference internal" href="../../contributor-guide/expression-audits/json_funcs.html#to-json"><span class="std std-ref">audit</span></a>)</p></td>
</tr>
</tbody>
</table>
@@ -2215,7 +2231,7 @@ expressions. The following function families are <strong>not currently planned</
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">map_from_entries</span></code></p></td>
<td><p>✅</p></td>
<td><p>Hybrid</p></td>
-<td><p>BinaryType key/value falls back (Incompatible) (<a class="reference internal" href="#compatibility/expressions/map.md"><span class="xref myst">details</span></a>)</p></td>
+<td><p>BinaryType keys/values and <code class="docutils literal notranslate"><span class="pre">spark.sql.mapKeyDedupPolicy=LAST_WIN</span></code> route through the JVM codegen dispatcher (<a class="reference internal" href="#compatibility/expressions/map.md"><span class="xref myst">details</span></a>)</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">map_keys</span></code></p></td>
<td><p>✅</p></td>
@@ -2501,7 +2517,7 @@ expressions. The following function families are <strong>not currently planned</
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">random</span></code></p></td>
<td><p>✅</p></td>
<td><p>Native</p></td>
-<td><p>Alias for <code class="docutils literal notranslate"><span class="pre">rand</span></code> (Spark 4.0+); seed must be a literal</p></td>
+<td><p>Alias for <code class="docutils literal notranslate"><span class="pre">rand</span></code>; seed must be a literal</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">randstr</span></code></p></td>
<td><p>✅</p></td>
@@ -2858,17 +2874,17 @@ expressions. The following function families are <strong>not currently planned</
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">regexp</span></code></p></td>
<td><p>✅</p></td>
<td><p>Hybrid</p></td>
-<td><p>In-subset literals run natively; others fall back by default (<a class="reference internal" href="compatibility/regex.html"><span class="std std-doc">details</span></a>)</p></td>
+<td><p>In-subset literal patterns run natively; other patterns route through the JVM codegen dispatcher (<a class="reference internal" href="compatibility/regex.html"><span class="std std-doc">details</span></a>)</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">regexp_like</span></code></p></td>
<td><p>✅</p></td>
<td><p>Hybrid</p></td>
-<td><p>In-subset literals run natively; others fall back by default (<a class="reference internal" href="compatibility/regex.html"><span class="std std-doc">details</span></a>)</p></td>
+<td><p>In-subset literal patterns run natively; other patterns route through the JVM codegen dispatcher (<a class="reference internal" href="compatibility/regex.html"><span class="std std-doc">details</span></a>)</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">rlike</span></code></p></td>
<td><p>✅</p></td>
<td><p>Hybrid</p></td>
-<td><p>In-subset literals run natively; others fall back by default (<a class="reference internal" href="compatibility/regex.html"><span class="std std-doc">details</span></a>)</p></td>
+<td><p>In-subset literal patterns run natively; other patterns route through the JVM codegen dispatcher (<a class="reference internal" href="compatibility/regex.html"><span class="std std-doc">details</span></a>)</p></td>
</tr>
</tbody>
</table>
@@ -3069,7 +3085,7 @@ expressions. The following function families are <strong>not currently planned</
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">regexp_count</span></code></p></td>
<td><p>✅</p></td>
<td><p>—</p></td>
-<td><p>Runs natively (rewrites to <code class="docutils literal notranslate"><span class="pre">size(regexp_extract_all(...))</span></code>)</p></td>
+<td><p>Rewrites to <code class="docutils literal notranslate"><span class="pre">size(regexp_extract_all(...))</span></code>; <code class="docutils literal notranslate"><span class="pre">regexp_extract_all</span></code> routes through the JVM codegen dispatcher by default</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">regexp_extract</span></code></p></td>
<td><p>✅</p></td>
@@ -3094,7 +3110,7 @@ expressions. The following function families are <strong>not currently planned</
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">regexp_substr</span></code></p></td>
<td><p>✅</p></td>
<td><p>—</p></td>
-<td><p>Runs natively (rewrites to <code class="docutils literal notranslate"><span class="pre">nullif(regexp_extract(...),</span> <span class="pre">'')</span></code>)</p></td>
+<td><p>Rewrites to <code class="docutils literal notranslate"><span class="pre">nullif(regexp_extract(...),</span> <span class="pre">'')</span></code>; <code class="docutils literal notranslate"><span class="pre">regexp_extract</span></code> routes through the JVM codegen dispatcher by default</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">repeat</span></code></p></td>
<td><p>✅</p></td>
@@ -3164,7 +3180,7 @@ expressions. The following function families are <strong>not currently planned</
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">to_binary</span></code></p></td>
<td><p>✅</p></td>
<td><p>—</p></td>
-<td><p>Hex and base64 forms accelerated natively; the <code class="docutils literal notranslate"><span class="pre">utf-8</span></code> form rewrites to <code class="docutils literal notranslate"><span class="pre">encode</span></code> and runs via codegen dispatch on Spark 4.0+</p></td>
+<td><p>The hex form runs natively; the base64 and <code class="docutils literal notranslate"><span class="pre">utf-8</span></code> forms route through the JVM codegen dispatcher</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">to_char</span></code></p></td>
<td><p>✅</p></td>
@@ -3194,7 +3210,7 @@ expressions. The following function families are <strong>not currently planned</
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">try_to_binary</span></code></p></td>
<td><p>✅</p></td>
<td><p>—</p></td>
-<td><p>Runs natively (rewrites to <code class="docutils literal notranslate"><span class="pre">try_eval(to_binary(...))</span></code>)</p></td>
+<td><p>Rewrites to <code class="docutils literal notranslate"><span class="pre">try_eval(to_binary(...))</span></code>, which routes through the JVM codegen dispatcher</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">try_to_number</span></code></p></td>
<td><p>✅</p></td>
@@ -3209,7 +3225,7 @@ expressions. The following function families are <strong>not currently planned</
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">unbase64</span></code></p></td>
<td><p>✅</p></td>
<td><p>Hybrid</p></td>
-<td><p></p></td>
+<td><p>Column or literal input runs natively; other child expressions, and the strict form used by <code class="docutils literal notranslate"><span class="pre">to_binary(str,</span> <span class="pre">'base64')</span></code>, route through the JVM codegen dispatcher</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">upper</span></code></p></td>
<td><p>✅</p></td>
@@ -3263,17 +3279,22 @@ expressions. The following function families are <strong>not currently planned</
<td><p>Native</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">try_url_decode</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">try_parse_url</span></code></p></td>
+<td><p>✅</p></td>
+<td><p>—</p></td>
+<td><p>Spark 4.0+; runs natively and returns NULL for an invalid URL</p></td>
+</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">try_url_decode</span></code></p></td>
<td><p>✅</p></td>
<td><p>—</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">url_decode</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">url_decode</span></code></p></td>
<td><p>✅</p></td>
<td><p>—</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">url_encode</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">url_encode</span></code></p></td>
<td><p>✅</p></td>
<td><p>—</p></td>
<td><p></p></td>
@@ -3438,6 +3459,8 @@ for the full list of supported functions, frames, and fallback cases.</p>
<li><p><strong>Accessor expressions (subscript and field access, not functions):</strong> struct field access (<code class="docutils literal notranslate"><span class="pre">col.field</span></code>), array element access (<code class="docutils literal notranslate"><span class="pre">arr[i]</span></code>), and map value access (<code class="docutils literal notranslate"><span class="pre">map[key]</span></code>).</p></li>
<li><p><strong>Internal decimal arithmetic:</strong> <code class="docutils literal notranslate"><span class="pre">CheckOverflow</span></code>, <code class="docutils literal notranslate"><span class="pre">MakeDecimal</span></code>, and <code class="docutils literal notranslate"><span class="pre">UnscaledValue</span></code>, which the analyzer inserts around decimal operations.</p></li>
<li><p><strong>User-defined functions:</strong> Scala UDFs registered through the DataFrame or SQL API.</p></li>
+<li><p><strong>DataSource V2 catalog functions:</strong> Iceberg’s system functions <code class="docutils literal notranslate"><span class="pre">bucket</span></code>, <code class="docutils literal notranslate"><span class="pre">truncate</span></code>, <code class="docutils literal notranslate"><span class="pre">years</span></code>, <code class="docutils literal notranslate"><span class="pre">months</span></code>, <code class="docutils literal notranslate"><span class="pre">days</span></c [...]
+<li><p><strong>Lowered built-ins:</strong> Spark lowers some built-in functions to <code class="docutils literal notranslate"><span class="pre">StaticInvoke</span></code> or <code class="docutils literal notranslate"><span class="pre">Invoke</span></code> calls. Those without a native mapping run through the JVM codegen dispatcher when their input and output types are supported.</p></li>
<li><p><strong>Structural expressions:</strong> aliases, attribute references, literals, sort orders, and <code class="docutils literal notranslate"><span class="pre">CASE</span> <span class="pre">WHEN</span></code>.</p></li>
</ul>
<p>This list is illustrative, not exhaustive: the per-function tables are not the complete set of expressions Comet can accelerate.</p>
diff --git a/user-guide/latest/iceberg-writes.html b/user-guide/latest/iceberg-writes.html
index c6ffe340db..328077110f 100644
--- a/user-guide/latest/iceberg-writes.html
+++ b/user-guide/latest/iceberg-writes.html
@@ -800,7 +800,7 @@ and it is the foundation for the second toggle: when
<code class="docutils literal notranslate"><span class="pre">spark.comet.iceberg.write.enabled=true</span></code> and the write passes the eligibility check below, the
<code class="docutils literal notranslate"><span class="pre">IcebergWrite</span></code> operator’s per-task Parquet write is delegated to
<a class="reference external" href="https://github.com/apache/iceberg-rust">iceberg-rust</a> via Comet’s native execution pipeline
-(<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/5308">#5308</a>).</p>
+(<a class="reference external" href="https://github.com/apache/datafusion-comet/pull/5361">#5361</a>).</p>
</section>
<section id="how-the-native-write-works">
<h2>How the native write works<a class="headerlink" href="#how-the-native-write-works" title="Link to this heading">#</a></h2>
@@ -881,7 +881,7 @@ trade-off, only no plan change.</p>
<section id="native-parquet-write-eligibility">
<h2>Native Parquet write eligibility<a class="headerlink" href="#native-parquet-write-eligibility" title="Link to this heading">#</a></h2>
<p>When <code class="docutils literal notranslate"><span class="pre">spark.comet.iceberg.write.enabled=true</span></code>
-(<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/5308">#5308</a>), the <code class="docutils literal notranslate"><span class="pre">IcebergWrite</span></code> operator’s
+(<a class="reference external" href="https://github.com/apache/datafusion-comet/pull/5361">#5361</a>), the <code class="docutils literal notranslate"><span class="pre">IcebergWrite</span></code> operator’s
per-task Parquet write is delegated to <a class="reference external" href="https://github.com/apache/iceberg-rust">iceberg-rust</a>.
The native writer must produce the same outcome as iceberg-java — the same Parquet features,
statistics, and manifest metadata — so a write is only eligible when every table property it
@@ -890,7 +890,9 @@ feeding the write is fully Comet-native. For a partitioned table that plan inclu
distribution and local sort Iceberg requests on its partition transforms; those stay native
because the transforms themselves have native implementations (see
<a class="reference internal" href="iceberg.html"><span class="std std-doc">Iceberg system functions</span></a>). Ineligible writes run through iceberg-java unchanged,
-with the reason reported as a fall-back reason in Comet’s extended EXPLAIN output.</p>
+with the reason reported as a fall-back reason in Comet’s extended EXPLAIN output. A write that
+runs natively shows <code class="docutils literal notranslate"><span class="pre">CometIcebergWrite</span></code> under <code class="docutils literal notranslate"><span class="pre">IcebergCommit</span></code> in the physical plan; an ineligible
+write keeps <code class="docutils literal notranslate"><span class="pre">IcebergWrite</span></code>.</p>
<p>The native writer reads its input as Arrow batches from a Comet operator, so the write’s input
must itself run in Comet. A write whose input is a local relation, such as <code class="docutils literal notranslate"><span class="pre">INSERT</span> <span class="pre">...</span> <span class="pre">VALUES</span></code> or
<code class="docutils literal notranslate"><span class="pre">df.writeTo(...).append()</span></code> on a DataFrame built from local data, is fed by Spark’s
@@ -964,7 +966,8 @@ iceberg-java resolves per-write options and <code class="docutils literal notran
everything not listed above must be absent: unvetted <code class="docutils literal notranslate"><span class="pre">write.parquet.*</span></code> keys (e.g.
<code class="docutils literal notranslate"><span class="pre">bloom-filter-max-bytes</span></code>, <code class="docutils literal notranslate"><span class="pre">stats-enabled.column.*</span></code>, keys added by future Iceberg versions),
any <code class="docutils literal notranslate"><span class="pre">parquet.*</span></code> table property (including <code class="docutils literal notranslate"><span class="pre">parquet.enable.dictionary</span></code>), and any <code class="docutils literal notranslate"><span class="pre">parquet.*</span></code>
-key in the session Hadoop configuration (with <code class="docutils literal notranslate"><span class="pre">HadoopFileIO</span></code>-backed output those reach
+key in the session Hadoop configuration other than the reader-only
+<code class="docutils literal notranslate"><span class="pre">parquet.hadoop.vectored.io.enabled</span></code> (with <code class="docutils literal notranslate"><span class="pre">HadoopFileIO</span></code>-backed output those reach
iceberg-java’s writer but not the native one). Also gated explicitly: any <code class="docutils literal notranslate"><span class="pre">encryption.*</span></code> key,
<code class="docutils literal notranslate"><span class="pre">write.object-storage.enabled=true</span></code>, <code class="docutils literal notranslate"><span class="pre">write.location-provider.impl</span></code>, and <code class="docutils literal notranslate"><span class="pre">io-impl</span></code>.</p>
<p>Two checks look past properties at the table’s instantiated state, because both can be
diff --git a/user-guide/latest/iceberg.html b/user-guide/latest/iceberg.html
index f43eb2dc6f..e84ad65e37 100644
--- a/user-guide/latest/iceberg.html
+++ b/user-guide/latest/iceberg.html
@@ -783,7 +783,7 @@ under the License.
then serialized to Comet’s native execution engine (see
<a class="reference external" href="https://github.com/apache/datafusion-comet/pull/2528">PR #2528</a>).</p>
<p>The example below uses Spark’s package downloader to retrieve Comet 1.1.0-SNAPSHOT and Iceberg
-1.8.1, but Comet has been tested with Iceberg 1.5, 1.7, 1.8, 1.9, 1.10, and 1.11. The native Iceberg
+1.8.1, but Comet has been tested with Iceberg 1.5, 1.8, 1.9, 1.10, and 1.11. The native Iceberg
reader is enabled by default. To disable it, set <code class="docutils literal notranslate"><span class="pre">spark.comet.scan.icebergNative.enabled=false</span></code>.</p>
<p>The example uses the Spark 3.5 / Scala 2.12 build of Comet; substitute the Comet artifact
matching your Spark and Scala versions (Comet also ships Spark 3.5 / Scala 2.13 and Spark
@@ -849,7 +849,7 @@ Spark (no AES-192-GCM in the underlying crypto).</p></li>
<li><p>Equality and comparison predicates (<code class="docutils literal notranslate"><span class="pre">=</span></code>, <code class="docutils literal notranslate"><span class="pre">!=</span></code>, <code class="docutils literal notranslate"><span class="pre">></span></code>, <code class="docutils literal notranslate"><span class="pre">>=</span></code>, <code class="docutils literal notranslate"><span class="pre"><</span></code>, <code class="docutils literal notranslate"><span [...]
<li><p>Logical operators (<code class="docutils literal notranslate"><span class="pre">AND</span></code>, <code class="docutils literal notranslate"><span class="pre">OR</span></code>)</p></li>
<li><p>NULL checks (<code class="docutils literal notranslate"><span class="pre">IS</span> <span class="pre">NULL</span></code>, <code class="docutils literal notranslate"><span class="pre">IS</span> <span class="pre">NOT</span> <span class="pre">NULL</span></code>) on primitive columns</p></li>
-<li><p><code class="docutils literal notranslate"><span class="pre">IN</span></code> and <code class="docutils literal notranslate"><span class="pre">NOT</span> <span class="pre">IN</span></code> list operations</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">IN</span></code> list operations (<code class="docutils literal notranslate"><span class="pre">NOT</span> <span class="pre">IN</span></code> is applied after the scan)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">BETWEEN</span></code> operations</p></li>
</ul>
<p>NULL checks on struct, array, and map columns still use native scans and return correct
@@ -903,6 +903,7 @@ configure Spark to use a REST catalog with Comet’s native Iceberg scan:</p>
<section id="object-store-configuration-s3">
<h3>Object store configuration (S3)<a class="headerlink" href="#object-store-configuration-s3" title="Link to this heading">#</a></h3>
<p>The native reader has its own Rust object store client and does not go through Iceberg’s JVM FileIO, neither <code class="docutils literal notranslate"><span class="pre">S3FileIO</span></code> nor the older Hadoop S3A filesystem. It configures that client from the catalog’s <code class="docutils literal notranslate"><span class="pre">s3.*</span></code> properties (the same keys <code class="docutils literal notranslate"><span class="pre">S3FileIO</span></code> reads), from <code class [...]
+<p>Per-bucket <code class="docutils literal notranslate"><span class="pre">fs.s3a.bucket.<bucket>.*</span></code> settings apply to the bucket that holds the table’s data and delete files. The native reader uses one object-store configuration per scan, so a scan whose data or delete files span more than one S3 bucket falls back to Spark.</p>
<p>For a custom S3-compatible endpoint, configure the catalog with the endpoint, path-style access, region, and credentials (Hive shown):</p>
<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span><span class="w"> </span>--conf<span class="w"> </span>spark.sql.catalog.s3_cat<span class="o">=</span>org.apache.iceberg.spark.SparkCatalog<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--conf<span class="w"> </span>spark.sql.catalog.s3_cat.type<span class="o">=</span>hive<span class="w"> </span><span class="se">\</span>
@@ -915,7 +916,7 @@ configure Spark to use a REST catalog with Comet’s native Iceberg scan:</p>
<span class="w"> </span>--conf<span class="w"> </span>spark.sql.catalog.s3_cat.s3.secret-access-key<span class="o">=</span>...
</pre></div>
</div>
-<p>These <code class="docutils literal notranslate"><span class="pre">s3.*</span></code> storage properties are not specific to the Hive catalog shown here. When <code class="docutils literal notranslate"><span class="pre">s3.access-key-id</span></code> / <code class="docutils literal notranslate"><span class="pre">s3.secret-access-key</span></code> are omitted, credentials come from the standard AWS chain (environment variables, instance profiles, and so on). <code class="docutils liter [...]
+<p>These <code class="docutils literal notranslate"><span class="pre">s3.*</span></code> storage properties are not specific to the Hive catalog shown here. When <code class="docutils literal notranslate"><span class="pre">s3.access-key-id</span></code> / <code class="docutils literal notranslate"><span class="pre">s3.secret-access-key</span></code> are omitted, credentials come from the standard AWS chain (environment variables, instance profiles, and so on). The region is not auto-dete [...]
</section>
<section id="current-limitations">
<h3>Current limitations<a class="headerlink" href="#current-limitations" title="Link to this heading">#</a></h3>
@@ -923,10 +924,14 @@ configure Spark to use a REST catalog with Comet’s native Iceberg scan:</p>
<ul class="simple">
<li><p>Iceberg table spec v4 or newer</p></li>
<li><p>v3 tables with columns that declare an initial default value</p></li>
-<li><p>v3 column types the native reader cannot read (<code class="docutils literal notranslate"><span class="pre">variant</span></code>, <code class="docutils literal notranslate"><span class="pre">geometry</span></code>, <code class="docutils literal notranslate"><span class="pre">geography</span></code>, <code class="docutils literal notranslate"><span class="pre">unknown</span></code>)</p></li>
+<li><p>v3 column types the native reader cannot read (<code class="docutils literal notranslate"><span class="pre">geometry</span></code>, <code class="docutils literal notranslate"><span class="pre">geography</span></code>, <code class="docutils literal notranslate"><span class="pre">unknown</span></code>), and
+<code class="docutils literal notranslate"><span class="pre">variant</span></code> columns the query reads (on Spark 4.0+, a table whose <code class="docutils literal notranslate"><span class="pre">variant</span></code> columns are not
+projected is read natively)</p></li>
<li><p>Encrypted tables with 192-bit data keys (no AES-192-GCM in the underlying crypto)</p></li>
<li><p>Delete files in a format other than Parquet or Puffin (Avro or ORC positional/equality deletes)</p></li>
<li><p>Tables backed by Avro or ORC data files (only Parquet is accelerated)</p></li>
+<li><p>Scans whose data or delete files span more than one S3 bucket (the native reader uses one
+object-store configuration per scan)</p></li>
<li><p>Tables partitioned on <code class="docutils literal notranslate"><span class="pre">BINARY</span></code> or <code class="docutils literal notranslate"><span class="pre">DECIMAL</span></code> (with precision >28) columns</p></li>
<li><p>Scans with residual filters using <code class="docutils literal notranslate"><span class="pre">truncate</span></code>, <code class="docutils literal notranslate"><span class="pre">bucket</span></code>, <code class="docutils literal notranslate"><span class="pre">year</span></code>, <code class="docutils literal notranslate"><span class="pre">month</span></code>, <code class="docutils literal notranslate"><span class="pre">day</span></code>, or <code class="docutils literal notrans [...]
transform functions (partition pruning still works, but row-level filtering of these
@@ -988,7 +993,7 @@ the expression fall back to Spark.</p>
</section>
<section id="task-input-metrics">
<h3>Task input metrics<a class="headerlink" href="#task-input-metrics" title="Link to this heading">#</a></h3>
-<p>The native Iceberg reader populates Spark’s task-level <code class="docutils literal notranslate"><span class="pre">inputMetrics.bytesRead</span></code> (visible in the Spark UI Stages tab) using the <code class="docutils literal notranslate"><span class="pre">bytes_read</span></code> counter from iceberg-rust’s <code class="docutils literal notranslate"><span class="pre">ScanMetrics</span></code>. This counter includes bytes read from both data files and delete files.</p>
+<p>The native Iceberg reader populates Spark’s task-level <code class="docutils literal notranslate"><span class="pre">inputMetrics.bytesRead</span></code> (visible in the Spark UI Stages tab) using the <code class="docutils literal notranslate"><span class="pre">bytes_read</span></code> counter from iceberg-rust’s <code class="docutils literal notranslate"><span class="pre">ScanMetrics</span></code>. This counter includes bytes read from both data files and delete files. The scan’s SQL [...]
<p>Iceberg Java does not explicitly report <code class="docutils literal notranslate"><span class="pre">bytesRead</span></code> to Spark’s task input metrics. On the iceberg Java path, any <code class="docutils literal notranslate"><span class="pre">bytesRead</span></code> value comes from Hadoop’s filesystem-level I/O counters, not from Iceberg itself. Because Comet’s native reader and the Hadoop filesystem use different counting mechanisms, the exact byte counts will differ between the [...]
</section>
</section>
diff --git a/user-guide/latest/in-memory-cache.html b/user-guide/latest/in-memory-cache.html
index fe2cbd9445..4bbae9189d 100644
--- a/user-guide/latest/in-memory-cache.html
+++ b/user-guide/latest/in-memory-cache.html
@@ -987,7 +987,7 @@ comparison with Spark’s own cache format. That comparison is under <a class="r
<p>Spark serializes a cached batch with <code class="docutils literal notranslate"><span class="pre">spark.serializer</span></code> whenever the block leaves the heap: the
<code class="docutils literal notranslate"><span class="pre">_SER</span></code> storage levels, replication, cross-executor fetches, and the disk half of the default
<code class="docutils literal notranslate"><span class="pre">MEMORY_AND_DISK</span></code>. So an ordinary <code class="docutils literal notranslate"><span class="pre">df.cache()</span></code> that spills is enough to reach it.</p>
-<p>If you run with <code class="docutils literal notranslate"><span class="pre">spark.kryo.registrationRequired=true</span></code>, register Comet’s classes:</p>
+<p>If you run Kryo with <code class="docutils literal notranslate"><span class="pre">spark.kryo.registrationRequired=true</span></code>, register Comet’s classes:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">spark</span><span class="o">.</span><span class="n">serializer</span><span class="o">=</span><span class="n">org</span><span class="o">.</span><span class="n">apache</span><span class="o">.</span><span class="n">spark</span><span class="o">.</span><span class="n">serializer</span><span class="o">.</span><span class="n">KryoSerializer</span>
<span class="n">spark</span><span class="o">.</span><span class="n">kryo</span><span class="o">.</span><span class="n">registrationRequired</span><span class="o">=</span><span class="n">true</span>
<span class="n">spark</span><span class="o">.</span><span class="n">kryo</span><span class="o">.</span><span class="n">registrator</span><span class="o">=</span><span class="n">org</span><span class="o">.</span><span class="n">apache</span><span class="o">.</span><span class="n">comet</span><span class="o">.</span><span class="n">CometKryoRegistrator</span>
@@ -997,7 +997,8 @@ comparison with Spark’s own cache format. That comparison is under <a class="r
<code class="docutils literal notranslate"><span class="pre">KryoSerializer</span></code> reads it when <code class="docutils literal notranslate"><span class="pre">SparkEnv</span></code> builds the serializer, which happens before any plugin
runs. Without it, caching fails with a “Class is not registered” error that does not name this
feature. Comet’s driver plugin warns at startup when it sees Kryo, <code class="docutils literal notranslate"><span class="pre">registrationRequired</span></code>, and no
-registrator.</p>
+registrator. Native broadcast needs the same registrator even when the cache is disabled; see
+<a class="reference internal" href="installation.html#kryo-serialization"><span class="std std-ref">Kryo serialization</span></a>.</p>
</section>
<section id="limitations">
<h2>Limitations<a class="headerlink" href="#limitations" title="Link to this heading">#</a></h2>
diff --git a/user-guide/latest/installation.html b/user-guide/latest/installation.html
index 2dff96fd3c..a1bad8a1fe 100644
--- a/user-guide/latest/installation.html
+++ b/user-guide/latest/installation.html
@@ -818,8 +818,8 @@ Other versions may work well enough for development and evaluation purposes.</p>
<div class="admonition warning">
<p class="admonition-title">Warning</p>
<p>Spark 3.4 support is deprecated as of the 1.0.0 release and will be removed in a future release.
-Apache Spark’s own SQL test suite is no longer run against Spark 3.4 on every change; it runs only
-on demand. We recommend moving to Spark 3.5 or later.</p>
+Apache Spark’s own SQL test suite no longer runs against Spark 3.4 automatically; it runs only on
+demand. We recommend moving to Spark 3.5 or later.</p>
</div>
<div class="pst-scrollable-table-container"><table class="table">
<thead>
@@ -834,34 +834,36 @@ on demand. We recommend moving to Spark 3.5 or later.</p>
<tr class="row-even"><td><p>3.4.3</p></td>
<td><p>17</p></td>
<td><p>2.12/2.13</p></td>
-<td><p>Yes</p></td>
+<td><p>Nightly</p></td>
<td><p>On demand</p></td>
</tr>
<tr class="row-odd"><td><p>3.5.9</p></td>
<td><p>17</p></td>
<td><p>2.12/2.13</p></td>
-<td><p>Yes</p></td>
-<td><p>Yes</p></td>
+<td><p>Nightly</p></td>
+<td><p>Nightly</p></td>
</tr>
<tr class="row-even"><td><p>4.0.4</p></td>
<td><p>17/21</p></td>
<td><p>2.13</p></td>
-<td><p>Yes</p></td>
-<td><p>Yes</p></td>
+<td><p>Nightly</p></td>
+<td><p>Nightly</p></td>
</tr>
<tr class="row-odd"><td><p>4.1.3</p></td>
<td><p>17/21</p></td>
<td><p>2.13</p></td>
-<td><p>Yes</p></td>
-<td><p>Yes</p></td>
+<td><p>Before merge</p></td>
+<td><p>Before merge</p></td>
</tr>
</tbody>
</table>
</div>
<p>Note that we do not test the full matrix of supported Java and Scala versions in CI for every Spark version.</p>
-<p>“On demand” in the table above means the suite is not run automatically before a change is merged.
-A contributor can still run it against an individual pull request, but Spark 3.4 is no longer
-covered by default.</p>
+<p>“Before merge” in the table above means the suite must pass before a change is merged. “Nightly” means
+the suite runs once a day against the <code class="docutils literal notranslate"><span class="pre">main</span></code> branch, so a regression it finds is caught after the change
+has been merged rather than before. “On demand” means the suite does not run automatically at all. A
+contributor can still run it against an individual pull request, but Spark 3.4 is no longer covered by
+default.</p>
<p>Experimental support is provided for the following versions of Apache Spark and is intended for development/testing
use only and should not be used in production yet.</p>
<div class="pst-scrollable-table-container"><table class="table">
@@ -877,7 +879,7 @@ use only and should not be used in production yet.</p>
<tr class="row-even"><td><p>4.2.0</p></td>
<td><p>17</p></td>
<td><p>2.13</p></td>
-<td><p>Yes</p></td>
+<td><p>Nightly</p></td>
<td><p>No</p></td>
</tr>
</tbody>
@@ -1021,6 +1023,21 @@ components which will then fail at runtime. For example:</p>
<p>In addition to Apache Spark memory configuration parameters, Comet introduces additional parameters to configure memory
allocation for native execution. See <a class="reference internal" href="tuning.html"><span class="std std-doc">Comet Memory Tuning</span></a> for details.</p>
</section>
+<section id="kryo-serialization">
+<h3>Kryo serialization<a class="headerlink" href="#kryo-serialization" title="Link to this heading">#</a></h3>
+<p>If the application uses Kryo (<code class="docutils literal notranslate"><span class="pre">spark.serializer=org.apache.spark.serializer.KryoSerializer</span></code>) with
+<code class="docutils literal notranslate"><span class="pre">spark.kryo.registrationRequired=true</span></code>, also register Comet’s classes with Kryo:</p>
+<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>--conf<span class="w"> </span>spark.kryo.registrator<span class="o">=</span>org.apache.comet.CometKryoRegistrator
+</pre></div>
+</div>
+<p>Without it, any query that uses Comet’s native broadcast exchange, which is enabled by default,
+fails with Kryo’s “Class is not registered” error, for example on the first broadcast hash join.
+The <a class="reference internal" href="in-memory-cache.html#kryo"><span class="std std-ref">in-memory cache</span></a> needs the same registrator. Set it before the
+<code class="docutils literal notranslate"><span class="pre">SparkContext</span></code> is created: <code class="docutils literal notranslate"><span class="pre">KryoSerializer</span></code> reads it before Comet’s plugin runs, so Comet cannot
+add it for you. <code class="docutils literal notranslate"><span class="pre">spark.kryo.registrator</span></code> accepts a comma-separated list, so an application with
+its own registrator can list both. Comet logs a warning at startup when Kryo requires registration
+and this registrator is missing.</p>
+</section>
</section>
</section>
@@ -1086,6 +1103,7 @@ allocation for native execution. See <a class="reference internal" href="tuning.
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#checking-the-comet-version">Checking the Comet Version</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#additional-configuration">Additional Configuration</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#memory-tuning">Memory tuning</a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#kryo-serialization">Kryo serialization</a></li>
</ul>
</li>
</ul>
diff --git a/user-guide/latest/kubernetes.html b/user-guide/latest/kubernetes.html
index 96312be080..0b8e526692 100644
--- a/user-guide/latest/kubernetes.html
+++ b/user-guide/latest/kubernetes.html
@@ -779,11 +779,13 @@ under the License.
<h1>Comet Kubernetes Support<a class="headerlink" href="#comet-kubernetes-support" title="Link to this heading">#</a></h1>
<section id="comet-docker-images">
<h2>Comet Docker Images<a class="headerlink" href="#comet-docker-images" title="Link to this heading">#</a></h2>
-<p>Run the following command from the root of this repository to build the Comet Docker image, or use a <a class="reference external" href="https://hub.docker.com/r/apache/datafusion-comet">published
-Docker image</a>.</p>
+<p>Run the following command from the root of this repository to build the Comet Docker image. The image is based on
+<code class="docutils literal notranslate"><span class="pre">apache/spark:4.1.3</span></code> and adds the Comet JAR for Spark 4.1 and Scala 2.13 to <code class="docutils literal notranslate"><span class="pre">$SPARK_HOME/jars</span></code>.</p>
<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>docker<span class="w"> </span>build<span class="w"> </span>-t<span class="w"> </span>apache/datafusion-comet<span class="w"> </span>-f<span class="w"> </span>kube/Dockerfile<span class="w"> </span>.
</pre></div>
</div>
+<p>Push the image to a registry that your cluster can pull from, and use that image name in the <code class="docutils literal notranslate"><span class="pre">image</span></code> field of the
+examples below.</p>
</section>
<section id="example-spark-submit">
<h2>Example Spark Submit<a class="headerlink" href="#example-spark-submit" title="Link to this heading">#</a></h2>
@@ -820,14 +822,17 @@ metadata:
spec:
<span class="w"> </span>type:<span class="w"> </span>Scala
<span class="w"> </span>mode:<span class="w"> </span>cluster
-<span class="w"> </span>image:<span class="w"> </span>apache/datafusion-comet:1.1.0-SNAPSHOT-spark3.5.5-scala2.12-java11
+<span class="w"> </span>image:<span class="w"> </span>apache/datafusion-comet
<span class="w"> </span>imagePullPolicy:<span class="w"> </span>IfNotPresent
<span class="w"> </span>mainClass:<span class="w"> </span>org.apache.spark.examples.SparkPi
<span class="w"> </span>mainApplicationFile:<span class="w"> </span>local:///opt/spark/examples/jars/spark-examples_2.13-4.1.3.jar
<span class="w"> </span>sparkConf:
-<span class="w"> </span><span class="s2">"spark.executor.extraClassPath"</span>:<span class="w"> </span><span class="s2">"/opt/spark/jars/comet-spark-spark3.5_2.12-1.1.0-SNAPSHOT.jar"</span>
-<span class="w"> </span><span class="s2">"spark.driver.extraClassPath"</span>:<span class="w"> </span><span class="s2">"/opt/spark/jars/comet-spark-spark3.5_2.12-1.1.0-SNAPSHOT.jar"</span>
+<span class="w"> </span><span class="s2">"spark.executor.extraClassPath"</span>:<span class="w"> </span><span class="s2">"/opt/spark/jars/comet-spark-spark4.1_2.13-1.1.0-SNAPSHOT.jar"</span>
+<span class="w"> </span><span class="s2">"spark.driver.extraClassPath"</span>:<span class="w"> </span><span class="s2">"/opt/spark/jars/comet-spark-spark4.1_2.13-1.1.0-SNAPSHOT.jar"</span>
<span class="w"> </span><span class="s2">"spark.plugins"</span>:<span class="w"> </span><span class="s2">"org.apache.spark.CometPlugin"</span>
+<span class="w"> </span><span class="s2">"spark.memory.offHeap.enabled"</span>:<span class="w"> </span><span class="s2">"true"</span>
+<span class="w"> </span><span class="s2">"spark.memory.offHeap.size"</span>:<span class="w"> </span><span class="s2">"1g"</span>
+<span class="w"> </span><span class="s2">"spark.executor.memoryOverhead"</span>:<span class="w"> </span><span class="s2">"1g"</span>
<span class="w"> </span><span class="s2">"spark.comet.enabled"</span>:<span class="w"> </span><span class="s2">"true"</span>
<span class="w"> </span><span class="s2">"spark.comet.exec.enabled"</span>:<span class="w"> </span><span class="s2">"true"</span>
<span class="w"> </span><span class="s2">"spark.comet.shuffle.enabled"</span>:<span class="w"> </span><span class="s2">"true"</span>
diff --git a/user-guide/latest/metrics.html b/user-guide/latest/metrics.html
index b8968c1f94..fbfe4f4593 100644
--- a/user-guide/latest/metrics.html
+++ b/user-guide/latest/metrics.html
@@ -780,8 +780,8 @@ under the License.
<section id="spark-sql-metrics">
<h2>Spark SQL Metrics<a class="headerlink" href="#spark-sql-metrics" title="Link to this heading">#</a></h2>
<p>Comet operators report the following metrics in the Spark SQL UI.</p>
-<section id="cometscanexec">
-<h3>CometScanExec<a class="headerlink" href="#cometscanexec" title="Link to this heading">#</a></h3>
+<section id="cometbatchscan">
+<h3>CometBatchScan<a class="headerlink" href="#cometbatchscan" title="Link to this heading">#</a></h3>
<div class="pst-scrollable-table-container"><table class="table">
<thead>
<tr class="row-odd"><th class="head"><p>Metric</p></th>
@@ -790,11 +790,13 @@ under the License.
</thead>
<tbody>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">scan</span> <span class="pre">time</span></code></p></td>
-<td><p>Total time to scan a Parquet file. This is not comparable to the same metric in Spark because Comet’s scan metric is more accurate. Although both Comet and Spark measure the time in nanoseconds, Spark rounds this time to the nearest millisecond per batch and Comet does not.</p></td>
+<td><p>Time spent reading batches from the wrapped DataSource V2 reader. Comet measures this time in nanoseconds and does not round it to the nearest millisecond per batch.</p></td>
</tr>
</tbody>
</table>
</div>
+<p>Parquet scans through the DataSource V1 API appear in plans as <code class="docutils literal notranslate"><span class="pre">CometNativeScan</span></code> and report the
+native metrics described under <a class="reference internal" href="#native-parquet-scans">Native Parquet scans</a> rather than <code class="docutils literal notranslate"><span class="pre">scan</span> <span class="pre">time</span></code>.</p>
</section>
<section id="cometicebergnativescan">
<h3>CometIcebergNativeScan<a class="headerlink" href="#cometicebergnativescan" title="Link to this heading">#</a></h3>
@@ -819,7 +821,7 @@ plain Spark + Iceberg <code class="docutils literal notranslate"><span class="pr
<td><p>File scan tasks (splits) read by this scan.</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">scan</span> <span class="pre">time</span></code></p></td>
-<td><p>Time spent in the native scan’s record-batch polling, covering the iceberg-rust reader plus Comet’s schema adaptation. It excludes time the stream spends waiting between polls, so it is decode/compute time, not end-to-end scan latency. This differs from the <code class="docutils literal notranslate"><span class="pre">scan</span> <span class="pre">time</span></code> under <code class="docutils literal notranslate"><span class="pre">CometScanExec</span></code>, which times Parquet f [...]
+<td><p>Time spent in the native scan’s record-batch polling, covering the iceberg-rust reader plus Comet’s schema adaptation. It excludes time the stream spends waiting between polls, so it is decode/compute time, not end-to-end scan latency. This differs from the <code class="docutils literal notranslate"><span class="pre">scan</span> <span class="pre">time</span></code> under <code class="docutils literal notranslate"><span class="pre">CometBatchScan</span></code>, which times batch re [...]
</tr>
</tbody>
</table>
@@ -888,6 +890,37 @@ scan planning and do not change during execution.</p>
counter, and Comet reads natively through iceberg-rust, which exposes no deletes-applied count, so
the value would always be 0.</p>
</section>
+<section id="comethashaggregate">
+<h3>CometHashAggregate<a class="headerlink" href="#comethashaggregate" title="Link to this heading">#</a></h3>
+<p>Native aggregates with grouping keys report these additional metrics:</p>
+<div class="pst-scrollable-table-container"><table class="table">
+<thead>
+<tr class="row-odd"><th class="head"><p>Metric</p></th>
+<th class="head"><p>Description</p></th>
+</tr>
+</thead>
+<tbody>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">rows</span> <span class="pre">bypassing</span> <span class="pre">partial</span> <span class="pre">aggregation</span></code></p></td>
+<td><p>Input rows passed through without partial aggregation. See <a class="reference internal" href="tuning.html#adaptive-partial-aggregation"><span class="std std-ref">Adaptive Partial Aggregation</span></a>.</p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">number</span> <span class="pre">of</span> <span class="pre">spills</span></code></p></td>
+<td><p>Number of times the aggregate spilled to disk.</p></td>
+</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">total</span> <span class="pre">spilled</span> <span class="pre">bytes</span></code></p></td>
+<td><p>Bytes written to aggregate spill files.</p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">number</span> <span class="pre">of</span> <span class="pre">spilled</span> <span class="pre">rows</span></code></p></td>
+<td><p>Rows written to aggregate spill files.</p></td>
+</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">peak</span> <span class="pre">native</span> <span class="pre">aggregate</span> <span class="pre">memory</span></code></p></td>
+<td><p>Peak memory used by the native aggregate.</p></td>
+</tr>
+</tbody>
+</table>
+</div>
+<p>Spill bytes from native sorts, aggregates, and sort-merge joins are also added to Spark’s task-level
+<code class="docutils literal notranslate"><span class="pre">diskBytesSpilled</span></code> metric in every stage, not only in shuffle stages.</p>
+</section>
<section id="hash-joins">
<h3>Hash Joins<a class="headerlink" href="#hash-joins" title="Link to this heading">#</a></h3>
<p>With <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.join.dynamicFilter.enabled=true</span></code>, native broadcast and shuffled hash joins
@@ -936,8 +969,8 @@ reader savings. Existing join, scan, and intervening filter metrics retain their
</tr>
</thead>
<tbody>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">native</span> <span class="pre">shuffle</span> <span class="pre">time</span></code></p></td>
-<td><p>Total time in native code excluding any child operators.</p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">native</span> <span class="pre">shuffle</span> <span class="pre">writer</span> <span class="pre">time</span></code></p></td>
+<td><p>Total time in the native shuffle writer, excluding any child operators.</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">repartition</span> <span class="pre">time</span></code></p></td>
<td><p>Time to repartition batches.</p></td>
@@ -945,21 +978,29 @@ reader savings. Existing join, scan, and intervening filter metrics retain their
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">partition</span> <span class="pre">interleaving</span> <span class="pre">time</span></code></p></td>
<td><p>Time to interleave partitioned batches before writing them.</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">memory</span> <span class="pre">pool</span> <span class="pre">time</span></code></p></td>
-<td><p>Time interacting with memory pool.</p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">encoding</span> <span class="pre">and</span> <span class="pre">compression</span> <span class="pre">time</span></code></p></td>
+<td><p>Time to encode batches in Arrow IPC format and compress them with the configured codec (<code class="docutils literal notranslate"><span class="pre">spark.comet.shuffle.compression.codec</span></code>, default <code class="docutils literal notranslate"><span class="pre">lz4</span></code>).</p></td>
+</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">decoding</span> <span class="pre">and</span> <span class="pre">decompression</span> <span class="pre">time</span></code></p></td>
+<td><p>Time to decompress and decode shuffle blocks when they are read.</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">encoding</span> <span class="pre">and</span> <span class="pre">compression</span> <span class="pre">time</span></code></p></td>
-<td><p>Time to encode batches in IPC format and compress using ZSTD.</p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">number</span> <span class="pre">of</span> <span class="pre">spills</span></code></p></td>
+<td><p>Number of native shuffle spills.</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">disk</span> <span class="pre">spilled</span> <span class="pre">bytes</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">disk</span> <span class="pre">spilled</span> <span class="pre">bytes</span></code></p></td>
<td><p>Actual bytes written to native shuffle spill files on disk.</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">memory</span> <span class="pre">spilled</span> <span class="pre">bytes</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">memory</span> <span class="pre">spilled</span> <span class="pre">bytes</span></code></p></td>
<td><p>Uncompressed Arrow backing-buffer and partition-index data before spilling.</p></td>
</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">number</span> <span class="pre">of</span> <span class="pre">input</span> <span class="pre">batches</span></code></p></td>
+<td><p>Batches received by the native shuffle writer.</p></td>
+</tr>
</tbody>
</table>
</div>
+<p>Comet exchanges also report Spark’s standard shuffle read and write metrics, including when native
+operators read shuffle blocks directly.</p>
<p>Disk and memory spilled bytes measure different representations of the same native shuffle spill.
Disk spill bytes count the actual bytes written to disk: compressed when shuffle compression is
enabled and uncompressed when <code class="docutils literal notranslate"><span class="pre">spark.shuffle.compress=false</span></code>. Memory spill bytes follow Spark’s
@@ -992,11 +1033,8 @@ logged for each native plan (and there is one plan per task, so this is very ver
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">elapsed_compute</span></code></p></td>
<td><p>Total time spent in this operator, fetching batches from a JVM iterator.</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">jvm_fetch_time</span></code></p></td>
-<td><p>Time spent in the JVM fetching input batches to be read by this <code class="docutils literal notranslate"><span class="pre">ScanExec</span></code> instance.</p></td>
-</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">arrow_ffi_time</span></code></p></td>
-<td><p>Time spent using Arrow FFI to create Arrow batches from the memory addresses returned from the JVM.</p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">cast_time</span></code></p></td>
+<td><p>Time spent casting columns to the requested data types during the scan.</p></td>
</tr>
</tbody>
</table>
@@ -1020,14 +1058,14 @@ logged for each native plan (and there is one plan per task, so this is very ver
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">interleave_time</span></code></p></td>
<td><p>Time to gather partitioned rows into output batches before writing.</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">ipc_time</span></code></p></td>
-<td><p>Time to encode batches in IPC format and compress using ZSTD.</p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">encode_time</span></code></p></td>
+<td><p>Time to encode batches in Arrow IPC format and compress them with the configured codec.</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">mempool_time</span></code></p></td>
-<td><p>Time interacting with memory pool.</p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">write_time</span></code></p></td>
+<td><p>Time spent writing encoded data to its destination.</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">write_time</span></code></p></td>
-<td><p>Time spent writing bytes to disk.</p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">input_batches</span></code></p></td>
+<td><p>Number of input batches.</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spill_count</span></code></p></td>
<td><p>Number of native shuffle spills.</p></td>
@@ -1176,8 +1214,9 @@ filesystem or network traffic accounting.</p>
<nav class="bd-toc-nav page-toc" aria-labelledby="pst-page-navigation-heading-2">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#spark-sql-metrics">Spark SQL Metrics</a><ul class="nav section-nav flex-column">
-<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#cometscanexec">CometScanExec</a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#cometbatchscan">CometBatchScan</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#cometicebergnativescan">CometIcebergNativeScan</a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#comethashaggregate">CometHashAggregate</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#hash-joins">Hash Joins</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#exchange">Exchange</a></li>
</ul>
diff --git a/user-guide/latest/migration-guide.html b/user-guide/latest/migration-guide.html
index 0aade6b1fb..eb31c882a1 100644
--- a/user-guide/latest/migration-guide.html
+++ b/user-guide/latest/migration-guide.html
@@ -807,6 +807,12 @@ which point the newer behavior becomes unconditional.</p>
legacy behavior, please open an issue describing your use case so it can be considered before the
key is removed.</p>
</section>
+<section id="upgrading-to-comet-1-1-0">
+<h2>Upgrading to Comet 1.1.0<a class="headerlink" href="#upgrading-to-comet-1-1-0" title="Link to this heading">#</a></h2>
+<p>Comet <code class="docutils literal notranslate"><span class="pre">1.1.0</span></code> makes no behavior changes that need a <code class="docutils literal notranslate"><span class="pre">spark.comet.legacy.*</span></code> key.</p>
+<p>Comet <code class="docutils literal notranslate"><span class="pre">1.1.0</span></code> requires JDK 17 or later. JDK 11 is no longer supported. See
+<a class="reference internal" href="installation.html"><span class="std std-doc">Installing Comet</span></a> for the supported Java, Scala, and Spark versions.</p>
+</section>
<section id="upgrading-to-comet-1-0-0">
<h2>Upgrading to Comet 1.0.0<a class="headerlink" href="#upgrading-to-comet-1-0-0" title="Link to this heading">#</a></h2>
<p>Comet <code class="docutils literal notranslate"><span class="pre">1.0.0</span></code> is the first release under the stable
@@ -865,6 +871,7 @@ release, review the release notes for the versions in between.</p>
<nav class="bd-toc-nav page-toc" aria-labelledby="pst-page-navigation-heading-2">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#legacy-configuration-keys">Legacy Configuration Keys</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#upgrading-to-comet-1-1-0">Upgrading to Comet 1.1.0</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#upgrading-to-comet-1-0-0">Upgrading to Comet 1.0.0</a></li>
</ul>
</nav></div>
diff --git a/user-guide/latest/operators.html b/user-guide/latest/operators.html
index 14221b6d2c..e1c1e2b9fd 100644
--- a/user-guide/latest/operators.html
+++ b/user-guide/latest/operators.html
@@ -834,7 +834,7 @@ omitted from the tables below and may be reconsidered based on demand:</p>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">BatchScanExec</span></code></p></td>
<td><p>✅</p></td>
-<td><p>Parquet, Apache Iceberg Parquet, and CSV (native) scans. See <a class="reference internal" href="compatibility/scans.html"><span class="std std-doc">Parquet Scan Compatibility</span></a> and the <a class="reference internal" href="iceberg.html"><span class="std std-doc">Iceberg Guide</span></a>.</p></td>
+<td><p>Apache Iceberg Parquet scans run natively. Native CSV scans are experimental and disabled by default. DataSource V2 Parquet scans are not accelerated. See <a class="reference internal" href="compatibility/scans.html"><span class="std std-doc">Parquet Scan Compatibility</span></a> and the <a class="reference internal" href="iceberg.html"><span class="std std-doc">Iceberg Guide</span></a>.</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">LocalTableScanExec</span></code></p></td>
<td><p>⚠️</p></td>
@@ -845,8 +845,8 @@ omitted from the tables below and may be reconsidered based on demand:</p>
<td><p>Spark 4.0 and later. See <a class="reference internal" href="compatibility/operators.html#empty-relations"><span class="std std-ref">Empty Relations</span></a> for native-input support and writer fallback.</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">InMemoryTableScanExec</span></code></p></td>
-<td><p>🔜</p></td>
-<td><p>Cached / in-memory table scans fall back today.</p></td>
+<td><p>⚠️</p></td>
+<td><p>Experimental, disabled by default. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.inMemoryCache.enabled=true</span></code> before the application starts so Comet installs its Arrow cache serializer. Relations with unsupported column types stay in Spark’s cache format and fall back. See <a class="reference internal" href="in-memory-cache.html"><span class="std std-doc">In-Memory Cache</span></a>.</p></td>
</tr>
</tbody>
</table>
@@ -928,7 +928,7 @@ omitted from the tables below and may be reconsidered based on demand:</p>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">ObjectHashAggregateExec</span></code></p></td>
<td><p>✅</p></td>
-<td><p>Supports a limited set of aggregates, such as <code class="docutils literal notranslate"><span class="pre">bloom_filter_agg</span></code>. Falls back when Comet shuffle is disabled, which would otherwise split the aggregate across Comet and Spark. See the <a class="reference internal" href="tuning.html"><span class="std std-doc">Tuning Guide</span></a>.</p></td>
+<td><p>Runs the object-buffer aggregates Comet supports, such as <code class="docutils literal notranslate"><span class="pre">collect_list</span></code>, <code class="docutils literal notranslate"><span class="pre">collect_set</span></code>, <code class="docutils literal notranslate"><span class="pre">percentile</span></code>, <code class="docutils literal notranslate"><span class="pre">approx_percentile</span></code>, <code class="docutils literal notranslate"><span class="pre">mode</sp [...]
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">SortAggregateExec</span></code></p></td>
<td><p>🔜</p></td>
@@ -958,7 +958,7 @@ omitted from the tables below and may be reconsidered based on demand:</p>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">SortMergeJoinExec</span></code></p></td>
<td><p>✅</p></td>
-<td><p></p></td>
+<td><p>Supports <code class="docutils literal notranslate"><span class="pre">BINARY</span></code> join keys. Nested-type (struct, array, map) and collated-string join keys fall back to Spark.</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">BroadcastNestedLoopJoinExec</span></code></p></td>
<td><p>✅</p></td>
@@ -1006,7 +1006,7 @@ omitted from the tables below and may be reconsidered based on demand:</p>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">WindowGroupLimitExec</span></code></p></td>
<td><p>✅</p></td>
-<td><p>Streaming per-partition top-K pushdown for <code class="docutils literal notranslate"><span class="pre">ROW_NUMBER</span></code>, <code class="docutils literal notranslate"><span class="pre">RANK</span></code>, and <code class="docutils literal notranslate"><span class="pre">DENSE_RANK</span></code>.</p></td>
+<td><p>Spark 3.5 and later. Streaming per-partition top-K pushdown for <code class="docutils literal notranslate"><span class="pre">ROW_NUMBER</span></code>, <code class="docutils literal notranslate"><span class="pre">RANK</span></code>, and <code class="docutils literal notranslate"><span class="pre">DENSE_RANK</span></code>.</p></td>
</tr>
</tbody>
</table>
@@ -1060,6 +1060,10 @@ omitted from the tables below and may be reconsidered based on demand:</p>
<td><p>⚠️</p></td>
<td><p>Spark 3.4/3.5 only. Experimental native Parquet writes, disabled by default (opt-in). Replaced by <code class="docutils literal notranslate"><span class="pre">WriteFilesExec</span></code> on Spark 4.0+ and removed with Spark 3.x support.</p></td>
</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">AppendDataExec</span></code>, <code class="docutils literal notranslate"><span class="pre">OverwriteByExpressionExec</span></code>, <code class="docutils literal notranslate"><span class="pre">OverwritePartitionsDynamicExec</span></code>, <code class="docutils literal notranslate"><span class="pre">ReplaceDataExec</span></code></p></td>
+<td><p>⚠️</p></td>
+<td><p>Apache Iceberg tables only. Experimental, disabled by default. See <a class="reference internal" href="iceberg-writes.html"><span class="std std-doc">Iceberg Writes</span></a>.</p></td>
+</tr>
</tbody>
</table>
</div>
@@ -1074,9 +1078,13 @@ omitted from the tables below and may be reconsidered based on demand:</p>
</tr>
</thead>
<tbody>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">ArrowEvalPythonExec</span></code>, <code class="docutils literal notranslate"><span class="pre">MapInArrowExec</span></code>, <code class="docutils literal notranslate"><span class="pre">MapInPandasExec</span></code>, <code class="docutils literal notranslate"><span class="pre">FlatMapGroupsInPandasExec</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">MapInArrowExec</span></code>, <code class="docutils literal notranslate"><span class="pre">MapInPandasExec</span></code></p></td>
+<td><p>⚠️</p></td>
+<td><p>Spark 4.0 and later. Experimental, disabled by default (<code class="docutils literal notranslate"><span class="pre">spark.comet.exec.pyarrowUDF.enabled</span></code>). See <a class="reference internal" href="pyarrow-udfs.html"><span class="std std-doc">PyArrow UDF Acceleration</span></a>.</p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">ArrowEvalPythonExec</span></code>, <code class="docutils literal notranslate"><span class="pre">FlatMapGroupsInPandasExec</span></code></p></td>
<td><p>🔜</p></td>
-<td><p>Experimental accelerated PyArrow UDF support is in progress (<a class="reference external" href="https://github.com/apache/datafusion-comet/pull/4234">#4234</a>).</p></td>
+<td><p>Scalar <code class="docutils literal notranslate"><span class="pre">@pandas_udf</span></code> (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/5386">#5386</a>) and grouped <code class="docutils literal notranslate"><span class="pre">applyInPandas</span></code> (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/5123">#5123</a>) fall back to Spark.</p></td>
</tr>
</tbody>
</table>
diff --git a/user-guide/latest/pyarrow-udfs.html b/user-guide/latest/pyarrow-udfs.html
index e6c667f615..4e4f2f7e50 100644
--- a/user-guide/latest/pyarrow-udfs.html
+++ b/user-guide/latest/pyarrow-udfs.html
@@ -786,7 +786,7 @@ RecordBatches or Pandas DataFrames. Under the hood, Spark communicates with the
using the Arrow IPC format.</p>
<p>Without Comet, the execution path for these UDFs involves unnecessary data conversions:</p>
<ol class="arabic simple">
-<li><p>Comet reads data in Arrow columnar format (via CometScan)</p></li>
+<li><p>Comet reads data in Arrow columnar format (via <code class="docutils literal notranslate"><span class="pre">CometNativeScan</span></code>)</p></li>
<li><p>Spark inserts a ColumnarToRow transition (converts Arrow to UnsafeRow)</p></li>
<li><p>The Python runner converts those rows back to Arrow to send to Python</p></li>
<li><p>Python executes the UDF on Arrow batches</p></li>
@@ -796,8 +796,8 @@ using the Arrow IPC format.</p>
</section>
<section id="how-comet-optimizes-this">
<h2>How Comet Optimizes This<a class="headerlink" href="#how-comet-optimizes-this" title="Link to this heading">#</a></h2>
-<p>When enabled, Comet detects <code class="docutils literal notranslate"><span class="pre">PythonMapInArrowExec</span></code> / <code class="docutils literal notranslate"><span class="pre">MapInArrowExec</span></code> and <code class="docutils literal notranslate"><span class="pre">MapInPandasExec</span></code>
-operators in the physical plan and replaces them with <code class="docutils literal notranslate"><span class="pre">CometMapInBatchExec</span></code>, which:</p>
+<p>When enabled, Comet detects <code class="docutils literal notranslate"><span class="pre">MapInArrowExec</span></code> and <code class="docutils literal notranslate"><span class="pre">MapInPandasExec</span></code> operators in the physical plan
+and replaces them with <code class="docutils literal notranslate"><span class="pre">CometMapInBatchExec</span></code> (shown as <code class="docutils literal notranslate"><span class="pre">CometMapInBatch</span></code> in plans), which:</p>
<ul class="simple">
<li><p>Reads Arrow columnar batches directly from the upstream Comet operator</p></li>
<li><p>Feeds them to the Python runner without the expensive UnsafeProjection copy</p></li>
@@ -811,16 +811,14 @@ copies that remain.</p>
<section id="plan-flow">
<h3>Plan flow<a class="headerlink" href="#plan-flow" title="Link to this heading">#</a></h3>
<p>Without Comet’s optimization:</p>
-<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">PythonMapInArrow</span> <span class="o">/</span> <span class="n">MapInArrow</span> <span class="o">/</span> <span class="n">MapInPandas</span>
-<span class="o">+-</span> <span class="n">ColumnarToRow</span> <span class="o"><-</span> <span class="n">Arrow</span> <span class="o">-></span> <span class="n">Row</span> <span class="n">copy</span>
- <span class="o">+-</span> <span class="n">CometNativeExec</span> <span class="o"><-</span> <span class="n">Arrow</span> <span class="n">batch</span>
- <span class="o">+-</span> <span class="n">CometScan</span>
+<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">MapInArrow</span> <span class="o">/</span> <span class="n">MapInPandas</span>
+<span class="o">+-</span> <span class="n">CometColumnarToRow</span> <span class="o"><-</span> <span class="n">Arrow</span> <span class="o">-></span> <span class="n">Row</span> <span class="n">copy</span>
+ <span class="o">+-</span> <span class="n">CometNativeScan</span> <span class="o"><-</span> <span class="n">Arrow</span> <span class="n">batch</span>
</pre></div>
</div>
<p>With the optimization enabled:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">CometMapInBatch</span> <span class="o"><-</span> <span class="n">Arrow</span> <span class="n">batch</span> <span class="ow">in</span><span class="o">/</span><span class="n">out</span><span class="p">,</span> <span class="n">Python</span> <span class="n">runner</span> <span class="n">attached</span>
-<span class="o">+-</span> <span class="n">CometNativeExec</span>
- <span class="o">+-</span> <span class="n">CometScan</span>
+<span class="o">+-</span> <span class="n">CometNativeScan</span>
</pre></div>
</div>
</section>
@@ -853,7 +851,7 @@ worker. Both confs can be set independently.</p>
</thead>
<tbody>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">df.mapInArrow(func,</span> <span class="pre">schema)</span></code></p></td>
-<td><p><code class="docutils literal notranslate"><span class="pre">PythonMapInArrowExec</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">MapInArrowExec</span></code></p></td>
<td><p>Yes</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">df.mapInPandas(func,</span> <span class="pre">schema)</span></code></p></td>
@@ -912,22 +910,20 @@ worker. Both confs can be set independently.</p>
</div>
<p>You should see:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">CometMapInBatch</span> <span class="o">...</span>
-<span class="o">+-</span> <span class="n">CometNativeExec</span> <span class="o">...</span>
- <span class="o">+-</span> <span class="n">CometScan</span> <span class="o">...</span>
+<span class="o">+-</span> <span class="n">CometNativeScan</span> <span class="n">parquet</span> <span class="o">...</span>
</pre></div>
</div>
<p>Instead of the unoptimized plan:</p>
-<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">PythonMapInArrow</span> <span class="o">...</span>
-<span class="o">+-</span> <span class="n">ColumnarToRow</span>
- <span class="o">+-</span> <span class="n">CometNativeExec</span> <span class="o">...</span>
- <span class="o">+-</span> <span class="n">CometScan</span> <span class="o">...</span>
+<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">MapInArrow</span> <span class="o">...</span>
+<span class="o">+-</span> <span class="n">CometColumnarToRow</span>
+ <span class="o">+-</span> <span class="n">CometNativeScan</span> <span class="n">parquet</span> <span class="o">...</span>
</pre></div>
</div>
<p>When AQE is enabled (the Spark default) and the query contains a shuffle, the
optimization is applied during stage materialization. Calling <code class="docutils literal notranslate"><span class="pre">explain()</span></code> before
running an action will show the unoptimized plan:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">AdaptiveSparkPlan</span> <span class="n">isFinalPlan</span><span class="o">=</span><span class="n">false</span>
-<span class="o">+-</span> <span class="n">PythonMapInArrow</span> <span class="o">...</span>
+<span class="o">+-</span> <span class="n">MapInArrow</span> <span class="o">...</span>
<span class="o">+-</span> <span class="n">CometExchange</span> <span class="o">...</span>
</pre></div>
</div>
diff --git a/user-guide/latest/s3-credential-providers.html b/user-guide/latest/s3-credential-providers.html
index 31a123a652..b580f1f2aa 100644
--- a/user-guide/latest/s3-credential-providers.html
+++ b/user-guide/latest/s3-credential-providers.html
@@ -777,7 +777,7 @@ under the License.
-->
<section id="s3-credential-providers">
<h1>S3 Credential Providers<a class="headerlink" href="#s3-credential-providers" title="Link to this heading">#</a></h1>
-<p>Comet’s native S3 readers normally fetch credentials from the standard AWS credential chain (static keys, instance profiles, environment variables, etc.). Some clusters use a vendor-managed mechanism instead, where credentials are issued per request based on a JWT or per S3 path. For those clusters, Comet supports loading a vendor-supplied bridge class that routes every native credential request through the vendor’s Java code.</p>
+<p>Comet’s native S3 readers and native Iceberg writer normally fetch credentials from the standard AWS credential chain (static keys, instance profiles, environment variables, etc.). Some clusters use a vendor-managed mechanism instead, where credentials are issued per request based on a JWT or per S3 path. For those clusters, Comet supports loading a vendor-supplied bridge class that routes every native credential request through the vendor’s Java code.</p>
<section id="do-i-need-this">
<h2>Do I need this?<a class="headerlink" href="#do-i-need-this" title="Link to this heading">#</a></h2>
<p>You don’t, if any of the following describe your cluster:</p>
@@ -845,13 +845,12 @@ under the License.
</section>
<section id="iceberg-explicit-s3-region-required">
<h2>Iceberg: explicit S3 region required<a class="headerlink" href="#iceberg-explicit-s3-region-required" title="Link to this heading">#</a></h2>
-<p>With the bridge configured, Comet wires a custom credential loader into <code class="docutils literal notranslate"><span class="pre">iceberg-storage-opendal</span></code>. <code class="docutils literal notranslate"><span class="pre">opendal</span></code>’s built-in S3 region auto-detection only runs when no custom loader is configured, so on the bridge path the region (and endpoint for non-AWS) must be set explicitly on the Spark catalog:</p>
+<p><code class="docutils literal notranslate"><span class="pre">iceberg-storage-opendal</span></code> does not auto-detect a bucket’s region, with or without the bridge. When neither the catalog (<code class="docutils literal notranslate"><span class="pre">s3.region</span></code> or <code class="docutils literal notranslate"><span class="pre">client.region</span></code>) nor the executor environment (<code class="docutils literal notranslate"><span class="pre">AWS_REGION</span></code> / [...]
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">spark</span><span class="o">.</span><span class="n">sql</span><span class="o">.</span><span class="n">catalog</span><span class="o">.<</span><span class="n">catalog</span><span class="o">>.</span><span class="n">s3</span><span class="o">.</span><span class="n">region</span> <span class="o">=</span> <span class="n">us</span><span class="o">-</span><span class="n">east</span><sp [...]
<span class="n">spark</span><span class="o">.</span><span class="n">sql</span><span class="o">.</span><span class="n">catalog</span><span class="o">.<</span><span class="n">catalog</span><span class="o">>.</span><span class="n">s3</span><span class="o">.</span><span class="n">endpoint</span> <span class="o">=</span> <span class="n">https</span><span class="p">:</span><span class="o">//...</span> <span class="p">(</span><span class="n">non</span><span class="o">-</span><span [...]
<span class="n">spark</span><span class="o">.</span><span class="n">sql</span><span class="o">.</span><span class="n">catalog</span><span class="o">.<</span><span class="n">catalog</span><span class="o">>.</span><span class="n">s3</span><span class="o">.</span><span class="n">path</span><span class="o">-</span><span class="n">style</span><span class="o">-</span><span class="n">access</span> <span class="o">=</span> <span class="n">true</span> <span class="p">(</span><span clas [...]
</pre></div>
</div>
-<p>If you hit <code class="docutils literal notranslate"><span class="pre">region</span> <span class="pre">is</span> <span class="pre">missing.</span> <span class="pre">Please</span> <span class="pre">find</span> <span class="pre">it</span> <span class="pre">by</span> <span class="pre">S3::detect_region()</span> <span class="pre">or</span> <span class="pre">set</span> <span class="pre">them</span> <span class="pre">in</span> <span class="pre">env</span></code>, this is the missing config.</p>
</section>
<section id="writing-a-bridge">
<h2>Writing a bridge<a class="headerlink" href="#writing-a-bridge" title="Link to this heading">#</a></h2>
@@ -1036,10 +1035,10 @@ under the License.
</thead>
<tbody>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">READ</span></code></p></td>
-<td><p>All native scan paths (raw Parquet, Iceberg). Comet today only sends READ.</p></td>
+<td><p>All native scan paths (raw Parquet, Iceberg).</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">WRITE</span></code></p></td>
-<td><p>Reserved for future native write paths.</p></td>
+<td><p>The native Iceberg writer (see <a class="reference internal" href="iceberg-writes.html"><span class="std std-doc">Iceberg Writes</span></a>). If the configured provider fails to initialize, the write fails rather than falling back to the default credential chain.</p></td>
</tr>
</tbody>
</table>
diff --git a/user-guide/latest/scala_java_udfs.html b/user-guide/latest/scala_java_udfs.html
index 955f3a75b5..7454b10c65 100644
--- a/user-guide/latest/scala_java_udfs.html
+++ b/user-guide/latest/scala_java_udfs.html
@@ -802,7 +802,7 @@ under the License.
<h2>Supported<a class="headerlink" href="#supported" title="Link to this heading">#</a></h2>
<ul class="simple">
<li><p>User functions registered via <code class="docutils literal notranslate"><span class="pre">udf(...)</span></code>, <code class="docutils literal notranslate"><span class="pre">spark.udf.register(...)</span></code> (Scala or Java functional interfaces), or SQL <code class="docutils literal notranslate"><span class="pre">CREATE</span> <span class="pre">FUNCTION</span> <span class="pre">...</span> <span class="pre">AS</span> <span class="pre">'com.example.MyUDF'</span></code>.</p></li>
-<li><p>Scalar input/output types: <code class="docutils literal notranslate"><span class="pre">Boolean</span></code>, <code class="docutils literal notranslate"><span class="pre">Byte</span></code>, <code class="docutils literal notranslate"><span class="pre">Short</span></code>, <code class="docutils literal notranslate"><span class="pre">Int</span></code>, <code class="docutils literal notranslate"><span class="pre">Long</span></code>, <code class="docutils literal notranslate"><span c [...]
+<li><p>Scalar input/output types: <code class="docutils literal notranslate"><span class="pre">Boolean</span></code>, <code class="docutils literal notranslate"><span class="pre">Byte</span></code>, <code class="docutils literal notranslate"><span class="pre">Short</span></code>, <code class="docutils literal notranslate"><span class="pre">Int</span></code>, <code class="docutils literal notranslate"><span class="pre">Long</span></code>, <code class="docutils literal notranslate"><span c [...]
<li><p>Complex input/output types with arbitrary nesting: <code class="docutils literal notranslate"><span class="pre">ArrayType</span></code>, <code class="docutils literal notranslate"><span class="pre">StructType</span></code>, <code class="docutils literal notranslate"><span class="pre">MapType</span></code>.</p></li>
<li><p>Composition with other Catalyst expressions inside the argument tree (e.g. <code class="docutils literal notranslate"><span class="pre">myUdf(upper(s))</span></code> runs as one unit in the Comet pipeline).</p></li>
<li><p>Higher-order functions (<code class="docutils literal notranslate"><span class="pre">transform</span></code>, <code class="docutils literal notranslate"><span class="pre">filter</span></code>, <code class="docutils literal notranslate"><span class="pre">exists</span></code>, <code class="docutils literal notranslate"><span class="pre">aggregate</span></code>, <code class="docutils literal notranslate"><span class="pre">zip_with</span></code>, <code class="docutils literal notransl [...]
@@ -815,7 +815,7 @@ under the License.
<li><p>Table UDFs and generators.</p></li>
<li><p>Python <code class="docutils literal notranslate"><span class="pre">@udf</span></code> and Pandas <code class="docutils literal notranslate"><span class="pre">@pandas_udf</span></code>.</p></li>
<li><p>Hive <code class="docutils literal notranslate"><span class="pre">GenericUDF</span></code> and <code class="docutils literal notranslate"><span class="pre">SimpleUDF</span></code>.</p></li>
-<li><p><code class="docutils literal notranslate"><span class="pre">CalendarIntervalType</span></code>, <code class="docutils literal notranslate"><span class="pre">NullType</span></code>, and <code class="docutils literal notranslate"><span class="pre">UserDefinedType</span></code> arguments and return types. UDT-typed columns fall back to Spark; to keep execution in the Comet pipeline, store and read the underlying representation directly (e.g. write MLlib <code class="docutils literal [...]
+<li><p><code class="docutils literal notranslate"><span class="pre">NullType</span></code> and <code class="docutils literal notranslate"><span class="pre">UserDefinedType</span></code> arguments and return types. UDT-typed columns fall back to Spark; to keep execution in the Comet pipeline, store and read the underlying representation directly (e.g. write MLlib <code class="docutils literal notranslate"><span class="pre">Vector</span></code> outputs as <code class="docutils literal notr [...]
<li><p>Trees whose total nested-field count (output plus all input columns the UDF tree references) exceeds <code class="docutils literal notranslate"><span class="pre">spark.sql.codegen.maxFields</span></code> (default 100). Comet refuses these at plan time and the operator falls back to Spark.</p></li>
</ul>
<p>When a UDF is rejected, the reason surfaces through Comet’s standard fallback diagnostics; the query still runs on Spark.</p>
diff --git a/user-guide/latest/source.html b/user-guide/latest/source.html
index 47ccc682f1..7f22b2be76 100644
--- a/user-guide/latest/source.html
+++ b/user-guide/latest/source.html
@@ -781,7 +781,7 @@ under the License.
<section id="using-a-published-source-release">
<h2>Using a Published Source Release<a class="headerlink" href="#using-a-published-source-release" title="Link to this heading">#</a></h2>
<p>This documentation is for the current development version of Comet. Published source releases are only available for released versions.
-To use this version of Comet, see the following section on building from the GitHubn repository.</p>
+To use this version of Comet, see the following section on building from the GitHub repository.</p>
</section>
<section id="building-from-the-github-repository">
<h2>Building from the GitHub repository<a class="headerlink" href="#building-from-the-github-repository" title="Link to this heading">#</a></h2>
diff --git a/user-guide/latest/tuning.html b/user-guide/latest/tuning.html
index a5103d2557..dd52834e3c 100644
--- a/user-guide/latest/tuning.html
+++ b/user-guide/latest/tuning.html
@@ -780,12 +780,11 @@ under the License.
<p>Comet provides some tuning options to help you get the best performance from your queries.</p>
<section id="configuring-tokio-runtime">
<h2>Configuring Tokio Runtime<a class="headerlink" href="#configuring-tokio-runtime" title="Link to this heading">#</a></h2>
-<p>Comet uses a global tokio runtime per executor process using tokio’s defaults of one worker thread per core and a
-maximum of 512 blocking threads. These values can be overridden using the environment variables <code class="docutils literal notranslate"><span class="pre">COMET_WORKER_THREADS</span></code>
-and <code class="docutils literal notranslate"><span class="pre">COMET_MAX_BLOCKING_THREADS</span></code>.</p>
-<p>It is recommended that <code class="docutils literal notranslate"><span class="pre">COMET_WORKER_THREADS</span></code> be set to the number of executor cores. This may not be necessary
-in some environments, such as Kubernetes, where the number of cores allocated to a pod will already be equal to the
-number of executor cores.</p>
+<p>Comet uses a global tokio runtime per executor process. By default it starts one worker thread per executor core
+(<code class="docutils literal notranslate"><span class="pre">spark.executor.cores</span></code>, or the thread count of <code class="docutils literal notranslate"><span class="pre">local[N]</span></code> and <code class="docutils literal notranslate"><span class="pre">local[*]</span></code> masters) and allows up to 512 blocking
+threads, which is tokio’s default. If <code class="docutils literal notranslate"><span class="pre">spark.executor.cores</span></code> is not set outside local mode, Comet starts a single
+worker thread. These values can be overridden using the environment variables <code class="docutils literal notranslate"><span class="pre">COMET_WORKER_THREADS</span></code> and
+<code class="docutils literal notranslate"><span class="pre">COMET_MAX_BLOCKING_THREADS</span></code>.</p>
</section>
<section id="adaptive-partial-aggregation">
<h2>Adaptive Partial Aggregation<a class="headerlink" href="#adaptive-partial-aggregation" title="Link to this heading">#</a></h2>
@@ -1131,13 +1130,17 @@ back to Spark for shuffle operations.</p>
<section id="native-shuffle">
<h4>Native Shuffle<a class="headerlink" href="#native-shuffle" title="Link to this heading">#</a></h4>
<p>Comet provides a fully native shuffle implementation, which generally provides the best performance. Native shuffle
-supports <code class="docutils literal notranslate"><span class="pre">HashPartitioning</span></code>, <code class="docutils literal notranslate"><span class="pre">RangePartitioning</span></code> and <code class="docutils literal notranslate"><span class="pre">SinglePartitioning</span></code> but currently only supports primitive type
-partitioning keys. Columns that are not partitioning keys may contain complex types like maps, structs, and arrays.</p>
+supports <code class="docutils literal notranslate"><span class="pre">HashPartitioning</span></code>, <code class="docutils literal notranslate"><span class="pre">RangePartitioning</span></code>, and <code class="docutils literal notranslate"><span class="pre">SinglePartition</span></code>, plus <code class="docutils literal notranslate"><span class="pre">RoundRobinPartitioning</span></code> when enabled
+(see <a class="reference internal" href="compatibility/operators.html#round-robin-partitioning"><span class="std std-ref">Round-Robin Partitioning</span></a>). Range partitioning keys must be
+scalar types. Hash partitioning keys must be scalar types unless
+<code class="docutils literal notranslate"><span class="pre">spark.comet.shuffle.native.partitioning.hash.nested.enabled=true</span></code>, which also admits struct, array, and (Spark 4.0
+and later) map keys. That setting is disabled by default until the performance of the nested hashing paths has been
+measured. Columns that are not partitioning keys may contain complex types like maps, structs, and arrays.</p>
</section>
<section id="columnar-jvm-shuffle">
<h4>Columnar (JVM) Shuffle<a class="headerlink" href="#columnar-jvm-shuffle" title="Link to this heading">#</a></h4>
<p>Comet Columnar shuffle is JVM-based and supports <code class="docutils literal notranslate"><span class="pre">HashPartitioning</span></code>, <code class="docutils literal notranslate"><span class="pre">RoundRobinPartitioning</span></code>, <code class="docutils literal notranslate"><span class="pre">RangePartitioning</span></code>, and
-<code class="docutils literal notranslate"><span class="pre">SinglePartitioning</span></code>. This shuffle implementation supports complex data types as partitioning keys.</p>
+<code class="docutils literal notranslate"><span class="pre">SinglePartition</span></code>. This shuffle implementation supports complex data types as partitioning keys.</p>
<p>By default, Comet will convert a Spark <code class="docutils literal notranslate"><span class="pre">ShuffleExchangeExec</span></code> to columnar shuffle even when the shuffle’s child is a
non-Comet (Spark) plan. The benefit is that the next query stage can start as native Comet execution, since the
shuffle output is already in Arrow format. The cost is a row to columnar conversion at the shuffle boundary on the
@@ -1148,17 +1151,18 @@ on Spark.</p>
</section>
<section id="automatic-revert-to-spark-shuffle">
<h4>Automatic Revert to Spark Shuffle<a class="headerlink" href="#automatic-revert-to-spark-shuffle" title="Link to this heading">#</a></h4>
-<p>When a Comet columnar shuffle ends up between two non-Comet operators (for example, a partial/final hash aggregate
-pair that Comet could not convert), Comet reverts it to Spark’s built-in shuffle. Keeping columnar shuffle between
-two row-based operators would add <code class="docutils literal notranslate"><span class="pre">row</span> <span class="pre">-></span> <span class="pre">Arrow</span> <span class="pre">-></span> <span class="pre">shuffle</span> <span class="pre">-></span> <span class="pre">Arrow</span> <span class="pre">-></span> <span class="pre">row</span></code> conversions with no Comet consumer on
-either side to benefit from columnar output.</p>
+<p>When a Comet columnar shuffle ends up between a partial and a final aggregate that Comet could not convert (both
+remain Spark <code class="docutils literal notranslate"><span class="pre">HashAggregateExec</span></code> or <code class="docutils literal notranslate"><span class="pre">ObjectHashAggregateExec</span></code> operators), Comet reverts it to Spark’s built-in shuffle.
+Keeping columnar shuffle between the two row-based aggregates would add <code class="docutils literal notranslate"><span class="pre">row</span> <span class="pre">-></span> <span class="pre">Arrow</span> <span class="pre">-></span> <span class="pre">shuffle</span> <span class="pre">-></span> <span class="pre">Arrow</span> <span class="pre">-></span> <span class="pre">row</span></code>
+conversions with no Comet consumer on either side to benefit from columnar output. Other shuffles between non-Comet
+operators are not reverted.</p>
<p>This shifts the affected shuffles from Comet’s off-heap memory pool back to the JVM execution memory pool. Clusters
tuned for a small JVM heap may see <code class="docutils literal notranslate"><span class="pre">ExternalSorter</span></code> spills on queries where this revert fires. Shuffle I/O may also
grow marginally because Spark’s row-based serializer generally compresses less well than Comet’s Arrow IPC format.</p>
<p>Each revert is logged at <code class="docutils literal notranslate"><span class="pre">INFO</span></code> level on the driver as <code class="docutils literal notranslate"><span class="pre">Reverting</span> <span class="pre">Comet</span> <span class="pre">columnar</span> <span class="pre">shuffle</span> <span class="pre">to</span> <span class="pre">Spark</span> <span class="pre">shuffle</span> <span class="pre">between</span> <span class="pre"><parent></span> <span class="pre">an [...]
<p>This optimization is enabled by default and can be disabled by setting
<code class="docutils literal notranslate"><span class="pre">spark.comet.shuffle.revertRedundantColumnar.enabled=false</span></code>, in which case Comet will keep the columnar shuffle
-even when both its parent and child are non-Comet operators.</p>
+even when both of those aggregates run on Spark.</p>
</section>
</section>
<section id="remote-shuffle-with-celeborn">
@@ -1249,26 +1253,28 @@ repeated columnar-to-row and row-to-columnar conversions that dominate stage run
<code class="docutils literal notranslate"><span class="pre">spark.comet.exec.transitionRevert.enabled=true</span></code> to have Comet revert the entire stage to Spark row execution
when the number of columnar-to-row transitions exceeds
<code class="docutils literal notranslate"><span class="pre">spark.comet.exec.transitionRevert.maxTransitions</span></code> (default <code class="docutils literal notranslate"><span class="pre">2</span></code>). This trades native execution of a small
-subset of operators for eliminating conversion overhead across the stage.</p>
-<section id="entire-plan-fallback-for-wide-or-deeply-nested-schemas">
-<h3>Entire-Plan Fallback for Wide or Deeply Nested Schemas<a class="headerlink" href="#entire-plan-fallback-for-wide-or-deeply-nested-schemas" title="Link to this heading">#</a></h3>
+subset of operators for eliminating conversion overhead across the stage. A stage is not reverted when it holds a
+native aggregate whose intermediate buffer Spark cannot exchange with Comet across a stage boundary, because
+reverting it would split that aggregate between the two engines.</p>
+<section id="wide-or-deeply-nested-schemas">
+<h3>Wide or Deeply Nested Schemas<a class="headerlink" href="#wide-or-deeply-nested-schemas" title="Link to this heading">#</a></h3>
<p>The cost of each conversion also grows sharply with schema shape: for wide or deeply nested schemas,
columnar-to-row conversion is especially expensive because the conversion work scales with the number of
columns and nested fields. If profiling shows these conversions dominating a query over such a schema, set
<code class="docutils literal notranslate"><span class="pre">spark.comet.exec.transitionRevert.enabled=true</span></code> and lower
-<code class="docutils literal notranslate"><span class="pre">spark.comet.exec.transitionRevert.maxTransitions</span></code> (default <code class="docutils literal notranslate"><span class="pre">2</span></code>) to <code class="docutils literal notranslate"><span class="pre">1</span></code>. Note that this causes the entire
-plan to fall back to Spark row-based execution — Comet removes its native operators rather than running a
+<code class="docutils literal notranslate"><span class="pre">spark.comet.exec.transitionRevert.maxTransitions</span></code> (default <code class="docutils literal notranslate"><span class="pre">2</span></code>) to <code class="docutils literal notranslate"><span class="pre">1</span></code>. Note that this reverts every stage that
+exceeds the threshold to Spark row-based execution — Comet removes the stage’s native operators rather than running a
mix of native and fallback operators joined by repeated conversions — which can be cheaper than paying the
expensive conversions again and again.</p>
</section>
</section>
<section id="metrics-overhead">
<h2>Metrics Overhead<a class="headerlink" href="#metrics-overhead" title="Link to this heading">#</a></h2>
-<p>Comet exposes rich native operator metrics for observability (see <a class="reference internal" href="metrics.html"><span class="std std-doc">Metrics</span></a>), but they are
-disabled by default because traversing the Spark plan on every task adds measurable overhead, and metrics
-require an external sink (for example Prometheus) to be useful. Enable them with
-<code class="docutils literal notranslate"><span class="pre">spark.comet.metrics.enabled=true</span></code> when you have a metrics sink configured. This setting must be applied
-before the <code class="docutils literal notranslate"><span class="pre">SparkSession</span></code> is created.</p>
+<p>The SQL metrics described in <a class="reference internal" href="metrics.html"><span class="std std-doc">Metrics</span></a> are always collected. Setting <code class="docutils literal notranslate"><span class="pre">spark.comet.metrics.enabled=true</span></code>
+additionally publishes plan-coverage counters (<code class="docutils literal notranslate"><span class="pre">operators.native</span></code>, <code class="docutils literal notranslate"><span class="pre">operators.spark</span></code>, <code class="docutils literal notranslate"><span class="pre">queries.planned</span></code>,
+<code class="docutils literal notranslate"><span class="pre">transitions</span></code>, and <code class="docutils literal notranslate"><span class="pre">acceleration.ratio</span></code>) through Spark’s metrics system under the <code class="docutils literal notranslate"><span class="pre">comet</span></code> source. It is disabled by
+default because it walks every executed plan on the driver after each query, and the counters are only useful with an
+external sink (for example Prometheus) configured. This setting must be applied before the <code class="docutils literal notranslate"><span class="pre">SparkSession</span></code> is created.</p>
</section>
<section id="explain-plan">
<h2>Explain Plan<a class="headerlink" href="#explain-plan" title="Link to this heading">#</a></h2>
@@ -1358,7 +1364,7 @@ fallback to Spark works, see <a class="reference internal" href="understanding-c
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#reducing-row-columnar-conversion-overhead">Reducing Row/Columnar Conversion Overhead</a><ul class="nav section-nav flex-column">
-<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#entire-plan-fallback-for-wide-or-deeply-nested-schemas">Entire-Plan Fallback for Wide or Deeply Nested Schemas</a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#wide-or-deeply-nested-schemas">Wide or Deeply Nested Schemas</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#metrics-overhead">Metrics Overhead</a></li>
diff --git a/user-guide/latest/understanding-comet-plans.html b/user-guide/latest/understanding-comet-plans.html
index 00c30fb81d..12c62cf450 100644
--- a/user-guide/latest/understanding-comet-plans.html
+++ b/user-guide/latest/understanding-comet-plans.html
@@ -1079,6 +1079,15 @@ by role. Names match what is shown in the plan output.</p>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">CometCsvNativeScan</span></code></p></td>
<td><p>CSV scan that runs entirely in Rust via DataFusion (experimental).</p></td>
</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">CometInMemoryTableScan</span></code></p></td>
+<td><p>JVM-side scan of a table cached in Comet’s Arrow format (experimental, disabled by default).</p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">CometLocalTableScan</span></code></p></td>
+<td><p>JVM-side scan of a <code class="docutils literal notranslate"><span class="pre">LocalTableScanExec</span></code> (disabled by default).</p></td>
+</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">CometEmptyRelation</span></code></p></td>
+<td><p>Empty native input that replaces <code class="docutils literal notranslate"><span class="pre">EmptyRelationExec</span></code> (Spark 4.0 and later).</p></td>
+</tr>
</tbody>
</table>
</div>
@@ -1130,12 +1139,27 @@ consecutively in a plan, they execute as a single fused block.</p>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">CometSortMergeJoin</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">SortMergeJoinExec</span></code></p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">CometWindow</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">CometWindowExec</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">WindowExec</span></code></p></td>
</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">CometWindowGroupLimitExec</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">WindowGroupLimitExec</span></code> (Spark 3.5 and later)</p></td>
+</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">CometSample</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">SampleExec</span></code> (sampling without replacement)</p></td>
+</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">CometTakeOrderedAndProject</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">TakeOrderedAndProjectExec</span></code></p></td>
</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">CometWriteFiles</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">WriteFilesExec</span></code> (Spark 4.0 and later, experimental native Parquet writes)</p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">CometNativeWrite</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">DataWritingCommandExec</span></code> (Spark 3.x, experimental native Parquet writes)</p></td>
+</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">CometIcebergWrite</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">IcebergWrite</span></code> (experimental native Iceberg data-file writes)</p></td>
+</tr>
</tbody>
</table>
</div>
@@ -1165,6 +1189,15 @@ consecutively in a plan, they execute as a single fused block.</p>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">CometSubqueryBroadcast</span></code></p></td>
<td><p>Companion to <code class="docutils literal notranslate"><span class="pre">CometBroadcastExchange</span></code> for dynamic partition pruning subqueries.</p></td>
</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">CometMapInBatch</span></code></p></td>
+<td><p>Runs <code class="docutils literal notranslate"><span class="pre">mapInArrow</span></code> / <code class="docutils literal notranslate"><span class="pre">mapInPandas</span></code> Python UDFs on Comet’s Arrow batches (experimental). See <a class="reference internal" href="pyarrow-udfs.html"><span class="std std-doc">PyArrow UDF Acceleration</span></a>.</p></td>
+</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">IcebergWrite</span></code></p></td>
+<td><p>Executor-side Iceberg data-file write in Comet’s split-operator Iceberg write plan (experimental). See <a class="reference internal" href="iceberg-writes.html"><span class="std std-doc">Iceberg Writes</span></a>.</p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">IcebergCommit</span></code></p></td>
+<td><p>Driver-side commit for Comet’s split-operator Iceberg write plan (experimental).</p></td>
+</tr>
</tbody>
</table>
</div>
@@ -1177,21 +1210,26 @@ use:</p>
<li><p><strong><code class="docutils literal notranslate"><span class="pre">CometExchange</span></code></strong> is the <strong>native shuffle</strong> path. The child must already
be a Comet operator producing columnar Arrow batches; the node calls
<code class="docutils literal notranslate"><span class="pre">executeColumnar()</span></code> on its child and the partition, encode, and compress
-steps run in Rust. Hash and range partitioning <strong>keys</strong> must be
-primitive types because the Rust hashing and ordering do not support complex
-types, but the data columns themselves can include <code class="docutils literal notranslate"><span class="pre">StructType</span></code>,
-<code class="docutils literal notranslate"><span class="pre">ArrayType</span></code>, and <code class="docutils literal notranslate"><span class="pre">MapType</span></code> since batches are serialized via the Arrow IPC
-writer.</p></li>
+steps run in Rust. Range partitioning <strong>keys</strong> must be scalar types. Hash
+partitioning keys must also be scalar types unless
+<code class="docutils literal notranslate"><span class="pre">spark.comet.shuffle.native.partitioning.hash.nested.enabled=true</span></code>, which
+admits struct, array, and (Spark 4.0 and later) map keys. The data columns
+themselves can include <code class="docutils literal notranslate"><span class="pre">StructType</span></code>, <code class="docutils literal notranslate"><span class="pre">ArrayType</span></code>, and <code class="docutils literal notranslate"><span class="pre">MapType</span></code> since
+batches are serialized via the Arrow IPC writer.</p></li>
<li><p><strong><code class="docutils literal notranslate"><span class="pre">CometColumnarExchange</span></code></strong> is the <strong>JVM columnar shuffle</strong> path. It accepts
either Spark row-based input or Comet columnar input, which makes it the
fallback when the child is not a Comet operator or when a hash/range key
-type is not supported by native shuffle (for example, collated strings). It
-is still preferred over Spark’s native shuffle when Comet shuffle is
-enabled.</p></li>
+type is not supported by native shuffle (for example, a struct or array
+hash key while nested hash keys are disabled). It is still preferred over
+Spark’s native shuffle when Comet shuffle is enabled. Keys with a
+non-default string collation are supported by neither path and use Spark’s
+shuffle.</p></li>
</ul>
<p>Both paths support the same set of partitioning schemes
(<code class="docutils literal notranslate"><span class="pre">HashPartitioning</span></code>, <code class="docutils literal notranslate"><span class="pre">RangePartitioning</span></code>, <code class="docutils literal notranslate"><span class="pre">RoundRobinPartitioning</span></code>,
-<code class="docutils literal notranslate"><span class="pre">SinglePartition</span></code>) and both can carry complex types in data columns.</p>
+<code class="docutils literal notranslate"><span class="pre">SinglePartition</span></code>) and both can carry complex types in data columns. Native
+round-robin partitioning is disabled by default; see
+<a class="reference internal" href="compatibility/operators.html#round-robin-partitioning"><span class="std std-ref">Round-Robin Partitioning</span></a>.</p>
<p>The choice between the two is automatic. See the
<a class="reference internal" href="tuning.html#shuffle"><span class="std std-ref">Tuning Guide shuffle section</span></a> for how to enable Comet
shuffle and choose between the implementations.</p>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org