You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/07/28 14:34:13 UTC
(datafusion-comet) branch asf-site updated: Publish built docs triggered by c27eed3398cd48fe8c87c417959ce28eaca5138c
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 6f309b4fb6 Publish built docs triggered by c27eed3398cd48fe8c87c417959ce28eaca5138c
6f309b4fb6 is described below
commit 6f309b4fb6fa98b0799d4b45565b75d1ddbc7342
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Tue Jul 28 14:34:09 2026 +0000
Publish built docs triggered by c27eed3398cd48fe8c87c417959ce28eaca5138c
---
.../adding_a_new_spark_version.md.txt | 2 +-
_sources/contributor-guide/bug_triage.md.txt | 13 +++--
.../expression-audits/agg_funcs.md.txt | 2 +-
.../expression-audits/array_funcs.md.txt | 6 +--
.../expression-audits/bitwise_funcs.md.txt | 2 +-
.../expression-audits/collection_funcs.md.txt | 8 +--
.../expression-audits/conversion_funcs.md.txt | 16 +++---
.../expression-audits/datetime_funcs.md.txt | 4 +-
.../expression-audits/json_funcs.md.txt | 4 +-
.../expression-audits/map_funcs.md.txt | 4 +-
.../expression-audits/math_funcs.md.txt | 12 ++---
.../expression-audits/predicate_funcs.md.txt | 4 +-
.../expression-audits/string_funcs.md.txt | 58 +++++++++++-----------
.../expression-audits/url_funcs.md.txt | 2 +-
.../expression-audits/window_funcs.md.txt | 2 +-
_sources/contributor-guide/roadmap.md.txt | 9 ++--
.../s3-credential-provider-design.md.txt | 2 +-
.../user-guide/latest/compatibility/scans.md.txt | 2 +-
_sources/user-guide/latest/datasources.md.txt | 2 +-
_sources/user-guide/latest/datatypes.md.txt | 20 +++++---
_sources/user-guide/latest/expressions.md.txt | 34 ++++++-------
_sources/user-guide/latest/operators.md.txt | 32 ++++++------
_sources/user-guide/latest/pyarrow-udfs.md.txt | 7 +--
_sources/user-guide/latest/tuning.md.txt | 2 +-
contributor-guide/adding_a_new_spark_version.html | 2 +-
contributor-guide/bug_triage.html | 13 +++--
contributor-guide/expression-audits/agg_funcs.html | 2 +-
.../expression-audits/array_funcs.html | 6 +--
.../expression-audits/bitwise_funcs.html | 2 +-
.../expression-audits/collection_funcs.html | 8 +--
.../expression-audits/conversion_funcs.html | 16 +++---
.../expression-audits/datetime_funcs.html | 4 +-
.../expression-audits/json_funcs.html | 4 +-
contributor-guide/expression-audits/map_funcs.html | 4 +-
.../expression-audits/math_funcs.html | 12 ++---
.../expression-audits/predicate_funcs.html | 4 +-
.../expression-audits/string_funcs.html | 58 +++++++++++-----------
contributor-guide/expression-audits/url_funcs.html | 2 +-
.../expression-audits/window_funcs.html | 2 +-
contributor-guide/roadmap.html | 6 +--
.../s3-credential-provider-design.html | 2 +-
searchindex.js | 2 +-
user-guide/latest/compatibility/scans.html | 2 +-
user-guide/latest/datasources.html | 2 +-
user-guide/latest/datatypes.html | 20 +++++---
user-guide/latest/expressions.html | 34 ++++++-------
user-guide/latest/operators.html | 8 +--
user-guide/latest/pyarrow-udfs.html | 7 +--
user-guide/latest/tuning.html | 2 +-
49 files changed, 244 insertions(+), 229 deletions(-)
diff --git a/_sources/contributor-guide/adding_a_new_spark_version.md.txt b/_sources/contributor-guide/adding_a_new_spark_version.md.txt
index 65109e4d0c..45c9ebc15c 100644
--- a/_sources/contributor-guide/adding_a_new_spark_version.md.txt
+++ b/_sources/contributor-guide/adding_a_new_spark_version.md.txt
@@ -116,7 +116,7 @@ version from blocking unrelated PRs.
When CI capacity is constrained (the macOS runners in particular), it is
acceptable to drop an older minor version from the macOS PR matrix while a
-preview version is being stabilized. PR #4104 ("ci: reduce macOS PR matrix
+preview version is being stabilized. PR [#4104](https://github.com/apache/datafusion-comet/pull/4104) ("ci: reduce macOS PR matrix
to single Spark 4.0 profile") is a precedent for this kind of trim.
### What to Avoid in Stage 1
diff --git a/_sources/contributor-guide/bug_triage.md.txt b/_sources/contributor-guide/bug_triage.md.txt
index 23a385a6ad..b2bacd7567 100644
--- a/_sources/contributor-guide/bug_triage.md.txt
+++ b/_sources/contributor-guide/bug_triage.md.txt
@@ -162,14 +162,17 @@ Common patterns include:
Several bugs relate to how Comet plans hash aggregates across stage boundaries. The key issue is
that Spark's AQE may materialize a Comet partial aggregate but then run the final aggregate in
-Spark (or vice versa), and the intermediate formats may not be compatible. See the
-[EPIC #2892](https://github.com/apache/datafusion-comet/issues/2892) for the full picture.
+Spark (or vice versa), and the intermediate formats may not be compatible. The closed
+EPIC [#2892](https://github.com/apache/datafusion-comet/issues/2892) collects the historical
+reports and is a good starting point when a new one comes in.
### Native Writer Bugs
-The native Parquet writer has a cluster of known test failures tracked as individual issues
-(#3417–#3430). These are lower priority since the native writer is still maturing, but they
-should be addressed before the writer is promoted to production-ready status.
+The native Parquet writer has a cluster of known test failures tracked as individual issues, from
+[#3417](https://github.com/apache/datafusion-comet/issues/3417) to
+[#3430](https://github.com/apache/datafusion-comet/issues/3430). These are lower priority since the
+native writer is still maturing, but they should be addressed before the writer is promoted to
+production-ready status.
## How to Help with Triage
diff --git a/_sources/contributor-guide/expression-audits/agg_funcs.md.txt b/_sources/contributor-guide/expression-audits/agg_funcs.md.txt
index 63f4ca84b7..37c8e9d63a 100644
--- a/_sources/contributor-guide/expression-audits/agg_funcs.md.txt
+++ b/_sources/contributor-guide/expression-audits/agg_funcs.md.txt
@@ -73,6 +73,6 @@
- Spark 3.5.8 (audited 2026-06-24): ordering centralized via `PhysicalDataType.ordering`; behavior identical to 3.4.3.
- Spark 4.0.1 (audited 2026-06-24): adds `PercentileCont`/`PercentileDisc` builders and `SupportsOrderingWithinGroup`, enabling `percentile_cont(p) WITHIN GROUP (ORDER BY col)`, which rewrites to `Percentile(col, p, reverse)`. The ascending form runs natively; the `DESC` form sets `reverse = true` and falls back to Spark because the native `percentile_cont` always interpolates in ascending order.
- Spark 4.1.1 (audited 2026-06-24): identical to 4.0.1.
-- `CometPercentile` reports `Incompatible` for the otherwise-supported form because DataFusion's `percentile_cont` quantizes the interpolation weight to 6 decimal places (`INTERPOLATION_PRECISION = 1e6`), so a deeply-interpolated value can differ from Spark by up to roughly `(upper - lower) * 1e-6`. The native path is opt-in via `spark.comet.expression.Percentile.allowIncompatible=true` ([#4719](https://github.com/apache/datafusion-comet/issues/4719)).
+- `CometPercentile` reports `Compatible` for the single-literal-percentage, default-frequency, numeric-input, ascending form and runs it natively by default. Every other form is `Unsupported` and falls back to Spark.
[Spark Expression Support]: ../../user-guide/latest/expressions.md
diff --git a/_sources/contributor-guide/expression-audits/array_funcs.md.txt b/_sources/contributor-guide/expression-audits/array_funcs.md.txt
index bd10f55c23..31261b2542 100644
--- a/_sources/contributor-guide/expression-audits/array_funcs.md.txt
+++ b/_sources/contributor-guide/expression-audits/array_funcs.md.txt
@@ -24,7 +24,7 @@
## array
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
-- Spark 3.5.8 (audited 2026-05-27): baseline. `CreateArray(children, useStringTypeWhenEmpty)`; element type is the common type of children. Comet routes via `CometCreateArray` (native `make_array`) and special-cases the empty-array case to dodge a known DataFusion `coerce_types` issue (#3338).
+- Spark 3.5.8 (audited 2026-05-27): baseline. `CreateArray(children, useStringTypeWhenEmpty)`; element type is the common type of children. Comet routes via `CometCreateArray` (native `make_array`) and special-cases the empty-array case to dodge a known DataFusion `coerce_types` issue ([#3338](https://github.com/apache/datafusion-comet/issues/3338)).
- Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
- Spark 4.1.1 (audited 2026-05-27): adds `contextIndependentFoldable` override; runtime semantics unchanged.
@@ -84,8 +84,8 @@
## array_join
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
-- Spark 3.5.8 (audited 2026-05-27): baseline. `ArrayJoin(array, delimiter, nullReplacement)`. Comet routes via `CometArrayJoin` to DataFusion's `array_to_string` and is unconditionally flagged `Incompatible` ("Null handling may differ from Spark", #3178).
-- Spark 4.0.1 (audited 2026-05-27): `inputTypes` widened to `AbstractArrayType(StringTypeWithCollation(supportsTrimCollation = true))`; non-binary collations not propagated (https://github.com/apache/datafusion-comet/issues/2190).
+- Spark 3.5.8 (audited 2026-05-27): baseline. `ArrayJoin(array, delimiter, nullReplacement)`. Comet routes via `CometArrayJoin` to DataFusion's `array_to_string` and is unconditionally flagged `Incompatible` ("Null handling may differ from Spark", [#3178](https://github.com/apache/datafusion-comet/issues/3178)).
+- Spark 4.0.1 (audited 2026-05-27): `inputTypes` widened to `AbstractArrayType(StringTypeWithCollation(supportsTrimCollation = true))`; non-binary collations not propagated ([#2190](https://github.com/apache/datafusion-comet/issues/2190)).
- Spark 4.1.1 (audited 2026-05-27): adds `contextIndependentFoldable` override; runtime unchanged.
## array_max
diff --git a/_sources/contributor-guide/expression-audits/bitwise_funcs.md.txt b/_sources/contributor-guide/expression-audits/bitwise_funcs.md.txt
index 56967b60fd..60d93c32ba 100644
--- a/_sources/contributor-guide/expression-audits/bitwise_funcs.md.txt
+++ b/_sources/contributor-guide/expression-audits/bitwise_funcs.md.txt
@@ -80,7 +80,7 @@
## shiftrightunsigned
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
-- Spark 3.5.8 (audited 2026-05-27): baseline. `ShiftRightUnsigned(left, right) extends BinaryExpression`; uses Java's `>>>` (zero-fill) semantics. Comet wires as `CometScalarFunction("shiftrightunsigned")` (added in #4375).
+- Spark 3.5.8 (audited 2026-05-27): baseline. `ShiftRightUnsigned(left, right) extends BinaryExpression`; uses Java's `>>>` (zero-fill) semantics. Comet wires as `CometScalarFunction("shiftrightunsigned")` (added in [#4375](https://github.com/apache/datafusion-comet/pull/4375)).
- Spark 4.0.1 (audited 2026-05-27): now extends the shared `BitShiftOperation` trait; eval semantics unchanged. The `>>>` operator alias is added at the parser level.
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
diff --git a/_sources/contributor-guide/expression-audits/collection_funcs.md.txt b/_sources/contributor-guide/expression-audits/collection_funcs.md.txt
index 94faddcf4a..bcdebc165f 100644
--- a/_sources/contributor-guide/expression-audits/collection_funcs.md.txt
+++ b/_sources/contributor-guide/expression-audits/collection_funcs.md.txt
@@ -31,7 +31,7 @@
- Spark 3.5.8 (audited 2026-05-27): baseline. `Concat(children) extends ComplexTypeMergingExpression with QueryErrorsBase`; `allowedTypes = Seq(StringType, BinaryType, ArrayType)`; result type is the merged child type. Empty children is allowed and returns the empty string of the result type.
- Spark 4.0.1 (audited 2026-05-27): `allowedTypes` widens `StringType` to `StringTypeWithCollation(supportsTrimCollation = true)`. Error-formatting helper changes from `paramIndex` to `ordinalNumber`. Runtime semantics unchanged for `UTF8_BINARY`.
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- Known limitation: Comet only supports `StringType` children natively; `BinaryType` and `ArrayType` inputs fall back to Spark (https://github.com/apache/datafusion-comet/issues/4471). Non-default Spark 4.0 string collations are not propagated (https://github.com/apache/datafusion-comet/issues/2190).
+- Known limitation: Comet only supports `StringType` children natively; `BinaryType` and `ArrayType` inputs fall back to Spark ([#4471](https://github.com/apache/datafusion-comet/issues/4471)). Non-default Spark 4.0 string collations are not propagated ([#2190](https://github.com/apache/datafusion-comet/issues/2190)).
## reverse
@@ -39,7 +39,7 @@
- Spark 3.5.8 (audited 2026-05-27): baseline. `Reverse(child) extends UnaryExpression with ImplicitCastInputTypes with NullIntolerant`; `inputTypes = Seq(TypeCollection(StringType, ArrayType))`; `dataType = child.dataType`. For string, calls `UTF8String.reverse()`; for array, reverses element order in-place via `GenericArrayData`.
- Spark 4.0.1 (audited 2026-05-27): `NullIntolerant` trait replaced by `override def nullIntolerant: Boolean = true`; `inputTypes` widened to `Seq(TypeCollection(StringTypeWithCollation(supportsTrimCollation = true), ArrayType))`. Semantics unchanged for `UTF8_BINARY`.
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- Known limitation: `Reverse` on an array containing `BinaryType` elements is reported as `Incompatible` and falls back unless explicitly enabled (https://github.com/apache/datafusion-comet/issues/2763).
+- Known limitation: `Reverse` on an array containing `BinaryType` elements is reported as `Incompatible` and falls back unless explicitly enabled ([#2763](https://github.com/apache/datafusion-comet/issues/2763)).
## size
@@ -47,7 +47,7 @@
- Spark 3.5.8 (audited 2026-05-27): baseline. `Size(child, legacySizeOfNull) extends UnaryExpression with ExpectsInputTypes`; `inputTypes = Seq(TypeCollection(ArrayType, MapType)) -> IntegerType`. `legacySizeOfNull=true` returns `-1` for NULL input; `false` returns NULL. Comet routes via `CometSize`, which emits a `CaseWhen(isNotNull(child), size_scalar(child), Literal(legacySizeOfNull))`.
- Spark 4.0.1 (audited 2026-05-27): byte-for-byte identical to 3.5.8.
- Spark 4.1.1 (audited 2026-05-27): byte-for-byte identical to 3.5.8.
-- Known limitation: `Size` over `MapType` falls back to Spark (https://github.com/apache/datafusion-comet/issues/4472).
-- Performance (tuned 2026-07-10, PR #4877): compute list row sizes from the offset buffer instead of allocating a sliced `ArrayRef` per row via `list_array.value(i)`, removing one heap allocation per row. ~94% faster. Benchmark: `benches/array_size.rs`.
+- Both `ArrayType` and `MapType` inputs are `Compatible` and run natively; every other child type is `Unsupported`.
+- Performance (tuned 2026-07-10, PR [#4877](https://github.com/apache/datafusion-comet/pull/4877)): compute list row sizes from the offset buffer instead of allocating a sliced `ArrayRef` per row via `list_array.value(i)`, removing one heap allocation per row. ~94% faster. Benchmark: `benches/array_size.rs`.
[Spark Expression Support]: ../../user-guide/latest/expressions.md
diff --git a/_sources/contributor-guide/expression-audits/conversion_funcs.md.txt b/_sources/contributor-guide/expression-audits/conversion_funcs.md.txt
index 3a2c4a7553..c11d441a23 100644
--- a/_sources/contributor-guide/expression-audits/conversion_funcs.md.txt
+++ b/_sources/contributor-guide/expression-audits/conversion_funcs.md.txt
@@ -28,14 +28,14 @@
- Spark 4.0.1 (audited 2026-05-27): `VariantType` added; `StringType` literals replaced with `_: StringType` to accommodate collated strings. `(TimestampType, ByteType|ShortType|IntegerType)` added to `canAnsiCast`. `NullIntolerant` -> `nullIntolerant: Boolean` refactor. New `ToPrettyString.BinaryFormatter` semantics for `Binary -> String` are replicated natively via `spark_binary_formatter`. Numeric-to-numeric matrix unchanged.
- Spark 4.1.1 (audited 2026-05-27): `TimeType` added; many `TimeType` arms in `canCast`/`canAnsiCast`. Geospatial `GeographyType` / `GeometryType` types added with their own conversion rules. Numeric-to-numeric matrix unchanged.
- Known divergences and gaps:
- - `CAST(<binary> AS STRING)` decodes the bytes with `decode_utf8_spark_lossy`, replacing invalid UTF-8 with `U+FFFD` exactly as the JVM's `new String(bytes, UTF_8)` does (the same decoder as the native shuffle). It matches Spark's rendered output, but decoding is not byte-preserving, so it diverges from Spark for operations that read the underlying bytes: round-trips such as `CAST(CAST(x AS STRING) AS BINARY)`, and value identity, since distinct ill-formed byte sequences all decode to [...]
- - Spark 4.0 collated `StringType` is not explicitly guarded; pattern equality is expected to keep collated-string casts falling back, but there is no test (https://github.com/apache/datafusion-comet/issues/4489; umbrella #2190).
- - Spark 4.1 `TimeType` casts have no explicit `Unsupported` arm; they fall back implicitly but do not appear in the auto-generated compatibility doc (https://github.com/apache/datafusion-comet/issues/4490).
- - `CAST(<map> AS <map>)` falls back to Spark even though native `cast_map_to_map` exists (https://github.com/apache/datafusion-comet/issues/4491).
- - `spark.sql.legacy.castComplexTypesToString.enabled=true` is not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4492).
- - `CAST(<float|double> AS DECIMAL)` rounding may differ from Spark (`Incompatible`, gated by `spark.comet.expression.Cast.allowIncompatible`, tracked at https://github.com/apache/datafusion-comet/issues/1371).
+ - `CAST(<binary> AS STRING)` decodes the bytes with `decode_utf8_spark_lossy`, replacing invalid UTF-8 with `U+FFFD` exactly as the JVM's `new String(bytes, UTF_8)` does (the same decoder as the native shuffle). It matches Spark's rendered output, but decoding is not byte-preserving, so it diverges from Spark for operations that read the underlying bytes: round-trips such as `CAST(CAST(x AS STRING) AS BINARY)`, and value identity, since distinct ill-formed byte sequences all decode to [...]
+ - Spark 4.0 collated `StringType` is not explicitly guarded; pattern equality is expected to keep collated-string casts falling back, but there is no test ([#4489](https://github.com/apache/datafusion-comet/issues/4489); umbrella [#2190](https://github.com/apache/datafusion-comet/issues/2190)).
+ - Spark 4.1 `TimeType` casts have no explicit `Unsupported` arm; they fall back implicitly but do not appear in the auto-generated compatibility doc ([#4490](https://github.com/apache/datafusion-comet/issues/4490)).
+ - `CAST(<map> AS <map>)` runs natively via `cast_map_to_map`; `CometCast.isSupported` recurses into the key and value casts, so the map arm inherits whichever support level the inner casts report.
+ - `spark.sql.legacy.castComplexTypesToString.enabled=true` is `Unsupported` for any array/map/struct-to-string cast, so those casts fall back to Spark while the flag is on. Comet only implements the default (`{}`-wrapped, NULL-rendering) formatting. The flag is internal from Spark 4.0 onward and defaults to false.
+ - `CAST(<float|double> AS DECIMAL)` rounding may differ from Spark (`Incompatible`, gated by `spark.comet.expression.Cast.allowIncompatible`, tracked at [#1371](https://github.com/apache/datafusion-comet/issues/1371)).
- Spark registers the type-name conversion functions (`bigint`, `binary`, `boolean`, `date`, `decimal`, `double`, `float`, `int`, `smallint`, `string`, `timestamp`, `tinyint`) as cast aliases. Each lowers to the same `Cast` node, so Comet handles it via the `cast` implementation with the same compatibility profile.
-- Performance (tuned 2026-07-14, PR #4920): narrowing integer casts (`spark_cast_int_to_int`) map the values buffer in a single pass with Arrow `unary`/`try_unary` and carry the null buffer over untouched, replacing an element-by-element `Option`/`Result` iterator-collect. Up to 100x faster on narrowing casts. Benchmark: `benches/cast_numeric.rs`.
-- Performance (tuned 2026-07-15, PR #4940): float/double-to-decimal casts (`cast_floating_point_to_decimal128`) now convert in a single vectorized `unary_opt` pass that maps out-of-range values (NaN, infinity, precision overflow) to null, replacing the per-element `Decimal128Builder` loop. ANSI raises via an O(1) null-count check plus a rare element-wise rescan. 15-36% faster with no regression on any shape. Benchmark: `benches/cast_float_to_decimal.rs`.
+- Performance (tuned 2026-07-14, PR [#4920](https://github.com/apache/datafusion-comet/pull/4920)): narrowing integer casts (`spark_cast_int_to_int`) map the values buffer in a single pass with Arrow `unary`/`try_unary` and carry the null buffer over untouched, replacing an element-by-element `Option`/`Result` iterator-collect. Up to 100x faster on narrowing casts. Benchmark: `benches/cast_numeric.rs`.
+- Performance (tuned 2026-07-15, PR [#4940](https://github.com/apache/datafusion-comet/pull/4940)): float/double-to-decimal casts (`cast_floating_point_to_decimal128`) now convert in a single vectorized `unary_opt` pass that maps out-of-range values (NaN, infinity, precision overflow) to null, replacing the per-element `Decimal128Builder` loop. ANSI raises via an O(1) null-count check plus a rare element-wise rescan. 15-36% faster with no regression on any shape. Benchmark: `benches/cast [...]
[Spark Expression Support]: ../../user-guide/latest/expressions.md
diff --git a/_sources/contributor-guide/expression-audits/datetime_funcs.md.txt b/_sources/contributor-guide/expression-audits/datetime_funcs.md.txt
index aad1945f5e..9ff3eb3a98 100644
--- a/_sources/contributor-guide/expression-audits/datetime_funcs.md.txt
+++ b/_sources/contributor-guide/expression-audits/datetime_funcs.md.txt
@@ -42,7 +42,7 @@
- Spark 3.4.3 (audited 2026-05-12): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-12): baseline.
- Spark 4.0.1 (audited 2026-05-12): `inputTypes` widened to `StringTypeWithCollation`; behaviour unchanged for ASCII timezone strings.
-- Marked `Incompatible`: Comet's native timezone parser only accepts IANA zone IDs (e.g. `America/Los_Angeles`) and fixed `+HH:MM` offsets, while Spark also accepts legacy forms (`GMT+1`, `UTC+1`, three-letter abbreviations like `PST`). By default it runs through the codegen dispatcher (Spark-correct) and uses the native path only when incompatible expressions are explicitly allowed, where legacy zone forms throw a native parse error at execution (https://github.com/apache/datafusion-com [...]
+- Marked `Incompatible`: Comet's native timezone parser only accepts IANA zone IDs (e.g. `America/Los_Angeles`) and fixed `+HH:MM` offsets, while Spark also accepts legacy forms (`GMT+1`, `UTC+1`, three-letter abbreviations like `PST`). By default it runs through the codegen dispatcher (Spark-correct) and uses the native path only when incompatible expressions are explicitly allowed, where legacy zone forms throw a native parse error at execution.
## make_timestamp_ltz
@@ -81,7 +81,7 @@
- Spark 3.4.3 (audited 2026-05-12): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-12): baseline.
- Spark 4.0.1 (audited 2026-05-12): `inputTypes` widened to `StringTypeWithCollation`; behaviour unchanged for ASCII timezone strings.
-- Marked `Incompatible`: Comet's native timezone parser only accepts IANA zone IDs (e.g. `America/Los_Angeles`) and fixed `+HH:MM` offsets, while Spark also accepts legacy forms (`GMT+1`, `UTC+1`, three-letter abbreviations like `PST`). By default it runs through the codegen dispatcher (Spark-correct) and uses the native path only when incompatible expressions are explicitly allowed, where legacy zone forms throw a native parse error at execution (https://github.com/apache/datafusion-com [...]
+- Marked `Incompatible`: Comet's native timezone parser only accepts IANA zone IDs (e.g. `America/Los_Angeles`) and fixed `+HH:MM` offsets, while Spark also accepts legacy forms (`GMT+1`, `UTC+1`, three-letter abbreviations like `PST`). By default it runs through the codegen dispatcher (Spark-correct) and uses the native path only when incompatible expressions are explicitly allowed, where legacy zone forms throw a native parse error at execution.
## try_make_timestamp
diff --git a/_sources/contributor-guide/expression-audits/json_funcs.md.txt b/_sources/contributor-guide/expression-audits/json_funcs.md.txt
index 9518cc6f70..8a027e3679 100644
--- a/_sources/contributor-guide/expression-audits/json_funcs.md.txt
+++ b/_sources/contributor-guide/expression-audits/json_funcs.md.txt
@@ -31,7 +31,7 @@
- Spark 3.5.8 (audited 2026-05-27): baseline. `BinaryExpression with ExpectsInputTypes with CodegenFallback`; `inputTypes = Seq(StringType, StringType) -> StringType`. Eval is inline and uses Jackson with `RawStyle` output. Foldable paths are parsed once. Returns NULL for invalid JSON, missing paths, or `JsonProcessingException`.
- Spark 4.0.1 (audited 2026-05-27): the eval is extracted into a `GetJsonObjectEvaluator` helper (no behaviour change). The trait set now mixes in `DefaultStringProducingExpression`, and `inputTypes` is widened to `StringTypeWithCollation(supportsTrimCollation = true)` for both arguments.
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- Known incompatibility: Spark accepts single-quoted JSON and unescaped control characters; Comet's native parser (built on `serde_json`) rejects both, so those inputs require `spark.comet.expression.GetJsonObject.allowIncompatible=true` and may still produce different results. Non-default Spark 4.0 string collations are not propagated (https://github.com/apache/datafusion-comet/issues/2190).
+- Known incompatibility: Spark accepts single-quoted JSON and unescaped control characters; Comet's native parser (built on `serde_json`) rejects both, so those inputs require `spark.comet.expression.GetJsonObject.allowIncompatible=true` and may still produce different results. Non-default Spark 4.0 string collations are not propagated ([#2190](https://github.com/apache/datafusion-comet/issues/2190)).
## json_array_length
@@ -42,6 +42,6 @@
## to_json
- Partial native support; options and map/array inputs fall back.
-- Performance (tuned 2026-07-13, PR #4902): `escape_string` returns `Cow` (zero-alloc borrow when nothing needs escaping) and bulk-copies unescaped byte runs instead of pushing char-by-char per value. 2x faster. Benchmark: `benches/to_json.rs`.
+- Performance (tuned 2026-07-13, PR [#4902](https://github.com/apache/datafusion-comet/pull/4902)): `escape_string` returns `Cow` (zero-alloc borrow when nothing needs escaping) and bulk-copies unescaped byte runs instead of pushing char-by-char per value. 2x faster. Benchmark: `benches/to_json.rs`.
[Spark Expression Support]: ../../user-guide/latest/expressions.md
diff --git a/_sources/contributor-guide/expression-audits/map_funcs.md.txt b/_sources/contributor-guide/expression-audits/map_funcs.md.txt
index 2ceeb7b68b..ea13e6ab13 100644
--- a/_sources/contributor-guide/expression-audits/map_funcs.md.txt
+++ b/_sources/contributor-guide/expression-audits/map_funcs.md.txt
@@ -45,7 +45,7 @@
## map_from_arrays
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
-- Spark 3.5.8 (audited 2026-05-27): baseline. `MapFromArrays(left, right) extends BinaryExpression with NullIntolerant`; Spark uses `ArrayBasedMapBuilder` to detect duplicate keys (subject to `spark.sql.mapKeyDedupPolicy`) and rejects null keys with `RuntimeException("Cannot use null as map key")`. Comet `CometMapFromArrays` wraps the inputs in `CaseWhen(IsNotNull(left) AND IsNotNull(right), map(left, right), null)` so NULL-array inputs return NULL rather than triggering the previously r [...]
+- Spark 3.5.8 (audited 2026-05-27): baseline. `MapFromArrays(left, right) extends BinaryExpression with NullIntolerant`; Spark uses `ArrayBasedMapBuilder` to detect duplicate keys (subject to `spark.sql.mapKeyDedupPolicy`) and rejects null keys with `RuntimeException("Cannot use null as map key")`. Comet `CometMapFromArrays` wraps the inputs in `CaseWhen(IsNotNull(left) AND IsNotNull(right), map(left, right), null)` so NULL-array inputs return NULL rather than triggering the previously r [...]
- Spark 4.0.1 (audited 2026-05-27): semantics unchanged; `NullIntolerant` trait replaced by `nullIntolerant: Boolean`.
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
@@ -76,6 +76,6 @@
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. `StringToMap(text, pairDelim, keyValueDelim) extends TernaryExpression`; splits `text` on `pairDelim`, then each pair on `keyValueDelim` (default `","` and `":"`). Uses `ArrayBasedMapBuilder` for duplicate-key handling. Wired as `CometScalarFunction("str_to_map")`.
- Spark 4.0.1 (audited 2026-05-27): `inputTypes` widened to `StringTypeNonCSAICollation`; uses `CollationAwareUTF8String.splitSQL` with a `collationId`. Runtime unchanged for `UTF8_BINARY`.
-- Spark 4.1.1 (audited 2026-05-27): adds the `legacySplitTruncate` flag (driven by `spark.sql.legacy.truncateForEmptyRegexSplit`) to both `splitSQL` calls (https://github.com/apache/datafusion-comet/issues/4477). The Comet native impl does not honour this flag; behaviour matches the non-legacy default.
+- Spark 4.1.1 (audited 2026-05-27): adds the `legacySplitTruncate` flag (driven by `spark.sql.legacy.truncateForEmptyRegexSplit`) to both `splitSQL` calls. The Comet native impl always behaves as if the flag were false, so `CometStrToMap` reads the config by string key and reports `Incompatible` when it is enabled; the `CodegenDispatchFallback` trait then routes the expression through the JVM codegen dispatcher rather than falling the whole projection back to Spark. Non-UTF8_BINARY colla [...]
[Spark Expression Support]: ../../user-guide/latest/expressions.md
diff --git a/_sources/contributor-guide/expression-audits/math_funcs.md.txt b/_sources/contributor-guide/expression-audits/math_funcs.md.txt
index 6c9607efc5..87bfa27cb8 100644
--- a/_sources/contributor-guide/expression-audits/math_funcs.md.txt
+++ b/_sources/contributor-guide/expression-audits/math_funcs.md.txt
@@ -23,7 +23,7 @@
## `%`
-- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `Remainder(left, right, evalMode)` signature identical across versions. Native path uses Rust `spark_modulo` UDF; non-ANSI returns NULL on divide-by-zero, ANSI raises `DIVIDE_BY_ZERO` / `REMAINDER_BY_ZERO`. `CometRemainder` rejects `EvalMode.TRY`, so `try_mod` (Spark 4.0+) falls back to Spark (https://github.com/apache/datafusion-comet/issues/4484).
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `Remainder(left, right, evalMode)` signature identical across versions. Native path uses Rust `spark_modulo` UDF; non-ANSI returns NULL on divide-by-zero, ANSI raises `DIVIDE_BY_ZERO` / `REMAINDER_BY_ZERO`. `CometRemainder` gates only on the left input's data type, so all three eval modes, including the `EvalMode.TRY` form used by `try_mod` (Spark 4.0+), are serialized natively.
## `*`
@@ -45,7 +45,7 @@
Internal decimal wrapper emitted around every decimal `+ - * /`, `sum`, and `avg` result to null out (non-ANSI) or raise on (ANSI) values that exceed the declared precision. Native impl: `math_funcs/internal/checkoverflow.rs`.
-- Performance (tuned 2026-07-15, PR #4937): both the ANSI and non-ANSI paths share a no-overflow fast path built on `is_valid_decimal_precision`, a small inlined bounds check scanned with `all` (short-circuits at the first overflow). When nothing overflows (the common shape) the input buffers are reused via `to_data()` (cheap Arc metadata clone) instead of allocating through `null_if_overflow_precision` (non-ANSI) or running the heavier per-value `validate_decimal_precision` (ANSI). The [...]
+- Performance (tuned 2026-07-15, PR [#4937](https://github.com/apache/datafusion-comet/pull/4937)): both the ANSI and non-ANSI paths share a no-overflow fast path built on `is_valid_decimal_precision`, a small inlined bounds check scanned with `all` (short-circuits at the first overflow). When nothing overflows (the common shape) the input buffers are reused via `to_data()` (cheap Arc metadata clone) instead of allocating through `null_if_overflow_precision` (non-ANSI) or running the hea [...]
## abs
@@ -123,7 +123,7 @@ Internal decimal wrapper emitted around every decimal `+ - * /`, `sum`, and `avg
Internal fused expression that rescales a Decimal128 value (changing scale) and checks output precision in one pass, replacing the `CheckOverflow(Cast(expr, Decimal128(p, s)))` pattern used by decimal-to-decimal casts. Native impl: `math_funcs/internal/decimal_rescale_check.rs`.
-- Performance (tuned 2026-07-15, PR #4938): the legacy path ran `null_if_overflow_precision` (a second full pass that allocates a new array) on every batch to turn overflow sentinels into nulls, even when nothing overflowed. Now that pass runs only when a sentinel is present (`contains(&i128::MAX)`, short-circuiting), so the common no-overflow case skips the allocation. 8 to 26% faster on no-overflow shapes; overflow and ANSI shapes unchanged. Benchmark: `benches/decimal_rescale.rs`.
+- Performance (tuned 2026-07-15, PR [#4938](https://github.com/apache/datafusion-comet/pull/4938)): the legacy path ran `null_if_overflow_precision` (a second full pass that allocates a new array) on every batch to turn overflow sentinels into nulls, even when nothing overflowed. Now that pass runs only when a sentinel is present (`contains(&i128::MAX)`, short-circuiting), so the common no-overflow case skips the allocation. 8 to 26% faster on no-overflow shapes; overflow and ANSI shapes [...]
## e
@@ -227,7 +227,7 @@ Internal fused expression that rescales a Decimal128 value (changing scale) and
## shiftleft
-- See `bitwise_funcs / <<` (audited in PR #4479). Same support as the operator alias added in 4.0.
+- See `bitwise_funcs / <<` (audited in PR [#4479](https://github.com/apache/datafusion-comet/pull/4479)). Same support as the operator alias added in 4.0.
## sign
@@ -276,12 +276,12 @@ Internal fused expression that rescales a Decimal128 value (changing scale) and
## unhex
- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `Unhex(child, failOnError)`. Spark 4.x widens input to `StringTypeWithCollation` and wraps the inner call in try/catch; Comet `CometUnhex` forwards `failOnError` to native `spark_unhex` but does not gate on collation.
-- Performance (tuned 2026-07-11, PR #4876): compile-time 256-entry hex lookup table plus a preallocated `BinaryBuilder`, cutting per-byte branching and repeated buffer reallocations. Up to 31% faster on long strings. Benchmark: `benches/unhex.rs`.
+- Performance (tuned 2026-07-11, PR [#4876](https://github.com/apache/datafusion-comet/pull/4876)): compile-time 256-entry hex lookup table plus a preallocated `BinaryBuilder`, cutting per-byte branching and repeated buffer reallocations. Up to 31% faster on long strings. Benchmark: `benches/unhex.rs`.
## width_bucket
- Spark 3.5.8 (audited 2026-05-27): introduced; not available in 3.4.3.
- Spark 4.0.1, 4.1.1 (audited 2026-05-27): same semantics; `NullIntolerant` -> `nullIntolerant: Boolean` refactor.
-- Known limitation: wired via per-version `CometExprShim` rather than a `CometExpressionSerde`, so it bypasses the support-level framework and the auto-generated compatibility doc (https://github.com/apache/datafusion-comet/issues/4485). Native path uses datafusion-spark `SparkWidthBucket`; interval input types are not exercised by Comet tests.
+- Known limitation: wired via per-version `CometExprShim` rather than a `CometExpressionSerde`, so it bypasses the support-level framework and the auto-generated compatibility doc ([#4485](https://github.com/apache/datafusion-comet/issues/4485)). Native path uses datafusion-spark `SparkWidthBucket`; interval input types are not exercised by Comet tests.
[Spark Expression Support]: ../../user-guide/latest/expressions.md
diff --git a/_sources/contributor-guide/expression-audits/predicate_funcs.md.txt b/_sources/contributor-guide/expression-audits/predicate_funcs.md.txt
index 0731abf7f7..74d32b67ab 100644
--- a/_sources/contributor-guide/expression-audits/predicate_funcs.md.txt
+++ b/_sources/contributor-guide/expression-audits/predicate_funcs.md.txt
@@ -128,7 +128,7 @@
## like
-- See `string_funcs / like` (audited in PR #4461). `CometLike` only supports the default `\\` escape character.
+- See `string_funcs / like` (audited in PR [#4461](https://github.com/apache/datafusion-comet/pull/4461)). `CometLike` only supports the default `\\` escape character.
## not
@@ -146,6 +146,6 @@
## rlike
-- See `string_funcs / regexp_replace` and the `CometRLike` notes (audited in PR #4461). Uses the Rust `regex` crate, which differs from Java's `Pattern` engine; requires `spark.comet.expression.regexp.allowIncompatible=true`.
+- See `string_funcs / regexp_replace` and the `CometRLike` notes (audited in PR [#4461](https://github.com/apache/datafusion-comet/pull/4461)). Uses the Rust `regex` crate, which differs from Java's `Pattern` engine; requires `spark.comet.expression.regexp.allowIncompatible=true`.
[Spark Expression Support]: ../../user-guide/latest/expressions.md
diff --git a/_sources/contributor-guide/expression-audits/string_funcs.md.txt b/_sources/contributor-guide/expression-audits/string_funcs.md.txt
index 51154e3a09..f0bcb18ce2 100644
--- a/_sources/contributor-guide/expression-audits/string_funcs.md.txt
+++ b/_sources/contributor-guide/expression-audits/string_funcs.md.txt
@@ -25,20 +25,20 @@
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. `StringType -> IntegerType`; `nullSafeEval` returns `codePointAt(0)` of the first char, or `0` for the empty string. Wired via `CometScalarFunction("ascii")` and resolved to DataFusion `ascii` (`chars().next() as i32`); first-code-point semantics match for ASCII, BMP, and supplementary code points.
-- Spark 4.0.1 (audited 2026-05-27): `inputTypes` widened to `StringTypeWithCollation(supportsTrimCollation = true)`; behaviour unchanged for `UTF8_BINARY`. Comet does not propagate collation, so non-default collations may diverge silently (https://github.com/apache/datafusion-comet/issues/4496).
+- Spark 4.0.1 (audited 2026-05-27): `inputTypes` widened to `StringTypeWithCollation(supportsTrimCollation = true)`; behaviour unchanged for `UTF8_BINARY`. Comet does not propagate collation, so non-default collations may diverge silently ([#4496](https://github.com/apache/datafusion-comet/issues/4496)).
## bit_length
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. `(StringType|BinaryType) -> IntegerType`; eval returns `numBytes * 8` for strings and `.length * 8` for binary.
-- Spark 4.0.1 (audited 2026-05-27): `inputTypes` widened to `StringTypeWithCollation(supportsTrimCollation = true)`; semantics unchanged for `UTF8_BINARY`. Non-default collations not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4496).
+- Spark 4.0.1 (audited 2026-05-27): `inputTypes` widened to `StringTypeWithCollation(supportsTrimCollation = true)`; semantics unchanged for `UTF8_BINARY`. Non-default collations not honoured by Comet ([#4496](https://github.com/apache/datafusion-comet/issues/4496)).
- `BinaryType` input is reported `Unsupported` in `getSupportLevel`, so `bit_length(<binary>)` falls back to Spark cleanly (DataFusion's `BitLengthFunc` signature accepts string types only).
## btrim
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. `StringTrimBoth` is `RuntimeReplaceable` and rewritten to `StringTrim(srcStr, trimStr)` before serde runs. Support is provided by the `trim` entry; no dedicated serde registration.
-- Spark 4.0.1 (audited 2026-05-27): `StringTrim` (the rewrite target) routes through `CollationSupport.StringTrim.exec` and uses `StringTypeNonCSAICollation(supportsTrimCollation = true)`; semantics unchanged for `UTF8_BINARY`. Non-default collations may diverge in Comet (https://github.com/apache/datafusion-comet/issues/4496).
+- Spark 4.0.1 (audited 2026-05-27): `StringTrim` (the rewrite target) routes through `CollationSupport.StringTrim.exec` and uses `StringTypeNonCSAICollation(supportsTrimCollation = true)`; semantics unchanged for `UTF8_BINARY`. Non-default collations may diverge in Comet ([#4496](https://github.com/apache/datafusion-comet/issues/4496)).
## char
@@ -68,13 +68,13 @@
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. `Seq[Expression] -> StringType`; NULL separator yields NULL, NULL element values are skipped, children can be `StringType` or `ArrayType(StringType)`. Comet serde rewrites a NULL-literal separator to a NULL of the result type and bails out on all-foldable inputs so Spark's `ConstantFolding` handles them; otherwise delegates to DataFusion `concat_ws`.
-- Spark 4.0.1 (audited 2026-05-27): `inputTypes` widened to `StringTypeWithCollation` / `AbstractArrayType`; `dataType` becomes `children.head.dataType` (collation-derived). Semantics unchanged for `UTF8_BINARY`. Non-default collations not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4496).
+- Spark 4.0.1 (audited 2026-05-27): `inputTypes` widened to `StringTypeWithCollation` / `AbstractArrayType`; `dataType` becomes `children.head.dataType` (collation-derived). Semantics unchanged for `UTF8_BINARY`. Non-default collations not honoured by Comet ([#4496](https://github.com/apache/datafusion-comet/issues/4496)).
## contains
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. `UTF8String.contains` on `StringType`; the parser routes `(BinaryType, BinaryType)` to `BinaryPredicate`, so Comet only ever sees the String form.
-- Spark 4.0.1 (audited 2026-05-27): routes through `CollationSupport.Contains.exec(..., collationId)`; behaviour identical for `UTF8_BINARY`. Non-default collations not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4496).
+- Spark 4.0.1 (audited 2026-05-27): routes through `CollationSupport.Contains.exec(..., collationId)`; behaviour identical for `UTF8_BINARY`. Non-default collations not honoured by Comet ([#4496](https://github.com/apache/datafusion-comet/issues/4496)).
## decode
@@ -82,25 +82,25 @@
- Spark 3.5.8 (audited 2026-05-27): baseline. `StringDecode(bin, charset)` evaluated directly; invalid sequences silently substitute replacement characters via `new String(bytes, charset)`.
- Spark 4.0.1 (audited 2026-05-27): refactored to `RuntimeReplaceable` whose `replacement` is a `StaticInvoke(StringDecode.decode, bin, charset, legacyCharsets, legacyErrorAction)`; the 4-arg form raises on malformed input unless legacy flags are set.
- `decode` runs through the codegen dispatcher on all versions (Spark 3.x via `CometStringDecode`, Spark 4.0 via the `StaticInvoke` replacement routed to `CometStaticInvokeCodegenDispatch`), so Spark's own evaluation runs inside the Comet pipeline. This honours the `charset` argument and the Spark 4.0 `legacyCharsets` / `legacyErrorAction` flags, and falls back to Spark when the dispatcher is disabled.
-- Known limitation: because there is no `CometExpressionSerde[StringDecode]` registration, `decode` does not surface in the auto-generated compatibility docs (https://github.com/apache/datafusion-comet/issues/4466).
+- Known limitation: because there is no `CometExpressionSerde[StringDecode]` registration, `decode` does not surface in the auto-generated compatibility docs ([#4466](https://github.com/apache/datafusion-comet/issues/4466)).
## endswith
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. `UTF8String.endsWith` on `StringType`; binary form routed to `BinaryPredicate` before Comet.
-- Spark 4.0.1 (audited 2026-05-27): routes through `CollationSupport.EndsWith.exec`; semantics unchanged for `UTF8_BINARY`. Non-default collations not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4496).
+- Spark 4.0.1 (audited 2026-05-27): routes through `CollationSupport.EndsWith.exec`; semantics unchanged for `UTF8_BINARY`. Non-default collations not honoured by Comet ([#4496](https://github.com/apache/datafusion-comet/issues/4496)).
## initcap
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
-- Spark 3.5.8 (audited 2026-05-27): baseline. `string.toLowerCase.toTitleCase` on `UTF8String`; word boundary is Java `Character.isWhitespace`. Comet routes to DataFusion `initcap`, which splits on `!is_alphanumeric()` (hyphens, apostrophes, and punctuation all split words), so Comet is unconditionally `Incompatible` (https://github.com/apache/datafusion-comet/issues/1052).
-- Spark 4.0.1 (audited 2026-05-27): routes through `CollationSupport.InitCap.exec` (collation- and ICU-aware) and propagates `child.dataType`. Comet ignores collation; 3.x divergences persist plus collation/ICU mismatches (https://github.com/apache/datafusion-comet/issues/4496).
+- Spark 3.5.8 (audited 2026-05-27): baseline. `string.toLowerCase.toTitleCase` on `UTF8String`; word boundary is Java `Character.isWhitespace`. DataFusion `initcap` splits on `!is_alphanumeric()` (hyphens, apostrophes, and punctuation all split words), so `CometInitCap` reports `Compatible` with a `NativeOptIn` and dispatches to Spark's generated code by default. The native path is used only when `spark.comet.expression.InitCap.allowIncompatible=true`.
+- Spark 4.0.1 (audited 2026-05-27): routes through `CollationSupport.InitCap.exec` (collation- and ICU-aware) and propagates `child.dataType`. Comet ignores collation; 3.x divergences persist plus collation/ICU mismatches ([#4496](https://github.com/apache/datafusion-comet/issues/4496)).
## instr
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. `StringInstr(str, substr) -> IntegerType`; returns `string.indexOf(sub, 0) + 1` (1-based, 0 when not found, 1 on empty substring). Resolves to DataFusion `strpos` (alias `instr`) with matching semantics.
-- Spark 4.0.1 (audited 2026-05-27): routes through `CollationSupport.StringInstr.exec`; semantics unchanged for `UTF8_BINARY`. Non-default collations not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4496).
+- Spark 4.0.1 (audited 2026-05-27): routes through `CollationSupport.StringInstr.exec`; semantics unchanged for `UTF8_BINARY`. Non-default collations not honoured by Comet ([#4496](https://github.com/apache/datafusion-comet/issues/4496)).
## lcase
@@ -112,7 +112,7 @@
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. `RuntimeReplaceable` with `replacement = Substring(str, Literal(1), len)`; accepts `StringType` or `BinaryType` plus `IntegerType`. Comet serde serialises `expr.replacement`, routing through `SparkSubstring` — so non-literal `len` is supported.
-- Spark 4.0.1 (audited 2026-05-27): `inputTypes` widened with `StringTypeWithCollation`; behaviour unchanged for `UTF8_BINARY`. Non-default collations not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4496).
+- Spark 4.0.1 (audited 2026-05-27): `inputTypes` widened with `StringTypeWithCollation`; behaviour unchanged for `UTF8_BINARY`. Non-default collations not honoured by Comet ([#4496](https://github.com/apache/datafusion-comet/issues/4496)).
## len
@@ -124,32 +124,32 @@
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. `(StringType|BinaryType) -> IntegerType`; eval returns `numChars` for strings and `.length` for binary. `BinaryType` input falls back via `Unsupported` (DataFusion's `character_length` accepts string types only).
-- Spark 4.0.1 (audited 2026-05-27): `inputTypes` widened to `StringTypeWithCollation(supportsTrimCollation = true)`; semantics unchanged. Non-default collations not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4496).
+- Spark 4.0.1 (audited 2026-05-27): `inputTypes` widened to `StringTypeWithCollation(supportsTrimCollation = true)`; semantics unchanged. Non-default collations not honoured by Comet ([#4496](https://github.com/apache/datafusion-comet/issues/4496)).
## lower
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. JVM default-locale `toLowerCase` on `UTF8String`. Comet routes to DataFusion `lower` (Rust Unicode default case mapping, no locale awareness) and is unconditionally `Incompatible`; users opt in via the standard `spark.comet.expression.Lower.allowIncompatible=true`.
-- Spark 4.0.1 (audited 2026-05-27): routes through `CollationSupport.Lower.exec(v, collationId, useICU)` with `SQLConf.ICU_CASE_MAPPINGS_ENABLED`; `inputTypes` widened to `StringTypeWithCollation`. Comet ignores collation and ICU mode, so non-default collations or `ICU_CASE_MAPPINGS_ENABLED=true` diverge even after opting in (https://github.com/apache/datafusion-comet/issues/2190).
+- Spark 4.0.1 (audited 2026-05-27): routes through `CollationSupport.Lower.exec(v, collationId, useICU)` with `SQLConf.ICU_CASE_MAPPINGS_ENABLED`; `inputTypes` widened to `StringTypeWithCollation`. Comet ignores collation and ICU mode, so non-default collations or `ICU_CASE_MAPPINGS_ENABLED=true` diverge even after opting in ([#2190](https://github.com/apache/datafusion-comet/issues/2190)).
## lpad
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. `StringLPad(str, len, pad) -> StringType`; `len <= 0` returns the empty string, empty `pad` returns `str` unchanged, NULL inputs propagate. Comet serde requires `str` to be a column and `pad` to be a literal; otherwise falls back.
-- Spark 4.0.1 (audited 2026-05-27): `NullIntolerant` trait replaced by `override def nullIntolerant: Boolean = true`; `inputTypes` widened to `StringTypeWithCollation(supportsTrimCollation = true)`. Semantics unchanged for `UTF8_BINARY`. Non-default collations not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4496).
+- Spark 4.0.1 (audited 2026-05-27): `NullIntolerant` trait replaced by `override def nullIntolerant: Boolean = true`; `inputTypes` widened to `StringTypeWithCollation(supportsTrimCollation = true)`. Semantics unchanged for `UTF8_BINARY`. Non-default collations not honoured by Comet ([#4496](https://github.com/apache/datafusion-comet/issues/4496)).
- Known limitation: `lpad(<binary>, ...)` is rewritten by Spark to `BinaryPad / StaticInvoke(ByteArray.lpad)` before serde runs and always falls back to Spark.
## ltrim
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. `StringTrimLeft` extends `String2TrimExpression`; no-arg form strips ASCII space `0x20` only. The two-arg parser form `ltrim(trimStr, srcStr)` is swapped to `(srcStr, Option(trimStr))` by Spark's secondary constructor, so children match DataFusion `ltrim(str, chars)`.
-- Spark 4.0.1 (audited 2026-05-27): routes through `CollationSupport.StringTrimLeft.exec`; semantics unchanged for `UTF8_BINARY`. Non-default collations not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4496).
+- Spark 4.0.1 (audited 2026-05-27): routes through `CollationSupport.StringTrimLeft.exec`; semantics unchanged for `UTF8_BINARY`. Non-default collations not honoured by Comet ([#4496](https://github.com/apache/datafusion-comet/issues/4496)).
## octet_length
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. `(StringType|BinaryType) -> IntegerType`; eval returns `numBytes` for strings and `.length` for binary.
-- Spark 4.0.1 (audited 2026-05-27): `inputTypes` widened to `StringTypeWithCollation`; semantics unchanged for `UTF8_BINARY`. Non-default collations not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4496).
+- Spark 4.0.1 (audited 2026-05-27): `inputTypes` widened to `StringTypeWithCollation`; semantics unchanged for `UTF8_BINARY`. Non-default collations not honoured by Comet ([#4496](https://github.com/apache/datafusion-comet/issues/4496)).
- `BinaryType` input is reported `Unsupported` in `getSupportLevel`, so `octet_length(<binary>)` falls back to Spark cleanly (DataFusion's `OctetLengthFunc` signature accepts string types only).
## regexp_replace
@@ -163,32 +163,32 @@
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. `StringRepeat(str, times)` with `nullSafeEval(s, n) = s.repeat(n)`; `UTF8String.repeat` returns the empty string for `n <= 0`. Comet casts `times` to `LongType` and delegates to DataFusion `repeat`, which mirrors Spark for negative counts.
-- Spark 4.0.1 (audited 2026-05-27): adds `nullIntolerant: Boolean` field; `dataType` becomes `str.dataType` (collation-tracking). Semantics unchanged for `UTF8_BINARY`. Non-default collations not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4496).
+- Spark 4.0.1 (audited 2026-05-27): adds `nullIntolerant: Boolean` field; `dataType` becomes `str.dataType` (collation-tracking). Semantics unchanged for `UTF8_BINARY`. Non-default collations not honoured by Comet ([#4496](https://github.com/apache/datafusion-comet/issues/4496)).
## replace
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
-- Spark 3.5.8 (audited 2026-05-27): baseline. `StringReplace(src, search, replace)`; when `search` is empty, Spark returns `src` unchanged (short-circuit on `search.numBytes == 0`). DataFusion `replace` instead inserts `replace` between every character, so `CometStringReplace.getSupportLevel` marks `Incompatible(Some(reason))` when `search` is a literal empty string and falls back to Spark by default (https://github.com/apache/datafusion-comet/issues/4497).
-- Spark 4.0.1 (audited 2026-05-27): routes through `CollationSupport.StringReplace.exec`; semantics unchanged for `UTF8_BINARY`. Non-default collations not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4496).
+- Spark 3.5.8 (audited 2026-05-27): baseline. `StringReplace(src, search, replace)`; when `search` is empty, Spark returns `src` unchanged (short-circuit on `search.numBytes == 0`). DataFusion `replace` instead inserts `replace` between every character, so `CometStringReplace` reports `Compatible` with a `NativeOptIn` and runs Spark's own generated code inside the Comet pipeline by default. The native DataFusion `replace` is used only when `spark.comet.expression.StringReplace.allowIncom [...]
+- Spark 4.0.1 (audited 2026-05-27): routes through `CollationSupport.StringReplace.exec`; semantics unchanged for `UTF8_BINARY`. Non-default collations not honoured by Comet ([#4496](https://github.com/apache/datafusion-comet/issues/4496)).
## right
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. `RuntimeReplaceable` with `replacement = If(IsNull(str), null, If(len <= 0, "", Substring(str, -len, len)))`; accepts `StringType` plus `IntegerType`. Comet serde serialises `expr.replacement`, so NULL propagation for `len <= 0` and non-literal `len` are handled by the replacement tree itself.
-- Spark 4.0.1 (audited 2026-05-27): `inputTypes` widened with collation; uses `UnaryMinus(len, failOnError = false)` to avoid integer-overflow exceptions on `len = Int.MinValue`. Semantics unchanged for `UTF8_BINARY`. Non-default collations not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4496).
+- Spark 4.0.1 (audited 2026-05-27): `inputTypes` widened with collation; uses `UnaryMinus(len, failOnError = false)` to avoid integer-overflow exceptions on `len = Int.MinValue`. Semantics unchanged for `UTF8_BINARY`. Non-default collations not honoured by Comet ([#4496](https://github.com/apache/datafusion-comet/issues/4496)).
## rpad
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. `StringRPad(str, len, pad) -> StringType`; same edge-case behaviour as `lpad` (negative len, empty pad, NULL propagation). Comet serde requires column `str` and literal `pad`.
-- Spark 4.0.1 (audited 2026-05-27): same evolution as `lpad`; default-pad literal type tightened; semantics unchanged for `UTF8_BINARY`. Non-default collations not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4496).
+- Spark 4.0.1 (audited 2026-05-27): same evolution as `lpad`; default-pad literal type tightened; semantics unchanged for `UTF8_BINARY`. Non-default collations not honoured by Comet ([#4496](https://github.com/apache/datafusion-comet/issues/4496)).
- Known limitation: same `BinaryPad / StaticInvoke` rewrite as `lpad` causes `rpad(<binary>, ...)` to fall back.
## rtrim
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. `StringTrimRight` extends `String2TrimExpression`; semantically symmetric to `ltrim`.
-- Spark 4.0.1 (audited 2026-05-27): routes through `CollationSupport.StringTrimRight.exec`; semantics unchanged for `UTF8_BINARY`. Non-default collations not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4496).
+- Spark 4.0.1 (audited 2026-05-27): routes through `CollationSupport.StringTrimRight.exec`; semantics unchanged for `UTF8_BINARY`. Non-default collations not honoured by Comet ([#4496](https://github.com/apache/datafusion-comet/issues/4496)).
## space
@@ -200,13 +200,13 @@
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. `StringSplit(str, regex, limit)`; `limit > 0` permits at most `limit-1` splits, `limit <= 0` is unlimited. Comet registers `split` as a custom UDF (`native/spark-expr/src/string_funcs/split.rs`) using the Rust `regex` crate, and is unconditionally `Incompatible` due to regex-engine differences.
-- Spark 4.0.1 (audited 2026-05-27): wraps the regex via `CollationSupport.collationAwareRegex` and changes `dataType` to `ArrayType(str.dataType, ...)`. Comet does not honour collation flags (https://github.com/apache/datafusion-comet/issues/4496).
+- Spark 4.0.1 (audited 2026-05-27): wraps the regex via `CollationSupport.collationAwareRegex` and changes `dataType` to `ArrayType(str.dataType, ...)`. Comet does not honour collation flags ([#4496](https://github.com/apache/datafusion-comet/issues/4496)).
## startswith
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. `UTF8String.startsWith` on `StringType`; binary form routed to `BinaryPredicate`.
-- Spark 4.0.1 (audited 2026-05-27): routes through `CollationSupport.StartsWith.exec`; semantics unchanged for `UTF8_BINARY`. Non-default collations not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4496).
+- Spark 4.0.1 (audited 2026-05-27): routes through `CollationSupport.StartsWith.exec`; semantics unchanged for `UTF8_BINARY`. Non-default collations not honoured by Comet ([#4496](https://github.com/apache/datafusion-comet/issues/4496)).
## substr
@@ -218,26 +218,26 @@
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. `TernaryExpression`; two-arg form defaults `len = Integer.MAX_VALUE`; supports `StringType` and `BinaryType`. Comet routes through the `datafusion-spark` `SparkSubstring` UDF (registered as `substring`), which accepts non-literal `pos`/`len` and Utf8/Utf8View/Binary/LargeBinary/BinaryView inputs.
-- Spark 4.0.1 (audited 2026-05-27): `inputTypes` widened with `StringTypeWithCollation`; semantics unchanged for `UTF8_BINARY`. `SparkSubstring` implements Spark's 1-indexed, negative-start, and negative-length clamping. Non-default collations not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4496).
+- Spark 4.0.1 (audited 2026-05-27): `inputTypes` widened with `StringTypeWithCollation`; semantics unchanged for `UTF8_BINARY`. `SparkSubstring` implements Spark's 1-indexed, negative-start, and negative-length clamping. Non-default collations not honoured by Comet ([#4496](https://github.com/apache/datafusion-comet/issues/4496)).
## substring_index
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. `TernaryExpression(StringType, StringType, IntegerType) -> StringType`. Comet casts `count` to `LongType` and delegates to DataFusion's `substr_index` UDF (alias `substring_index`).
-- Spark 4.0.1 (audited 2026-05-27): routes through `CollationSupport.SubstringIndex.exec` and propagates `strExpr.dataType`; semantics unchanged for `UTF8_BINARY`. Non-default collations not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4496).
+- Spark 4.0.1 (audited 2026-05-27): routes through `CollationSupport.SubstringIndex.exec` and propagates `strExpr.dataType`; semantics unchanged for `UTF8_BINARY`. Non-default collations not honoured by Comet ([#4496](https://github.com/apache/datafusion-comet/issues/4496)).
## translate
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. `StringTranslate(src, from, to)`; `UTF8String.translate(dict)` is code-point based, and any character mapped explicitly to U+0000 in `to` is also deleted.
-- Spark 4.0.1 (audited 2026-05-27): routes through `CollationSupport.StringTranslate.exec`; semantics unchanged for `UTF8_BINARY`. Non-default collations not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4496).
+- Spark 4.0.1 (audited 2026-05-27): routes through `CollationSupport.StringTranslate.exec`; semantics unchanged for `UTF8_BINARY`. Non-default collations not honoured by Comet ([#4496](https://github.com/apache/datafusion-comet/issues/4496)).
- Marked `Incompatible`: DataFusion's `translate` iterates over Unicode graphemes (Spark uses code points) and substitutes U+0000 instead of treating it as a deletion sentinel. It falls back to Spark by default and runs natively only when incompatible expressions are explicitly allowed.
## trim
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. `StringTrim` no-arg form strips ASCII space `0x20` only (matches DataFusion `btrim`'s default); two-arg form's children are `(srcStr, trimStr)` after Spark's secondary-constructor swap.
-- Spark 4.0.1 (audited 2026-05-27): routes through `CollationSupport.StringTrim.exec` and uses `StringTypeNonCSAICollation`; semantics unchanged for `UTF8_BINARY`. Non-default collations not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4496).
+- Spark 4.0.1 (audited 2026-05-27): routes through `CollationSupport.StringTrim.exec` and uses `StringTypeNonCSAICollation`; semantics unchanged for `UTF8_BINARY`. Non-default collations not honoured by Comet ([#4496](https://github.com/apache/datafusion-comet/issues/4496)).
## ucase
@@ -249,6 +249,6 @@
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. JVM default-locale `toUpperCase` on `UTF8String`. Comet routes to DataFusion `upper` (Rust Unicode default case mapping, no locale awareness) and is unconditionally `Incompatible`; users opt in via the standard `spark.comet.expression.Upper.allowIncompatible=true`.
-- Spark 4.0.1 (audited 2026-05-27): routes through `CollationSupport.Upper.exec(v, collationId, useICU)` with `SQLConf.ICU_CASE_MAPPINGS_ENABLED`. Comet does not propagate collation or ICU mode; non-default collations or `ICU_CASE_MAPPINGS_ENABLED=true` diverge even after opting in (https://github.com/apache/datafusion-comet/issues/2190).
+- Spark 4.0.1 (audited 2026-05-27): routes through `CollationSupport.Upper.exec(v, collationId, useICU)` with `SQLConf.ICU_CASE_MAPPINGS_ENABLED`. Comet does not propagate collation or ICU mode; non-default collations or `ICU_CASE_MAPPINGS_ENABLED=true` diverge even after opting in ([#2190](https://github.com/apache/datafusion-comet/issues/2190)).
[Spark Expression Support]: ../../user-guide/latest/expressions.md
diff --git a/_sources/contributor-guide/expression-audits/url_funcs.md.txt b/_sources/contributor-guide/expression-audits/url_funcs.md.txt
index 2a61c742c8..6db1ebd368 100644
--- a/_sources/contributor-guide/expression-audits/url_funcs.md.txt
+++ b/_sources/contributor-guide/expression-audits/url_funcs.md.txt
@@ -23,7 +23,7 @@
## parse_url
-- Performance (tuned 2026-07-11, PR #4893): cache the compiled query regex in a thread-local single-slot cache keyed by the query key, eliminating per-row `Regex::new()` recompilation in the QUERY-with-key path (the key is constant across a batch in practice). Up to 50x faster. Benchmark: `benches/parse_url.rs`.
+- Performance (tuned 2026-07-11, PR [#4893](https://github.com/apache/datafusion-comet/pull/4893)): cache the compiled query regex in a thread-local single-slot cache keyed by the query key, eliminating per-row `Regex::new()` recompilation in the QUERY-with-key path (the key is constant across a batch in practice). Up to 50x faster. Benchmark: `benches/parse_url.rs`.
## try_url_decode
diff --git a/_sources/contributor-guide/expression-audits/window_funcs.md.txt b/_sources/contributor-guide/expression-audits/window_funcs.md.txt
index 8a6b79e583..c692a42a55 100644
--- a/_sources/contributor-guide/expression-audits/window_funcs.md.txt
+++ b/_sources/contributor-guide/expression-audits/window_funcs.md.txt
@@ -23,7 +23,7 @@
## avg (window)
-- 4.1.1, audited 2026-06-25: `AVG(<decimal>)` over a window falls back to Spark on Spark 4.x. Spark wraps the result as `Cast(avg(UnscaledValue(v)) / pow10 as decimal)`, a shape `CometWindowExec.convert` does not unwrap (it recognizes only `Alias(WindowExpression)` and `Alias(MakeDecimal(WindowExpression))`), so the `AvgDecimal` native window branch in `process_agg_func` is effectively dead on Spark 4.x. Results stay correct via fallback; this is a coverage gap, not a correctness diverge [...]
+- 4.1.1, audited 2026-06-25: `AVG(<decimal>)` over a window runs natively on Spark 4.x. Spark wraps the result as `Cast(avg(UnscaledValue(v)) / pow10 as decimal)`; `CometWindowExec.convert` unwraps that shape alongside `Alias(WindowExpression)` and `Alias(MakeDecimal(WindowExpression))`, so the `AvgDecimal` native window branch in `process_agg_func` is reachable. Non-decimal `AVG` runs natively. Sliding-frame decimal `AVG` is guarded by the same fallback as decimal `SUM` (see below) so t [...]
## lag
diff --git a/_sources/contributor-guide/roadmap.md.txt b/_sources/contributor-guide/roadmap.md.txt
index 59aba8e7c4..eaa6f0c097 100644
--- a/_sources/contributor-guide/roadmap.md.txt
+++ b/_sources/contributor-guide/roadmap.md.txt
@@ -65,20 +65,19 @@ Iceberg native scans support non-AQE DPP ([#3349], [#3511]) and, on Spark 3.5+,
We regularly publish benchmark results derived from TPC-H and TPC-DS to track performance against Spark. Closing
the remaining gaps and increasing the speedup on both benchmark suites is an ongoing focus, tracked under [#2004]
-(TPC-H), [#858] (TPC-DS), and [#3799] (improving the awslabs published TPC-DS results).
+(TPC-H) and [#858] (TPC-DS).
[#858]: https://github.com/apache/datafusion-comet/issues/858
[#2004]: https://github.com/apache/datafusion-comet/issues/2004
-[#3799]: https://github.com/apache/datafusion-comet/issues/3799
## Upstream Work in DataFusion
A growing number of Spark-compatible expressions live in the `datafusion-spark` crate in the core DataFusion
repository. Comet is migrating its expression implementations to that crate so that they can be shared by other
-DataFusion-based projects, tracked in [#2084]. Improvements to core DataFusion operators (joins, aggregates,
-window) made in support of Comet also benefit the wider ecosystem.
+DataFusion-based projects, and is wiring up the functions that crate already provides ([#4150]). Improvements to
+core DataFusion operators (joins, aggregates, window) made in support of Comet also benefit the wider ecosystem.
-[#2084]: https://github.com/apache/datafusion-comet/issues/2084
+[#4150]: https://github.com/apache/datafusion-comet/issues/4150
## Spillable Hash Join
diff --git a/_sources/contributor-guide/s3-credential-provider-design.md.txt b/_sources/contributor-guide/s3-credential-provider-design.md.txt
index 00504c3ea3..25d1951eca 100644
--- a/_sources/contributor-guide/s3-credential-provider-design.md.txt
+++ b/_sources/contributor-guide/s3-credential-provider-design.md.txt
@@ -40,7 +40,7 @@ An earlier iteration used `ServiceLoader` discovery. That was rejected because:
- ServiceLoader makes activation implicit on classpath presence. A vendor JAR drifting onto a cluster could silently change S3 auth behavior. The config key makes activation explicit.
- The activation key (`fs.s3a.comet.credential.provider.class`, with per-bucket override) follows the same shape as `fs.s3a.bucket.<name>.aws.credentials.provider`, so operators do not learn a new pattern.
-Activation is modeled on `parquet.crypto.factory.class` (Parquet Modular Encryption KMS, see Comet #2447): the user names a single vendor class and the vendor dispatches across multiple credential backends inside that class if they need to. This mirrors how Iceberg's `DecryptionPropertiesFactory` already behaves for Parquet keys.
+Activation is modeled on `parquet.crypto.factory.class` (Parquet Modular Encryption KMS, see Comet [#2447](https://github.com/apache/datafusion-comet/pull/2447)): the user names a single vendor class and the vendor dispatches across multiple credential backends inside that class if they need to. This mirrors how Iceberg's `DecryptionPropertiesFactory` already behaves for Parquet keys.
## Why `(FQCN, dispatchKey, catalogProperties)` keying
diff --git a/_sources/user-guide/latest/compatibility/scans.md.txt b/_sources/user-guide/latest/compatibility/scans.md.txt
index aa2be5b3ce..b7dca43ae8 100644
--- a/_sources/user-guide/latest/compatibility/scans.md.txt
+++ b/_sources/user-guide/latest/compatibility/scans.md.txt
@@ -72,7 +72,7 @@ The following limitations raise an error at scan time rather than falling back t
rejecting the read. This applies to all LTZ physical encodings (INT96, TIMESTAMP_MICROS,
TIMESTAMP_MILLIS). On Spark 4.0+, this read is permitted
([SPARK-47447](https://issues.apache.org/jira/browse/SPARK-47447)) and Comet matches Spark's
- behavior. See [#4219](https://github.com/apache/datafusion-comet/issues/4219).
+ behavior.
### Schema Mismatch Handling
diff --git a/_sources/user-guide/latest/datasources.md.txt b/_sources/user-guide/latest/datasources.md.txt
index 9d68748a75..42c721d63c 100644
--- a/_sources/user-guide/latest/datasources.md.txt
+++ b/_sources/user-guide/latest/datasources.md.txt
@@ -252,7 +252,7 @@ Comet's S3 support has the following limitations:
1. **Partial Hadoop S3A configuration support**: Not all Hadoop S3A configurations are currently supported. Only the configurations listed in the tables above are translated and applied to the underlying `object_store` crate.
-2. **Custom credential providers**: Custom implementations of AWS credential providers are not supported. The implementation only supports the standard credential providers listed in the table above. We are planning to add support for custom credential providers through a JNI-based adapter that will allow calling Java credential providers from Rust code. See [issue #1829](https://github.com/apache/datafusion-comet/issues/1829) for more details.
+2. **Custom credential providers**: Custom implementations of AWS credential providers are not supported. The implementation only supports the standard credential providers listed in the table above. We are planning to add support for custom credential providers through a JNI-based adapter that will allow calling Java credential providers from Rust code. See [#1829](https://github.com/apache/datafusion-comet/issues/1829) for more details.
## Azure
diff --git a/_sources/user-guide/latest/datatypes.md.txt b/_sources/user-guide/latest/datatypes.md.txt
index 80465121da..c2b942961c 100644
--- a/_sources/user-guide/latest/datatypes.md.txt
+++ b/_sources/user-guide/latest/datatypes.md.txt
@@ -80,14 +80,18 @@ the tables below and may be reconsidered based on demand:
## Interval
-Interval types fall back to Spark today. Native acceleration is tracked by
-[#4540](https://github.com/apache/datafusion-comet/issues/4540).
-
-| Type | Status | Notes |
-| ----------------------- | ------ | ----------------- |
-| `YearMonthIntervalType` | 🔜 | Tracked by #4540. |
-| `DayTimeIntervalType` | 🔜 | Tracked by #4540. |
-| `CalendarIntervalType` | 🔜 | Tracked by #4540. |
+All three interval types are mapped to Arrow and flow through serde, native shuffle, and the
+codegen dispatcher, so interval columns and the interval-producing expressions run natively.
+Several operators still gate on the type and fall back: Parquet scans of ANSI interval columns,
+single-column sorts, hash aggregates (`min` / `max` / `sum` / `avg`), `GROUP BY`, window
+functions, and hashing a `CalendarInterval`. Remaining work is tracked by
+[#5061](https://github.com/apache/datafusion-comet/issues/5061).
+
+| Type | Status | Notes |
+| ----------------------- | ------ | ------------------------------------------------------------------------------------------ |
+| `YearMonthIntervalType` | ⚠️ | Parquet scan, single-column sort, aggregate, `GROUP BY`, and window operators fall back. |
+| `DayTimeIntervalType` | ⚠️ | Parquet scan, single-column sort, aggregate, `GROUP BY`, and window operators fall back. |
+| `CalendarIntervalType` | ⚠️ | As above, plus `hash` / `xxhash64` of a `CalendarInterval` fails rather than falling back. |
## Complex
diff --git a/_sources/user-guide/latest/expressions.md.txt b/_sources/user-guide/latest/expressions.md.txt
index 917b3caa04..5384af37f0 100644
--- a/_sources/user-guide/latest/expressions.md.txt
+++ b/_sources/user-guide/latest/expressions.md.txt
@@ -104,7 +104,7 @@ The tables below list every Spark built-in expression with its current status.
| `first_value` | ✅ | Native | |
| `grouping` | ✅ | — | Grouping indicator for ROLLUP/CUBE/GROUPING SETS |
| `grouping_id` | ✅ | — | Grouping indicator for ROLLUP/CUBE/GROUPING SETS |
-| `kurtosis` | 🔜 | — | tracking [#4098](https://github.com/apache/datafusion-comet/issues/4098) |
+| `kurtosis` | 🔜 | — | Not yet implemented natively |
| `last` | ✅ | Native | |
| `last_value` | ✅ | Native | |
| `listagg` | 🔜 | — | String aggregation |
@@ -118,16 +118,16 @@ The tables below list every Spark built-in expression with its current status.
| `percentile` | ✅ | Native | Single literal percentage on numeric input runs natively; array of percentages and a frequency argument fall back to Spark |
| `percentile_cont` | ✅ | — | Spark 4.0+ `WITHIN GROUP (ORDER BY ...)`; ascending only runs natively, `DESC` falls back to Spark |
| `percentile_disc` | 🔜 | — | Percentile aggregate |
-| `regr_avgx` | ✅ | — | Native: Spark rewrites to `Average` (tests in [#4551](https://github.com/apache/datafusion-comet/issues/4551)) |
-| `regr_avgy` | ✅ | — | Native: Spark rewrites to `Average` (tests in [#4551](https://github.com/apache/datafusion-comet/issues/4551)) |
-| `regr_count` | ✅ | — | Native: Spark rewrites to `Count` (tests in [#4551](https://github.com/apache/datafusion-comet/issues/4551)) |
+| `regr_avgx` | ✅ | — | Native: Spark rewrites to `Average` (tests in [#4551](https://github.com/apache/datafusion-comet/pull/4551)) |
+| `regr_avgy` | ✅ | — | Native: Spark rewrites to `Average` (tests in [#4551](https://github.com/apache/datafusion-comet/pull/4551)) |
+| `regr_count` | ✅ | — | Native: Spark rewrites to `Count` (tests in [#4551](https://github.com/apache/datafusion-comet/pull/4551)) |
| `regr_intercept` | 🔜 | — | Falls back; can reuse `covar_pop`/`var_pop` accumulators ([#4552](https://github.com/apache/datafusion-comet/issues/4552)) |
| `regr_r2` | 🔜 | — | Falls back; can reuse the `corr` accumulator ([#4552](https://github.com/apache/datafusion-comet/issues/4552)) |
| `regr_slope` | 🔜 | — | Falls back; can reuse `covar_pop`/`var_pop` accumulators ([#4552](https://github.com/apache/datafusion-comet/issues/4552)) |
| `regr_sxx` | 🔜 | — | Falls back; can reuse `var_pop` accumulator ([#4552](https://github.com/apache/datafusion-comet/issues/4552)) |
| `regr_sxy` | 🔜 | — | Falls back; can reuse `covar_pop` accumulator ([#4552](https://github.com/apache/datafusion-comet/issues/4552)) |
| `regr_syy` | 🔜 | — | Falls back; can reuse `var_pop` accumulator ([#4552](https://github.com/apache/datafusion-comet/issues/4552)) |
-| `skewness` | 🔜 | — | tracking [#4098](https://github.com/apache/datafusion-comet/issues/4098) |
+| `skewness` | 🔜 | — | Not yet implemented natively |
| `some` | ✅ | — | |
| `std` | ✅ | Native | |
| `stddev` | ✅ | Native | |
@@ -170,7 +170,7 @@ The tables below list every Spark built-in expression with its current status.
| `get` | ✅ | — | |
| `sequence` | ✅ | Codegen dispatch | |
| `shuffle` | ✅ | Native | Binary/struct/map elements fall back |
-| `slice` | ✅ | Native | Native ([#4149](https://github.com/apache/datafusion-comet/issues/4149)) |
+| `slice` | ✅ | Native | Native ([#4149](https://github.com/apache/datafusion-comet/pull/4149)) |
| `sort_array` | ✅ | Hybrid | Nested struct/null arrays fall back |
---
@@ -277,7 +277,7 @@ The type-name conversion functions (`bigint`, `binary`, `boolean`, `date`, `deci
| `localtimestamp` | ✅ | — | |
| `make_date` | ✅ | Native | |
| `make_dt_interval` | ✅ | Codegen dispatch | |
-| `make_interval` | 🔜 | — | Produces legacy CalendarInterval; tracked by [#4540](https://github.com/apache/datafusion-comet/issues/4540) |
+| `make_interval` | 🔜 | — | Produces legacy CalendarInterval; tracked by [#5061](https://github.com/apache/datafusion-comet/issues/5061) |
| `make_time` | 🔜 | — | Spark 4.1 TIME type; tracked by [#4288](https://github.com/apache/datafusion-comet/issues/4288) |
| `make_timestamp` | ✅ | — | |
| `make_timestamp_ltz` | ✅ | — | 2-arg TIME form falls back |
@@ -305,7 +305,7 @@ The type-name conversion functions (`bigint`, `binary`, `boolean`, `date`, `deci
| `to_unix_timestamp` | ✅ | Hybrid | |
| `to_utc_timestamp` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default (handles all timezone forms); the native path is opt-in via allowIncompatible ([details](compatibility/expressions/datetime.md)) |
| `trunc` | ✅ | Hybrid | |
-| `try_make_interval` | 🔜 | — | Produces legacy CalendarInterval; tracked by [#4540](https://github.com/apache/datafusion-comet/issues/4540) |
+| `try_make_interval` | 🔜 | — | Produces legacy CalendarInterval; tracked by [#5061](https://github.com/apache/datafusion-comet/issues/5061) |
| `try_make_timestamp` | ✅ | — | |
| `try_to_date` | ✅ | — | Rewrites to `Cast`/`GetTimestamp` before Comet sees the plan; same support as `to_date` |
| `try_to_time` | 🔜 | — | Spark 4.1 TIME type; tracked by [#4288](https://github.com/apache/datafusion-comet/issues/4288) |
@@ -495,24 +495,24 @@ expression-level). The `outer` variants are wired but marked `Incompatible`; the
| `current_schema` | ✅ | — | Alias of `current_database`; resolved to a literal by the analyzer |
| `current_user` | ✅ | — | Resolved to a literal by the analyzer; same as `user` |
| `equal_null` | ✅ | — | Lowers to `<=>` (`EqualNullSafe`) |
-| `is_variant_null` | 🔜 | — | tracking [#4098](https://github.com/apache/datafusion-comet/issues/4098) |
+| `is_variant_null` | 🔜 | — | Requires `VariantType` support |
| `monotonically_increasing_id` | ✅ | Native | |
-| `parse_json` | 🔜 | — | tracking [#4098](https://github.com/apache/datafusion-comet/issues/4098) |
+| `parse_json` | 🔜 | — | Requires `VariantType` support |
| `raise_error` | 🔜 | — | Raises a runtime error |
| `rand` | ✅ | Native | Seed must be a literal |
| `randn` | ✅ | Native | Seed must be a literal |
-| `schema_of_variant` | 🔜 | — | tracking [#4098](https://github.com/apache/datafusion-comet/issues/4098) |
-| `schema_of_variant_agg` | 🔜 | — | tracking [#4098](https://github.com/apache/datafusion-comet/issues/4098) |
+| `schema_of_variant` | 🔜 | — | Requires `VariantType` support |
+| `schema_of_variant_agg` | 🔜 | — | Requires `VariantType` support |
| `session_user` | ✅ | — | Alias of `current_user`; resolved to a literal by the analyzer |
| `spark_partition_id` | ✅ | Native | |
-| `to_variant_object` | 🔜 | — | tracking [#4098](https://github.com/apache/datafusion-comet/issues/4098) |
+| `to_variant_object` | 🔜 | — | Requires `VariantType` support |
| `try_aes_decrypt` | ✅ | — | Routed through the JVM codegen dispatcher |
-| `try_parse_json` | 🔜 | — | tracking [#4098](https://github.com/apache/datafusion-comet/issues/4098) |
-| `try_variant_get` | 🔜 | — | tracking [#4098](https://github.com/apache/datafusion-comet/issues/4098) |
+| `try_parse_json` | 🔜 | — | Requires `VariantType` support |
+| `try_variant_get` | 🔜 | — | Requires `VariantType` support |
| `typeof` | ✅ | — | Foldable; resolved to a literal before Comet sees the plan |
| `user` | ✅ | — | Resolved to a literal by the Spark analyzer before reaching Comet |
| `uuid` | 🔜 | — | Nondeterministic random UUID |
-| `variant_get` | 🔜 | — | tracking [#4098](https://github.com/apache/datafusion-comet/issues/4098) |
+| `variant_get` | 🔜 | — | Requires `VariantType` support |
---
@@ -607,7 +607,7 @@ expression-level). The `outer` variants are wired but marked `Incompatible`; the
| `to_char` | ✅ | — | |
| `to_number` | ✅ | Codegen dispatch | |
| `to_varchar` | ✅ | — | |
-| `translate` | ✅ | Native | Falls back by default; opt-in via allowIncompatible ([#4463](https://github.com/apache/datafusion-comet/issues/4463)) |
+| `translate` | ✅ | Native | DataFusion's `translate` iterates over Unicode graphemes (Spark uses code points) and substitutes U+0000 instead of treating it as a deletion sentinel, so the native path is opt-in via allowIncompatible |
| `trim` | ✅ | Native | |
| `try_to_binary` | ✅ | — | Runs natively (rewrites to `try_eval(to_binary(...))`) |
| `try_to_number` | ✅ | Codegen dispatch | Routed through the JVM codegen dispatcher |
diff --git a/_sources/user-guide/latest/operators.md.txt b/_sources/user-guide/latest/operators.md.txt
index a29ffa1355..3425bdbb8b 100644
--- a/_sources/user-guide/latest/operators.md.txt
+++ b/_sources/user-guide/latest/operators.md.txt
@@ -45,16 +45,16 @@ omitted from the tables below and may be reconsidered based on demand:
- **Structured Streaming operators** (`StateStoreSaveExec`, `StateStoreRestoreExec`, `StreamingSymmetricHashJoinExec`, and similar): Comet targets batch execution.
- **Cartesian / cross joins** (`CartesianProductExec`): rare and expensive, with little acceleration benefit.
- **Sampling and range generation** (`SampleExec`, `RangeExec`): niche leaf operators.
-- **Pickled (non-Arrow) Python UDFs** (`BatchEvalPythonExec`): Comet accelerates Arrow-based Python UDFs only ([#4234](https://github.com/apache/datafusion-comet/issues/4234)).
+- **Pickled (non-Arrow) Python UDFs** (`BatchEvalPythonExec`): Comet accelerates Arrow-based Python UDFs only ([#4234](https://github.com/apache/datafusion-comet/pull/4234)).
## Scans
-| Operator | Status | Notes |
-| ----------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
-| `FileSourceScanExec` | ✅ | Parquet only. Some types and configurations fall back. See [Parquet Scan Compatibility](compatibility/scans.md). |
-| `BatchScanExec` | ✅ | Parquet, Apache Iceberg Parquet, and CSV (native) scans. See [Parquet Scan Compatibility](compatibility/scans.md) and the [Iceberg Guide](iceberg.md). |
-| `LocalTableScanExec` | ⚠️ | Disabled by default; there is no acceleration advantage and this operator is typically only used in test code. Can be opted into via config (#4393). |
-| `InMemoryTableScanExec` | 🔜 | Cached / in-memory table scans fall back today. |
+| Operator | Status | Notes |
+| ----------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| `FileSourceScanExec` | ✅ | Parquet only. Some types and configurations fall back. See [Parquet Scan Compatibility](compatibility/scans.md). |
+| `BatchScanExec` | ✅ | Parquet, Apache Iceberg Parquet, and CSV (native) scans. See [Parquet Scan Compatibility](compatibility/scans.md) and the [Iceberg Guide](iceberg.md). |
+| `LocalTableScanExec` | ⚠️ | Disabled by default; there is no acceleration advantage and this operator is typically only used in test code. Can be opted into via config ([#4393](https://github.com/apache/datafusion-comet/pull/4393)). |
+| `InMemoryTableScanExec` | 🔜 | Cached / in-memory table scans fall back today. |
## Projection and filtering
@@ -83,12 +83,12 @@ omitted from the tables below and may be reconsidered based on demand:
## Joins
-| Operator | Status | Notes |
-| ----------------------------- | ------ | --------------------------------------------------------------------------------------------------------------------- |
-| `BroadcastHashJoinExec` | ✅ | |
-| `ShuffledHashJoinExec` | ✅ | |
-| `SortMergeJoinExec` | ✅ | |
-| `BroadcastNestedLoopJoinExec` | ✅ | Falls back to Spark when the preserved side is broadcast (for example LEFT OUTER with BROADCAST on the left) (#4429). |
+| Operator | Status | Notes |
+| ----------------------------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `BroadcastHashJoinExec` | ✅ | |
+| `ShuffledHashJoinExec` | ✅ | |
+| `SortMergeJoinExec` | ✅ | |
+| `BroadcastNestedLoopJoinExec` | ✅ | Falls back to Spark when the preserved side is broadcast (for example LEFT OUTER with BROADCAST on the left) ([#4429](https://github.com/apache/datafusion-comet/pull/4429)). |
## Exchanges
@@ -121,9 +121,9 @@ omitted from the tables below and may be reconsidered based on demand:
## Python and UDF
-| Operator | Status | Notes |
-| --------------------------------------------------------------------------------------- | ------ | -------------------------------------------------------------------- |
-| `ArrowEvalPythonExec`, `MapInArrowExec`, `MapInPandasExec`, `FlatMapGroupsInPandasExec` | 🔜 | Experimental accelerated PyArrow UDF support is in progress (#4234). |
+| Operator | Status | Notes |
+| --------------------------------------------------------------------------------------- | ------ | ---------------------------------------------------------------------------------------------------------------------------- |
+| `ArrowEvalPythonExec`, `MapInArrowExec`, `MapInPandasExec`, `FlatMapGroupsInPandasExec` | 🔜 | Experimental accelerated PyArrow UDF support is in progress ([#4234](https://github.com/apache/datafusion-comet/pull/4234)). |
## See also
diff --git a/_sources/user-guide/latest/pyarrow-udfs.md.txt b/_sources/user-guide/latest/pyarrow-udfs.md.txt
index ddfa9cf8bd..418136e133 100644
--- a/_sources/user-guide/latest/pyarrow-udfs.md.txt
+++ b/_sources/user-guide/latest/pyarrow-udfs.md.txt
@@ -48,9 +48,10 @@ operators in the physical plan and replaces them with `CometMapInBatchExec`, whi
- Keeps the Python output in columnar format for downstream operators
This eliminates the ColumnarToRow transition and the output row conversion, reducing CPU overhead
-and memory allocations. The internal row-to-Arrow IPC re-encoding inside Spark's
-`ArrowPythonRunner` is unchanged in this version; full round-trip elimination is tracked in
-[#4240](https://github.com/apache/datafusion-comet/issues/4240).
+and memory allocations. The row-to-Arrow re-encoding that Spark's `ArrowPythonRunner` performed on
+the input side is also gone: `CometArrowPythonRunner` consumes `ColumnarBatch` directly, so batches
+are written straight from Comet's vectors into the IPC root. See [Limitations](#limitations) for the
+copies that remain.
### Plan flow
diff --git a/_sources/user-guide/latest/tuning.md.txt b/_sources/user-guide/latest/tuning.md.txt
index 6f0e9a684b..4e167f8eac 100644
--- a/_sources/user-guide/latest/tuning.md.txt
+++ b/_sources/user-guide/latest/tuning.md.txt
@@ -216,7 +216,7 @@ mismatch is amplified; setting `spark.sql.files.maxPartitionBytes` below 120 MB
distributes row groups across more splits and reduces the number of idle tasks. Smaller values produce
more splits overall, so some idle tasks may remain — tune the value against your file layout.
-See [issue #3817](https://github.com/apache/datafusion-comet/issues/3817#issuecomment-4193279630) for a
+See [#3817](https://github.com/apache/datafusion-comet/issues/3817#issuecomment-4193279630) for a
worked example and further discussion.
## Explain Plan
diff --git a/contributor-guide/adding_a_new_spark_version.html b/contributor-guide/adding_a_new_spark_version.html
index 0ce8329638..aedbf9e8f5 100644
--- a/contributor-guide/adding_a_new_spark_version.html
+++ b/contributor-guide/adding_a_new_spark_version.html
@@ -639,7 +639,7 @@ keeps the CI cost of stage 1 small and prevents test failures on the new
version from blocking unrelated PRs.</p>
<p>When CI capacity is constrained (the macOS runners in particular), it is
acceptable to drop an older minor version from the macOS PR matrix while a
-preview version is being stabilized. PR #4104 (“ci: reduce macOS PR matrix
+preview version is being stabilized. PR <a class="reference external" href="https://github.com/apache/datafusion-comet/pull/4104">#4104</a> (“ci: reduce macOS PR matrix
to single Spark 4.0 profile”) is a precedent for this kind of trim.</p>
</section>
<section id="what-to-avoid-in-stage-1">
diff --git a/contributor-guide/bug_triage.html b/contributor-guide/bug_triage.html
index 024abc9f8a..5553354915 100644
--- a/contributor-guide/bug_triage.html
+++ b/contributor-guide/bug_triage.html
@@ -749,14 +749,17 @@ inputs are literals (e.g., when <code class="docutils literal notranslate"><span
<h3>Aggregate Planning Bugs<a class="headerlink" href="#aggregate-planning-bugs" title="Link to this heading">#</a></h3>
<p>Several bugs relate to how Comet plans hash aggregates across stage boundaries. The key issue is
that Spark’s AQE may materialize a Comet partial aggregate but then run the final aggregate in
-Spark (or vice versa), and the intermediate formats may not be compatible. See the
-<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/2892">EPIC #2892</a> for the full picture.</p>
+Spark (or vice versa), and the intermediate formats may not be compatible. The closed
+EPIC <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/2892">#2892</a> collects the historical
+reports and is a good starting point when a new one comes in.</p>
</section>
<section id="native-writer-bugs">
<h3>Native Writer Bugs<a class="headerlink" href="#native-writer-bugs" title="Link to this heading">#</a></h3>
-<p>The native Parquet writer has a cluster of known test failures tracked as individual issues
-(#3417–#3430). These are lower priority since the native writer is still maturing, but they
-should be addressed before the writer is promoted to production-ready status.</p>
+<p>The native Parquet writer has a cluster of known test failures tracked as individual issues, from
+<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/3417">#3417</a> to
+<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/3430">#3430</a>. These are lower priority since the
+native writer is still maturing, but they should be addressed before the writer is promoted to
+production-ready status.</p>
</section>
</section>
<section id="how-to-help-with-triage">
diff --git a/contributor-guide/expression-audits/agg_funcs.html b/contributor-guide/expression-audits/agg_funcs.html
index 006cce6117..931d5173cb 100644
--- a/contributor-guide/expression-audits/agg_funcs.html
+++ b/contributor-guide/expression-audits/agg_funcs.html
@@ -619,7 +619,7 @@ under the License.
<li><p>Spark 3.5.8 (audited 2026-06-24): ordering centralized via <code class="docutils literal notranslate"><span class="pre">PhysicalDataType.ordering</span></code>; behavior identical to 3.4.3.</p></li>
<li><p>Spark 4.0.1 (audited 2026-06-24): adds <code class="docutils literal notranslate"><span class="pre">PercentileCont</span></code>/<code class="docutils literal notranslate"><span class="pre">PercentileDisc</span></code> builders and <code class="docutils literal notranslate"><span class="pre">SupportsOrderingWithinGroup</span></code>, enabling <code class="docutils literal notranslate"><span class="pre">percentile_cont(p)</span> <span class="pre">WITHIN</span> <span class="pre">GRO [...]
<li><p>Spark 4.1.1 (audited 2026-06-24): identical to 4.0.1.</p></li>
-<li><p><code class="docutils literal notranslate"><span class="pre">CometPercentile</span></code> reports <code class="docutils literal notranslate"><span class="pre">Incompatible</span></code> for the otherwise-supported form because DataFusion’s <code class="docutils literal notranslate"><span class="pre">percentile_cont</span></code> quantizes the interpolation weight to 6 decimal places (<code class="docutils literal notranslate"><span class="pre">INTERPOLATION_PRECISION</span> <span [...]
+<li><p><code class="docutils literal notranslate"><span class="pre">CometPercentile</span></code> reports <code class="docutils literal notranslate"><span class="pre">Compatible</span></code> for the single-literal-percentage, default-frequency, numeric-input, ascending form and runs it natively by default. Every other form is <code class="docutils literal notranslate"><span class="pre">Unsupported</span></code> and falls back to Spark.</p></li>
</ul>
</section>
</section>
diff --git a/contributor-guide/expression-audits/array_funcs.html b/contributor-guide/expression-audits/array_funcs.html
index 14102cf0d5..d77071f02c 100644
--- a/contributor-guide/expression-audits/array_funcs.html
+++ b/contributor-guide/expression-audits/array_funcs.html
@@ -556,7 +556,7 @@ under the License.
<h2>array<a class="headerlink" href="#array" title="Link to this heading">#</a></h2>
<ul class="simple">
<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
-<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">CreateArray(children,</span> <span class="pre">useStringTypeWhenEmpty)</span></code>; element type is the common type of children. Comet routes via <code class="docutils literal notranslate"><span class="pre">CometCreateArray</span></code> (native <code class="docutils literal notranslate"><span class="pre">make_array</span></code>) and special-cases the empty-array case to dod [...]
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">CreateArray(children,</span> <span class="pre">useStringTypeWhenEmpty)</span></code>; element type is the common type of children. Comet routes via <code class="docutils literal notranslate"><span class="pre">CometCreateArray</span></code> (native <code class="docutils literal notranslate"><span class="pre">make_array</span></code>) and special-cases the empty-array case to dod [...]
<li><p>Spark 4.0.1 (audited 2026-05-27): semantics unchanged.</p></li>
<li><p>Spark 4.1.1 (audited 2026-05-27): adds <code class="docutils literal notranslate"><span class="pre">contextIndependentFoldable</span></code> override; runtime semantics unchanged.</p></li>
</ul>
@@ -632,8 +632,8 @@ under the License.
<h2>array_join<a class="headerlink" href="#array-join" title="Link to this heading">#</a></h2>
<ul class="simple">
<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
-<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">ArrayJoin(array,</span> <span class="pre">delimiter,</span> <span class="pre">nullReplacement)</span></code>. Comet routes via <code class="docutils literal notranslate"><span class="pre">CometArrayJoin</span></code> to DataFusion’s <code class="docutils literal notranslate"><span class="pre">array_to_string</span></code> and is unconditionally flagged <code class="docutils lit [...]
-<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code> widened to <code class="docutils literal notranslate"><span class="pre">AbstractArrayType(StringTypeWithCollation(supportsTrimCollation</span> <span class="pre">=</span> <span class="pre">true))</span></code>; non-binary collations not propagated (https://github.com/apache/datafusion-comet/issues/2190).</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">ArrayJoin(array,</span> <span class="pre">delimiter,</span> <span class="pre">nullReplacement)</span></code>. Comet routes via <code class="docutils literal notranslate"><span class="pre">CometArrayJoin</span></code> to DataFusion’s <code class="docutils literal notranslate"><span class="pre">array_to_string</span></code> and is unconditionally flagged <code class="docutils lit [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code> widened to <code class="docutils literal notranslate"><span class="pre">AbstractArrayType(StringTypeWithCollation(supportsTrimCollation</span> <span class="pre">=</span> <span class="pre">true))</span></code>; non-binary collations not propagated (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/2190">#2190</a>).</p></li>
<li><p>Spark 4.1.1 (audited 2026-05-27): adds <code class="docutils literal notranslate"><span class="pre">contextIndependentFoldable</span></code> override; runtime unchanged.</p></li>
</ul>
</section>
diff --git a/contributor-guide/expression-audits/bitwise_funcs.html b/contributor-guide/expression-audits/bitwise_funcs.html
index f0c9ae86df..466837d5b4 100644
--- a/contributor-guide/expression-audits/bitwise_funcs.html
+++ b/contributor-guide/expression-audits/bitwise_funcs.html
@@ -628,7 +628,7 @@ under the License.
<h2>shiftrightunsigned<a class="headerlink" href="#shiftrightunsigned" title="Link to this heading">#</a></h2>
<ul class="simple">
<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
-<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">ShiftRightUnsigned(left,</span> <span class="pre">right)</span> <span class="pre">extends</span> <span class="pre">BinaryExpression</span></code>; uses Java’s <code class="docutils literal notranslate"><span class="pre">>>></span></code> (zero-fill) semantics. Comet wires as <code class="docutils literal notranslate"><span class="pre">CometScalarFunction("shiftrig [...]
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">ShiftRightUnsigned(left,</span> <span class="pre">right)</span> <span class="pre">extends</span> <span class="pre">BinaryExpression</span></code>; uses Java’s <code class="docutils literal notranslate"><span class="pre">>>></span></code> (zero-fill) semantics. Comet wires as <code class="docutils literal notranslate"><span class="pre">CometScalarFunction("shiftrig [...]
<li><p>Spark 4.0.1 (audited 2026-05-27): now extends the shared <code class="docutils literal notranslate"><span class="pre">BitShiftOperation</span></code> trait; eval semantics unchanged. The <code class="docutils literal notranslate"><span class="pre">>>></span></code> operator alias is added at the parser level.</p></li>
<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
</ul>
diff --git a/contributor-guide/expression-audits/collection_funcs.html b/contributor-guide/expression-audits/collection_funcs.html
index d20d067b1e..acd15124bd 100644
--- a/contributor-guide/expression-audits/collection_funcs.html
+++ b/contributor-guide/expression-audits/collection_funcs.html
@@ -565,7 +565,7 @@ under the License.
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">Concat(children)</span> <span class="pre">extends</span> <span class="pre">ComplexTypeMergingExpression</span> <span class="pre">with</span> <span class="pre">QueryErrorsBase</span></code>; <code class="docutils literal notranslate"><span class="pre">allowedTypes</span> <span class="pre">=</span> <span class="pre">Seq(StringType,</span> <span class="pre">BinaryType,</span> <spa [...]
<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">allowedTypes</span></code> widens <code class="docutils literal notranslate"><span class="pre">StringType</span></code> to <code class="docutils literal notranslate"><span class="pre">StringTypeWithCollation(supportsTrimCollation</span> <span class="pre">=</span> <span class="pre">true)</span></code>. Error-formatting helper changes from <code class="docutils literal notranslate"><span c [...]
<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
-<li><p>Known limitation: Comet only supports <code class="docutils literal notranslate"><span class="pre">StringType</span></code> children natively; <code class="docutils literal notranslate"><span class="pre">BinaryType</span></code> and <code class="docutils literal notranslate"><span class="pre">ArrayType</span></code> inputs fall back to Spark (https://github.com/apache/datafusion-comet/issues/4471). Non-default Spark 4.0 string collations are not propagated (https://github.com/apac [...]
+<li><p>Known limitation: Comet only supports <code class="docutils literal notranslate"><span class="pre">StringType</span></code> children natively; <code class="docutils literal notranslate"><span class="pre">BinaryType</span></code> and <code class="docutils literal notranslate"><span class="pre">ArrayType</span></code> inputs fall back to Spark (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4471">#4471</a>). Non-default Spark 4.0 string collati [...]
</ul>
</section>
<section id="reverse">
@@ -575,7 +575,7 @@ under the License.
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">Reverse(child)</span> <span class="pre">extends</span> <span class="pre">UnaryExpression</span> <span class="pre">with</span> <span class="pre">ImplicitCastInputTypes</span> <span class="pre">with</span> <span class="pre">NullIntolerant</span></code>; <code class="docutils literal notranslate"><span class="pre">inputTypes</span> <span class="pre">=</span> <span class="pre">Seq( [...]
<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">NullIntolerant</span></code> trait replaced by <code class="docutils literal notranslate"><span class="pre">override</span> <span class="pre">def</span> <span class="pre">nullIntolerant:</span> <span class="pre">Boolean</span> <span class="pre">=</span> <span class="pre">true</span></code>; <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code> widened to < [...]
<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
-<li><p>Known limitation: <code class="docutils literal notranslate"><span class="pre">Reverse</span></code> on an array containing <code class="docutils literal notranslate"><span class="pre">BinaryType</span></code> elements is reported as <code class="docutils literal notranslate"><span class="pre">Incompatible</span></code> and falls back unless explicitly enabled (https://github.com/apache/datafusion-comet/issues/2763).</p></li>
+<li><p>Known limitation: <code class="docutils literal notranslate"><span class="pre">Reverse</span></code> on an array containing <code class="docutils literal notranslate"><span class="pre">BinaryType</span></code> elements is reported as <code class="docutils literal notranslate"><span class="pre">Incompatible</span></code> and falls back unless explicitly enabled (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/2763">#2763</a>).</p></li>
</ul>
</section>
<section id="size">
@@ -585,8 +585,8 @@ under the License.
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">Size(child,</span> <span class="pre">legacySizeOfNull)</span> <span class="pre">extends</span> <span class="pre">UnaryExpression</span> <span class="pre">with</span> <span class="pre">ExpectsInputTypes</span></code>; <code class="docutils literal notranslate"><span class="pre">inputTypes</span> <span class="pre">=</span> <span class="pre">Seq(TypeCollection(ArrayType,</span> <s [...]
<li><p>Spark 4.0.1 (audited 2026-05-27): byte-for-byte identical to 3.5.8.</p></li>
<li><p>Spark 4.1.1 (audited 2026-05-27): byte-for-byte identical to 3.5.8.</p></li>
-<li><p>Known limitation: <code class="docutils literal notranslate"><span class="pre">Size</span></code> over <code class="docutils literal notranslate"><span class="pre">MapType</span></code> falls back to Spark (https://github.com/apache/datafusion-comet/issues/4472).</p></li>
-<li><p>Performance (tuned 2026-07-10, PR #4877): compute list row sizes from the offset buffer instead of allocating a sliced <code class="docutils literal notranslate"><span class="pre">ArrayRef</span></code> per row via <code class="docutils literal notranslate"><span class="pre">list_array.value(i)</span></code>, removing one heap allocation per row. ~94% faster. Benchmark: <code class="docutils literal notranslate"><span class="pre">benches/array_size.rs</span></code>.</p></li>
+<li><p>Both <code class="docutils literal notranslate"><span class="pre">ArrayType</span></code> and <code class="docutils literal notranslate"><span class="pre">MapType</span></code> inputs are <code class="docutils literal notranslate"><span class="pre">Compatible</span></code> and run natively; every other child type is <code class="docutils literal notranslate"><span class="pre">Unsupported</span></code>.</p></li>
+<li><p>Performance (tuned 2026-07-10, PR <a class="reference external" href="https://github.com/apache/datafusion-comet/pull/4877">#4877</a>): compute list row sizes from the offset buffer instead of allocating a sliced <code class="docutils literal notranslate"><span class="pre">ArrayRef</span></code> per row via <code class="docutils literal notranslate"><span class="pre">list_array.value(i)</span></code>, removing one heap allocation per row. ~94% faster. Benchmark: <code class="docut [...]
</ul>
</section>
</section>
diff --git a/contributor-guide/expression-audits/conversion_funcs.html b/contributor-guide/expression-audits/conversion_funcs.html
index 2fb8948e34..c48a408717 100644
--- a/contributor-guide/expression-audits/conversion_funcs.html
+++ b/contributor-guide/expression-audits/conversion_funcs.html
@@ -561,17 +561,17 @@ under the License.
<li><p>Spark 4.1.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">TimeType</span></code> added; many <code class="docutils literal notranslate"><span class="pre">TimeType</span></code> arms in <code class="docutils literal notranslate"><span class="pre">canCast</span></code>/<code class="docutils literal notranslate"><span class="pre">canAnsiCast</span></code>. Geospatial <code class="docutils literal notranslate"><span class="pre">GeographyType</span> [...]
<li><p>Known divergences and gaps:</p>
<ul>
-<li><p><code class="docutils literal notranslate"><span class="pre">CAST(<binary></span> <span class="pre">AS</span> <span class="pre">STRING)</span></code> decodes the bytes with <code class="docutils literal notranslate"><span class="pre">decode_utf8_spark_lossy</span></code>, replacing invalid UTF-8 with <code class="docutils literal notranslate"><span class="pre">U+FFFD</span></code> exactly as the JVM’s <code class="docutils literal notranslate"><span class="pre">new</span> <s [...]
-<li><p>Spark 4.0 collated <code class="docutils literal notranslate"><span class="pre">StringType</span></code> is not explicitly guarded; pattern equality is expected to keep collated-string casts falling back, but there is no test (https://github.com/apache/datafusion-comet/issues/4489; umbrella #2190).</p></li>
-<li><p>Spark 4.1 <code class="docutils literal notranslate"><span class="pre">TimeType</span></code> casts have no explicit <code class="docutils literal notranslate"><span class="pre">Unsupported</span></code> arm; they fall back implicitly but do not appear in the auto-generated compatibility doc (https://github.com/apache/datafusion-comet/issues/4490).</p></li>
-<li><p><code class="docutils literal notranslate"><span class="pre">CAST(<map></span> <span class="pre">AS</span> <span class="pre"><map>)</span></code> falls back to Spark even though native <code class="docutils literal notranslate"><span class="pre">cast_map_to_map</span></code> exists (https://github.com/apache/datafusion-comet/issues/4491).</p></li>
-<li><p><code class="docutils literal notranslate"><span class="pre">spark.sql.legacy.castComplexTypesToString.enabled=true</span></code> is not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4492).</p></li>
-<li><p><code class="docutils literal notranslate"><span class="pre">CAST(<float|double></span> <span class="pre">AS</span> <span class="pre">DECIMAL)</span></code> rounding may differ from Spark (<code class="docutils literal notranslate"><span class="pre">Incompatible</span></code>, gated by <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.Cast.allowIncompatible</span></code>, tracked at https://github.com/apache/datafusion-comet/issues/1371).</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">CAST(<binary></span> <span class="pre">AS</span> <span class="pre">STRING)</span></code> decodes the bytes with <code class="docutils literal notranslate"><span class="pre">decode_utf8_spark_lossy</span></code>, replacing invalid UTF-8 with <code class="docutils literal notranslate"><span class="pre">U+FFFD</span></code> exactly as the JVM’s <code class="docutils literal notranslate"><span class="pre">new</span> <s [...]
+<li><p>Spark 4.0 collated <code class="docutils literal notranslate"><span class="pre">StringType</span></code> is not explicitly guarded; pattern equality is expected to keep collated-string casts falling back, but there is no test (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4489">#4489</a>; umbrella <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/2190">#2190</a>).</p></li>
+<li><p>Spark 4.1 <code class="docutils literal notranslate"><span class="pre">TimeType</span></code> casts have no explicit <code class="docutils literal notranslate"><span class="pre">Unsupported</span></code> arm; they fall back implicitly but do not appear in the auto-generated compatibility doc (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4490">#4490</a>).</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">CAST(<map></span> <span class="pre">AS</span> <span class="pre"><map>)</span></code> runs natively via <code class="docutils literal notranslate"><span class="pre">cast_map_to_map</span></code>; <code class="docutils literal notranslate"><span class="pre">CometCast.isSupported</span></code> recurses into the key and value casts, so the map arm inherits whichever support level the inner casts report.</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">spark.sql.legacy.castComplexTypesToString.enabled=true</span></code> is <code class="docutils literal notranslate"><span class="pre">Unsupported</span></code> for any array/map/struct-to-string cast, so those casts fall back to Spark while the flag is on. Comet only implements the default (<code class="docutils literal notranslate"><span class="pre">{}</span></code>-wrapped, NULL-rendering) formatting. The flag is intern [...]
+<li><p><code class="docutils literal notranslate"><span class="pre">CAST(<float|double></span> <span class="pre">AS</span> <span class="pre">DECIMAL)</span></code> rounding may differ from Spark (<code class="docutils literal notranslate"><span class="pre">Incompatible</span></code>, gated by <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.Cast.allowIncompatible</span></code>, tracked at <a class="reference external" href="https://github.com/apac [...]
</ul>
</li>
<li><p>Spark registers the type-name conversion functions (<code class="docutils literal notranslate"><span class="pre">bigint</span></code>, <code class="docutils literal notranslate"><span class="pre">binary</span></code>, <code class="docutils literal notranslate"><span class="pre">boolean</span></code>, <code class="docutils literal notranslate"><span class="pre">date</span></code>, <code class="docutils literal notranslate"><span class="pre">decimal</span></code>, <code class="docut [...]
-<li><p>Performance (tuned 2026-07-14, PR #4920): narrowing integer casts (<code class="docutils literal notranslate"><span class="pre">spark_cast_int_to_int</span></code>) map the values buffer in a single pass with Arrow <code class="docutils literal notranslate"><span class="pre">unary</span></code>/<code class="docutils literal notranslate"><span class="pre">try_unary</span></code> and carry the null buffer over untouched, replacing an element-by-element <code class="docutils literal [...]
-<li><p>Performance (tuned 2026-07-15, PR #4940): float/double-to-decimal casts (<code class="docutils literal notranslate"><span class="pre">cast_floating_point_to_decimal128</span></code>) now convert in a single vectorized <code class="docutils literal notranslate"><span class="pre">unary_opt</span></code> pass that maps out-of-range values (NaN, infinity, precision overflow) to null, replacing the per-element <code class="docutils literal notranslate"><span class="pre">Decimal128Build [...]
+<li><p>Performance (tuned 2026-07-14, PR <a class="reference external" href="https://github.com/apache/datafusion-comet/pull/4920">#4920</a>): narrowing integer casts (<code class="docutils literal notranslate"><span class="pre">spark_cast_int_to_int</span></code>) map the values buffer in a single pass with Arrow <code class="docutils literal notranslate"><span class="pre">unary</span></code>/<code class="docutils literal notranslate"><span class="pre">try_unary</span></code> and carry [...]
+<li><p>Performance (tuned 2026-07-15, PR <a class="reference external" href="https://github.com/apache/datafusion-comet/pull/4940">#4940</a>): float/double-to-decimal casts (<code class="docutils literal notranslate"><span class="pre">cast_floating_point_to_decimal128</span></code>) now convert in a single vectorized <code class="docutils literal notranslate"><span class="pre">unary_opt</span></code> pass that maps out-of-range values (NaN, infinity, precision overflow) to null, replacin [...]
</ul>
</section>
</section>
diff --git a/contributor-guide/expression-audits/datetime_funcs.html b/contributor-guide/expression-audits/datetime_funcs.html
index 6783ad19bd..8868539368 100644
--- a/contributor-guide/expression-audits/datetime_funcs.html
+++ b/contributor-guide/expression-audits/datetime_funcs.html
@@ -582,7 +582,7 @@ under the License.
<li><p>Spark 3.4.3 (audited 2026-05-12): identical to 3.5.8.</p></li>
<li><p>Spark 3.5.8 (audited 2026-05-12): baseline.</p></li>
<li><p>Spark 4.0.1 (audited 2026-05-12): <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code> widened to <code class="docutils literal notranslate"><span class="pre">StringTypeWithCollation</span></code>; behaviour unchanged for ASCII timezone strings.</p></li>
-<li><p>Marked <code class="docutils literal notranslate"><span class="pre">Incompatible</span></code>: Comet’s native timezone parser only accepts IANA zone IDs (e.g. <code class="docutils literal notranslate"><span class="pre">America/Los_Angeles</span></code>) and fixed <code class="docutils literal notranslate"><span class="pre">+HH:MM</span></code> offsets, while Spark also accepts legacy forms (<code class="docutils literal notranslate"><span class="pre">GMT+1</span></code>, <code c [...]
+<li><p>Marked <code class="docutils literal notranslate"><span class="pre">Incompatible</span></code>: Comet’s native timezone parser only accepts IANA zone IDs (e.g. <code class="docutils literal notranslate"><span class="pre">America/Los_Angeles</span></code>) and fixed <code class="docutils literal notranslate"><span class="pre">+HH:MM</span></code> offsets, while Spark also accepts legacy forms (<code class="docutils literal notranslate"><span class="pre">GMT+1</span></code>, <code c [...]
</ul>
</section>
<section id="make-timestamp-ltz">
@@ -639,7 +639,7 @@ under the License.
<li><p>Spark 3.4.3 (audited 2026-05-12): identical to 3.5.8.</p></li>
<li><p>Spark 3.5.8 (audited 2026-05-12): baseline.</p></li>
<li><p>Spark 4.0.1 (audited 2026-05-12): <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code> widened to <code class="docutils literal notranslate"><span class="pre">StringTypeWithCollation</span></code>; behaviour unchanged for ASCII timezone strings.</p></li>
-<li><p>Marked <code class="docutils literal notranslate"><span class="pre">Incompatible</span></code>: Comet’s native timezone parser only accepts IANA zone IDs (e.g. <code class="docutils literal notranslate"><span class="pre">America/Los_Angeles</span></code>) and fixed <code class="docutils literal notranslate"><span class="pre">+HH:MM</span></code> offsets, while Spark also accepts legacy forms (<code class="docutils literal notranslate"><span class="pre">GMT+1</span></code>, <code c [...]
+<li><p>Marked <code class="docutils literal notranslate"><span class="pre">Incompatible</span></code>: Comet’s native timezone parser only accepts IANA zone IDs (e.g. <code class="docutils literal notranslate"><span class="pre">America/Los_Angeles</span></code>) and fixed <code class="docutils literal notranslate"><span class="pre">+HH:MM</span></code> offsets, while Spark also accepts legacy forms (<code class="docutils literal notranslate"><span class="pre">GMT+1</span></code>, <code c [...]
</ul>
</section>
<section id="try-make-timestamp">
diff --git a/contributor-guide/expression-audits/json_funcs.html b/contributor-guide/expression-audits/json_funcs.html
index 2c2f86f074..25d9cb705b 100644
--- a/contributor-guide/expression-audits/json_funcs.html
+++ b/contributor-guide/expression-audits/json_funcs.html
@@ -565,7 +565,7 @@ under the License.
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">BinaryExpression</span> <span class="pre">with</span> <span class="pre">ExpectsInputTypes</span> <span class="pre">with</span> <span class="pre">CodegenFallback</span></code>; <code class="docutils literal notranslate"><span class="pre">inputTypes</span> <span class="pre">=</span> <span class="pre">Seq(StringType,</span> <span class="pre">StringType)</span> <span class="pre">-& [...]
<li><p>Spark 4.0.1 (audited 2026-05-27): the eval is extracted into a <code class="docutils literal notranslate"><span class="pre">GetJsonObjectEvaluator</span></code> helper (no behaviour change). The trait set now mixes in <code class="docutils literal notranslate"><span class="pre">DefaultStringProducingExpression</span></code>, and <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code> is widened to <code class="docutils literal notranslate"><span class [...]
<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
-<li><p>Known incompatibility: Spark accepts single-quoted JSON and unescaped control characters; Comet’s native parser (built on <code class="docutils literal notranslate"><span class="pre">serde_json</span></code>) rejects both, so those inputs require <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.GetJsonObject.allowIncompatible=true</span></code> and may still produce different results. Non-default Spark 4.0 string collations are not propagated (ht [...]
+<li><p>Known incompatibility: Spark accepts single-quoted JSON and unescaped control characters; Comet’s native parser (built on <code class="docutils literal notranslate"><span class="pre">serde_json</span></code>) rejects both, so those inputs require <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.GetJsonObject.allowIncompatible=true</span></code> and may still produce different results. Non-default Spark 4.0 string collations are not propagated (<a [...]
</ul>
</section>
<section id="json-array-length">
@@ -580,7 +580,7 @@ under the License.
<h2>to_json<a class="headerlink" href="#to-json" title="Link to this heading">#</a></h2>
<ul class="simple">
<li><p>Partial native support; options and map/array inputs fall back.</p></li>
-<li><p>Performance (tuned 2026-07-13, PR #4902): <code class="docutils literal notranslate"><span class="pre">escape_string</span></code> returns <code class="docutils literal notranslate"><span class="pre">Cow</span></code> (zero-alloc borrow when nothing needs escaping) and bulk-copies unescaped byte runs instead of pushing char-by-char per value. 2x faster. Benchmark: <code class="docutils literal notranslate"><span class="pre">benches/to_json.rs</span></code>.</p></li>
+<li><p>Performance (tuned 2026-07-13, PR <a class="reference external" href="https://github.com/apache/datafusion-comet/pull/4902">#4902</a>): <code class="docutils literal notranslate"><span class="pre">escape_string</span></code> returns <code class="docutils literal notranslate"><span class="pre">Cow</span></code> (zero-alloc borrow when nothing needs escaping) and bulk-copies unescaped byte runs instead of pushing char-by-char per value. 2x faster. Benchmark: <code class="docutils li [...]
</ul>
</section>
</section>
diff --git a/contributor-guide/expression-audits/map_funcs.html b/contributor-guide/expression-audits/map_funcs.html
index d1a2dacc2f..c056571984 100644
--- a/contributor-guide/expression-audits/map_funcs.html
+++ b/contributor-guide/expression-audits/map_funcs.html
@@ -583,7 +583,7 @@ under the License.
<h2>map_from_arrays<a class="headerlink" href="#map-from-arrays" title="Link to this heading">#</a></h2>
<ul class="simple">
<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
-<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">MapFromArrays(left,</span> <span class="pre">right)</span> <span class="pre">extends</span> <span class="pre">BinaryExpression</span> <span class="pre">with</span> <span class="pre">NullIntolerant</span></code>; Spark uses <code class="docutils literal notranslate"><span class="pre">ArrayBasedMapBuilder</span></code> to detect duplicate keys (subject to <code class="docutils li [...]
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">MapFromArrays(left,</span> <span class="pre">right)</span> <span class="pre">extends</span> <span class="pre">BinaryExpression</span> <span class="pre">with</span> <span class="pre">NullIntolerant</span></code>; Spark uses <code class="docutils literal notranslate"><span class="pre">ArrayBasedMapBuilder</span></code> to detect duplicate keys (subject to <code class="docutils li [...]
<li><p>Spark 4.0.1 (audited 2026-05-27): semantics unchanged; <code class="docutils literal notranslate"><span class="pre">NullIntolerant</span></code> trait replaced by <code class="docutils literal notranslate"><span class="pre">nullIntolerant:</span> <span class="pre">Boolean</span></code>.</p></li>
<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
</ul>
@@ -622,7 +622,7 @@ under the License.
<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">StringToMap(text,</span> <span class="pre">pairDelim,</span> <span class="pre">keyValueDelim)</span> <span class="pre">extends</span> <span class="pre">TernaryExpression</span></code>; splits <code class="docutils literal notranslate"><span class="pre">text</span></code> on <code class="docutils literal notranslate"><span class="pre">pairDelim</span></code>, then each pair on < [...]
<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code> widened to <code class="docutils literal notranslate"><span class="pre">StringTypeNonCSAICollation</span></code>; uses <code class="docutils literal notranslate"><span class="pre">CollationAwareUTF8String.splitSQL</span></code> with a <code class="docutils literal notranslate"><span class="pre">collationId</span></code>. Runtime unchanged for <code class="docutil [...]
-<li><p>Spark 4.1.1 (audited 2026-05-27): adds the <code class="docutils literal notranslate"><span class="pre">legacySplitTruncate</span></code> flag (driven by <code class="docutils literal notranslate"><span class="pre">spark.sql.legacy.truncateForEmptyRegexSplit</span></code>) to both <code class="docutils literal notranslate"><span class="pre">splitSQL</span></code> calls (https://github.com/apache/datafusion-comet/issues/4477). The Comet native impl does not honour this flag; behavi [...]
+<li><p>Spark 4.1.1 (audited 2026-05-27): adds the <code class="docutils literal notranslate"><span class="pre">legacySplitTruncate</span></code> flag (driven by <code class="docutils literal notranslate"><span class="pre">spark.sql.legacy.truncateForEmptyRegexSplit</span></code>) to both <code class="docutils literal notranslate"><span class="pre">splitSQL</span></code> calls. The Comet native impl always behaves as if the flag were false, so <code class="docutils literal notranslate"><s [...]
</ul>
</section>
</section>
diff --git a/contributor-guide/expression-audits/math_funcs.html b/contributor-guide/expression-audits/math_funcs.html
index 0e68e9d7cf..db25ffd930 100644
--- a/contributor-guide/expression-audits/math_funcs.html
+++ b/contributor-guide/expression-audits/math_funcs.html
@@ -555,7 +555,7 @@ under the License.
<section id="id1">
<h2><code class="docutils literal notranslate"><span class="pre">%</span></code><a class="headerlink" href="#id1" title="Link to this heading">#</a></h2>
<ul class="simple">
-<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">Remainder(left,</span> <span class="pre">right,</span> <span class="pre">evalMode)</span></code> signature identical across versions. Native path uses Rust <code class="docutils literal notranslate"><span class="pre">spark_modulo</span></code> UDF; non-ANSI returns NULL on divide-by-zero, ANSI raises <code class="docutils literal notranslate"><span class="pre">DIVIDE [...]
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">Remainder(left,</span> <span class="pre">right,</span> <span class="pre">evalMode)</span></code> signature identical across versions. Native path uses Rust <code class="docutils literal notranslate"><span class="pre">spark_modulo</span></code> UDF; non-ANSI returns NULL on divide-by-zero, ANSI raises <code class="docutils literal notranslate"><span class="pre">DIVIDE [...]
</ul>
</section>
<section id="id2">
@@ -586,7 +586,7 @@ under the License.
<h2>CheckOverflow (internal)<a class="headerlink" href="#checkoverflow-internal" title="Link to this heading">#</a></h2>
<p>Internal decimal wrapper emitted around every decimal <code class="docutils literal notranslate"><span class="pre">+</span> <span class="pre">-</span> <span class="pre">*</span> <span class="pre">/</span></code>, <code class="docutils literal notranslate"><span class="pre">sum</span></code>, and <code class="docutils literal notranslate"><span class="pre">avg</span></code> result to null out (non-ANSI) or raise on (ANSI) values that exceed the declared precision. Native impl: <code cl [...]
<ul class="simple">
-<li><p>Performance (tuned 2026-07-15, PR #4937): both the ANSI and non-ANSI paths share a no-overflow fast path built on <code class="docutils literal notranslate"><span class="pre">is_valid_decimal_precision</span></code>, a small inlined bounds check scanned with <code class="docutils literal notranslate"><span class="pre">all</span></code> (short-circuits at the first overflow). When nothing overflows (the common shape) the input buffers are reused via <code class="docutils literal no [...]
+<li><p>Performance (tuned 2026-07-15, PR <a class="reference external" href="https://github.com/apache/datafusion-comet/pull/4937">#4937</a>): both the ANSI and non-ANSI paths share a no-overflow fast path built on <code class="docutils literal notranslate"><span class="pre">is_valid_decimal_precision</span></code>, a small inlined bounds check scanned with <code class="docutils literal notranslate"><span class="pre">all</span></code> (short-circuits at the first overflow). When nothing [...]
</ul>
</section>
<section id="abs">
@@ -701,7 +701,7 @@ under the License.
<h2>DecimalRescaleCheckOverflow (internal)<a class="headerlink" href="#decimalrescalecheckoverflow-internal" title="Link to this heading">#</a></h2>
<p>Internal fused expression that rescales a Decimal128 value (changing scale) and checks output precision in one pass, replacing the <code class="docutils literal notranslate"><span class="pre">CheckOverflow(Cast(expr,</span> <span class="pre">Decimal128(p,</span> <span class="pre">s)))</span></code> pattern used by decimal-to-decimal casts. Native impl: <code class="docutils literal notranslate"><span class="pre">math_funcs/internal/decimal_rescale_check.rs</span></code>.</p>
<ul class="simple">
-<li><p>Performance (tuned 2026-07-15, PR #4938): the legacy path ran <code class="docutils literal notranslate"><span class="pre">null_if_overflow_precision</span></code> (a second full pass that allocates a new array) on every batch to turn overflow sentinels into nulls, even when nothing overflowed. Now that pass runs only when a sentinel is present (<code class="docutils literal notranslate"><span class="pre">contains(&i128::MAX)</span></code>, short-circuiting), so the common no- [...]
+<li><p>Performance (tuned 2026-07-15, PR <a class="reference external" href="https://github.com/apache/datafusion-comet/pull/4938">#4938</a>): the legacy path ran <code class="docutils literal notranslate"><span class="pre">null_if_overflow_precision</span></code> (a second full pass that allocates a new array) on every batch to turn overflow sentinels into nulls, even when nothing overflowed. Now that pass runs only when a sentinel is present (<code class="docutils literal notranslate"> [...]
</ul>
</section>
<section id="e">
@@ -855,7 +855,7 @@ under the License.
<section id="shiftleft">
<h2>shiftleft<a class="headerlink" href="#shiftleft" title="Link to this heading">#</a></h2>
<ul class="simple">
-<li><p>See <code class="docutils literal notranslate"><span class="pre">bitwise_funcs</span> <span class="pre">/</span> <span class="pre"><<</span></code> (audited in PR #4479). Same support as the operator alias added in 4.0.</p></li>
+<li><p>See <code class="docutils literal notranslate"><span class="pre">bitwise_funcs</span> <span class="pre">/</span> <span class="pre"><<</span></code> (audited in PR <a class="reference external" href="https://github.com/apache/datafusion-comet/pull/4479">#4479</a>). Same support as the operator alias added in 4.0.</p></li>
</ul>
</section>
<section id="sign">
@@ -928,7 +928,7 @@ under the License.
<h2>unhex<a class="headerlink" href="#unhex" title="Link to this heading">#</a></h2>
<ul class="simple">
<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">Unhex(child,</span> <span class="pre">failOnError)</span></code>. Spark 4.x widens input to <code class="docutils literal notranslate"><span class="pre">StringTypeWithCollation</span></code> and wraps the inner call in try/catch; Comet <code class="docutils literal notranslate"><span class="pre">CometUnhex</span></code> forwards <code class="docutils literal notransl [...]
-<li><p>Performance (tuned 2026-07-11, PR #4876): compile-time 256-entry hex lookup table plus a preallocated <code class="docutils literal notranslate"><span class="pre">BinaryBuilder</span></code>, cutting per-byte branching and repeated buffer reallocations. Up to 31% faster on long strings. Benchmark: <code class="docutils literal notranslate"><span class="pre">benches/unhex.rs</span></code>.</p></li>
+<li><p>Performance (tuned 2026-07-11, PR <a class="reference external" href="https://github.com/apache/datafusion-comet/pull/4876">#4876</a>): compile-time 256-entry hex lookup table plus a preallocated <code class="docutils literal notranslate"><span class="pre">BinaryBuilder</span></code>, cutting per-byte branching and repeated buffer reallocations. Up to 31% faster on long strings. Benchmark: <code class="docutils literal notranslate"><span class="pre">benches/unhex.rs</span></code>. [...]
</ul>
</section>
<section id="width-bucket">
@@ -936,7 +936,7 @@ under the License.
<ul class="simple">
<li><p>Spark 3.5.8 (audited 2026-05-27): introduced; not available in 3.4.3.</p></li>
<li><p>Spark 4.0.1, 4.1.1 (audited 2026-05-27): same semantics; <code class="docutils literal notranslate"><span class="pre">NullIntolerant</span></code> -> <code class="docutils literal notranslate"><span class="pre">nullIntolerant:</span> <span class="pre">Boolean</span></code> refactor.</p></li>
-<li><p>Known limitation: wired via per-version <code class="docutils literal notranslate"><span class="pre">CometExprShim</span></code> rather than a <code class="docutils literal notranslate"><span class="pre">CometExpressionSerde</span></code>, so it bypasses the support-level framework and the auto-generated compatibility doc (https://github.com/apache/datafusion-comet/issues/4485). Native path uses datafusion-spark <code class="docutils literal notranslate"><span class="pre">SparkWid [...]
+<li><p>Known limitation: wired via per-version <code class="docutils literal notranslate"><span class="pre">CometExprShim</span></code> rather than a <code class="docutils literal notranslate"><span class="pre">CometExpressionSerde</span></code>, so it bypasses the support-level framework and the auto-generated compatibility doc (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4485">#4485</a>). Native path uses datafusion-spark <code class="docutils [...]
</ul>
</section>
</section>
diff --git a/contributor-guide/expression-audits/predicate_funcs.html b/contributor-guide/expression-audits/predicate_funcs.html
index df470ecb4d..321d536f3a 100644
--- a/contributor-guide/expression-audits/predicate_funcs.html
+++ b/contributor-guide/expression-audits/predicate_funcs.html
@@ -690,7 +690,7 @@ under the License.
<section id="like">
<h2>like<a class="headerlink" href="#like" title="Link to this heading">#</a></h2>
<ul class="simple">
-<li><p>See <code class="docutils literal notranslate"><span class="pre">string_funcs</span> <span class="pre">/</span> <span class="pre">like</span></code> (audited in PR #4461). <code class="docutils literal notranslate"><span class="pre">CometLike</span></code> only supports the default <code class="docutils literal notranslate"><span class="pre">\\</span></code> escape character.</p></li>
+<li><p>See <code class="docutils literal notranslate"><span class="pre">string_funcs</span> <span class="pre">/</span> <span class="pre">like</span></code> (audited in PR <a class="reference external" href="https://github.com/apache/datafusion-comet/pull/4461">#4461</a>). <code class="docutils literal notranslate"><span class="pre">CometLike</span></code> only supports the default <code class="docutils literal notranslate"><span class="pre">\\</span></code> escape character.</p></li>
</ul>
</section>
<section id="not">
@@ -714,7 +714,7 @@ under the License.
<section id="rlike">
<h2>rlike<a class="headerlink" href="#rlike" title="Link to this heading">#</a></h2>
<ul class="simple">
-<li><p>See <code class="docutils literal notranslate"><span class="pre">string_funcs</span> <span class="pre">/</span> <span class="pre">regexp_replace</span></code> and the <code class="docutils literal notranslate"><span class="pre">CometRLike</span></code> notes (audited in PR #4461). Uses the Rust <code class="docutils literal notranslate"><span class="pre">regex</span></code> crate, which differs from Java’s <code class="docutils literal notranslate"><span class="pre">Pattern</span> [...]
+<li><p>See <code class="docutils literal notranslate"><span class="pre">string_funcs</span> <span class="pre">/</span> <span class="pre">regexp_replace</span></code> and the <code class="docutils literal notranslate"><span class="pre">CometRLike</span></code> notes (audited in PR <a class="reference external" href="https://github.com/apache/datafusion-comet/pull/4461">#4461</a>). Uses the Rust <code class="docutils literal notranslate"><span class="pre">regex</span></code> crate, which d [...]
</ul>
</section>
</section>
diff --git a/contributor-guide/expression-audits/string_funcs.html b/contributor-guide/expression-audits/string_funcs.html
index 8d55ecac10..54de7399fc 100644
--- a/contributor-guide/expression-audits/string_funcs.html
+++ b/contributor-guide/expression-audits/string_funcs.html
@@ -557,7 +557,7 @@ under the License.
<ul class="simple">
<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">StringType</span> <span class="pre">-></span> <span class="pre">IntegerType</span></code>; <code class="docutils literal notranslate"><span class="pre">nullSafeEval</span></code> returns <code class="docutils literal notranslate"><span class="pre">codePointAt(0)</span></code> of the first char, or <code class="docutils literal notranslate"><span class="pre">0</span></code> f [...]
-<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code> widened to <code class="docutils literal notranslate"><span class="pre">StringTypeWithCollation(supportsTrimCollation</span> <span class="pre">=</span> <span class="pre">true)</span></code>; behaviour unchanged for <code class="docutils literal notranslate"><span class="pre">UTF8_BINARY</span></code>. Comet does not propagate collation, so non-default collations [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code> widened to <code class="docutils literal notranslate"><span class="pre">StringTypeWithCollation(supportsTrimCollation</span> <span class="pre">=</span> <span class="pre">true)</span></code>; behaviour unchanged for <code class="docutils literal notranslate"><span class="pre">UTF8_BINARY</span></code>. Comet does not propagate collation, so non-default collations [...]
</ul>
</section>
<section id="bit-length">
@@ -565,7 +565,7 @@ under the License.
<ul class="simple">
<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">(StringType|BinaryType)</span> <span class="pre">-></span> <span class="pre">IntegerType</span></code>; eval returns <code class="docutils literal notranslate"><span class="pre">numBytes</span> <span class="pre">*</span> <span class="pre">8</span></code> for strings and <code class="docutils literal notranslate"><span class="pre">.length</span> <span class="pre">*</span> <sp [...]
-<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code> widened to <code class="docutils literal notranslate"><span class="pre">StringTypeWithCollation(supportsTrimCollation</span> <span class="pre">=</span> <span class="pre">true)</span></code>; semantics unchanged for <code class="docutils literal notranslate"><span class="pre">UTF8_BINARY</span></code>. Non-default collations not honoured by Comet (https://github.c [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code> widened to <code class="docutils literal notranslate"><span class="pre">StringTypeWithCollation(supportsTrimCollation</span> <span class="pre">=</span> <span class="pre">true)</span></code>; semantics unchanged for <code class="docutils literal notranslate"><span class="pre">UTF8_BINARY</span></code>. Non-default collations not honoured by Comet (<a class="refere [...]
<li><p><code class="docutils literal notranslate"><span class="pre">BinaryType</span></code> input is reported <code class="docutils literal notranslate"><span class="pre">Unsupported</span></code> in <code class="docutils literal notranslate"><span class="pre">getSupportLevel</span></code>, so <code class="docutils literal notranslate"><span class="pre">bit_length(<binary>)</span></code> falls back to Spark cleanly (DataFusion’s <code class="docutils literal notranslate"><span cla [...]
</ul>
</section>
@@ -574,7 +574,7 @@ under the License.
<ul class="simple">
<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">StringTrimBoth</span></code> is <code class="docutils literal notranslate"><span class="pre">RuntimeReplaceable</span></code> and rewritten to <code class="docutils literal notranslate"><span class="pre">StringTrim(srcStr,</span> <span class="pre">trimStr)</span></code> before serde runs. Support is provided by the <code class="docutils literal notranslate"><span class="pre">tr [...]
-<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">StringTrim</span></code> (the rewrite target) routes through <code class="docutils literal notranslate"><span class="pre">CollationSupport.StringTrim.exec</span></code> and uses <code class="docutils literal notranslate"><span class="pre">StringTypeNonCSAICollation(supportsTrimCollation</span> <span class="pre">=</span> <span class="pre">true)</span></code>; semantics unchanged for <code [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">StringTrim</span></code> (the rewrite target) routes through <code class="docutils literal notranslate"><span class="pre">CollationSupport.StringTrim.exec</span></code> and uses <code class="docutils literal notranslate"><span class="pre">StringTypeNonCSAICollation(supportsTrimCollation</span> <span class="pre">=</span> <span class="pre">true)</span></code>; semantics unchanged for <code [...]
</ul>
</section>
<section id="char">
@@ -614,7 +614,7 @@ under the License.
<ul class="simple">
<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">Seq[Expression]</span> <span class="pre">-></span> <span class="pre">StringType</span></code>; NULL separator yields NULL, NULL element values are skipped, children can be <code class="docutils literal notranslate"><span class="pre">StringType</span></code> or <code class="docutils literal notranslate"><span class="pre">ArrayType(StringType)</span></code>. Comet serde rewrit [...]
-<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code> widened to <code class="docutils literal notranslate"><span class="pre">StringTypeWithCollation</span></code> / <code class="docutils literal notranslate"><span class="pre">AbstractArrayType</span></code>; <code class="docutils literal notranslate"><span class="pre">dataType</span></code> becomes <code class="docutils literal notranslate"><span class="pre">childr [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code> widened to <code class="docutils literal notranslate"><span class="pre">StringTypeWithCollation</span></code> / <code class="docutils literal notranslate"><span class="pre">AbstractArrayType</span></code>; <code class="docutils literal notranslate"><span class="pre">dataType</span></code> becomes <code class="docutils literal notranslate"><span class="pre">childr [...]
</ul>
</section>
<section id="contains">
@@ -622,7 +622,7 @@ under the License.
<ul class="simple">
<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">UTF8String.contains</span></code> on <code class="docutils literal notranslate"><span class="pre">StringType</span></code>; the parser routes <code class="docutils literal notranslate"><span class="pre">(BinaryType,</span> <span class="pre">BinaryType)</span></code> to <code class="docutils literal notranslate"><span class="pre">BinaryPredicate</span></code>, so Comet only ever [...]
-<li><p>Spark 4.0.1 (audited 2026-05-27): routes through <code class="docutils literal notranslate"><span class="pre">CollationSupport.Contains.exec(...,</span> <span class="pre">collationId)</span></code>; behaviour identical for <code class="docutils literal notranslate"><span class="pre">UTF8_BINARY</span></code>. Non-default collations not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4496).</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): routes through <code class="docutils literal notranslate"><span class="pre">CollationSupport.Contains.exec(...,</span> <span class="pre">collationId)</span></code>; behaviour identical for <code class="docutils literal notranslate"><span class="pre">UTF8_BINARY</span></code>. Non-default collations not honoured by Comet (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4496">#4496</a>).</p></li>
</ul>
</section>
<section id="decode">
@@ -632,7 +632,7 @@ under the License.
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">StringDecode(bin,</span> <span class="pre">charset)</span></code> evaluated directly; invalid sequences silently substitute replacement characters via <code class="docutils literal notranslate"><span class="pre">new</span> <span class="pre">String(bytes,</span> <span class="pre">charset)</span></code>.</p></li>
<li><p>Spark 4.0.1 (audited 2026-05-27): refactored to <code class="docutils literal notranslate"><span class="pre">RuntimeReplaceable</span></code> whose <code class="docutils literal notranslate"><span class="pre">replacement</span></code> is a <code class="docutils literal notranslate"><span class="pre">StaticInvoke(StringDecode.decode,</span> <span class="pre">bin,</span> <span class="pre">charset,</span> <span class="pre">legacyCharsets,</span> <span class="pre">legacyErrorAction)</ [...]
<li><p><code class="docutils literal notranslate"><span class="pre">decode</span></code> runs through the codegen dispatcher on all versions (Spark 3.x via <code class="docutils literal notranslate"><span class="pre">CometStringDecode</span></code>, Spark 4.0 via the <code class="docutils literal notranslate"><span class="pre">StaticInvoke</span></code> replacement routed to <code class="docutils literal notranslate"><span class="pre">CometStaticInvokeCodegenDispatch</span></code>), so S [...]
-<li><p>Known limitation: because there is no <code class="docutils literal notranslate"><span class="pre">CometExpressionSerde[StringDecode]</span></code> registration, <code class="docutils literal notranslate"><span class="pre">decode</span></code> does not surface in the auto-generated compatibility docs (https://github.com/apache/datafusion-comet/issues/4466).</p></li>
+<li><p>Known limitation: because there is no <code class="docutils literal notranslate"><span class="pre">CometExpressionSerde[StringDecode]</span></code> registration, <code class="docutils literal notranslate"><span class="pre">decode</span></code> does not surface in the auto-generated compatibility docs (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4466">#4466</a>).</p></li>
</ul>
</section>
<section id="endswith">
@@ -640,15 +640,15 @@ under the License.
<ul class="simple">
<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">UTF8String.endsWith</span></code> on <code class="docutils literal notranslate"><span class="pre">StringType</span></code>; binary form routed to <code class="docutils literal notranslate"><span class="pre">BinaryPredicate</span></code> before Comet.</p></li>
-<li><p>Spark 4.0.1 (audited 2026-05-27): routes through <code class="docutils literal notranslate"><span class="pre">CollationSupport.EndsWith.exec</span></code>; semantics unchanged for <code class="docutils literal notranslate"><span class="pre">UTF8_BINARY</span></code>. Non-default collations not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4496).</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): routes through <code class="docutils literal notranslate"><span class="pre">CollationSupport.EndsWith.exec</span></code>; semantics unchanged for <code class="docutils literal notranslate"><span class="pre">UTF8_BINARY</span></code>. Non-default collations not honoured by Comet (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4496">#4496</a>).</p></li>
</ul>
</section>
<section id="initcap">
<h2>initcap<a class="headerlink" href="#initcap" title="Link to this heading">#</a></h2>
<ul class="simple">
<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
-<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">string.toLowerCase.toTitleCase</span></code> on <code class="docutils literal notranslate"><span class="pre">UTF8String</span></code>; word boundary is Java <code class="docutils literal notranslate"><span class="pre">Character.isWhitespace</span></code>. Comet routes to DataFusion <code class="docutils literal notranslate"><span class="pre">initcap</span></code>, which splits [...]
-<li><p>Spark 4.0.1 (audited 2026-05-27): routes through <code class="docutils literal notranslate"><span class="pre">CollationSupport.InitCap.exec</span></code> (collation- and ICU-aware) and propagates <code class="docutils literal notranslate"><span class="pre">child.dataType</span></code>. Comet ignores collation; 3.x divergences persist plus collation/ICU mismatches (https://github.com/apache/datafusion-comet/issues/4496).</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">string.toLowerCase.toTitleCase</span></code> on <code class="docutils literal notranslate"><span class="pre">UTF8String</span></code>; word boundary is Java <code class="docutils literal notranslate"><span class="pre">Character.isWhitespace</span></code>. DataFusion <code class="docutils literal notranslate"><span class="pre">initcap</span></code> splits on <code class="docutil [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): routes through <code class="docutils literal notranslate"><span class="pre">CollationSupport.InitCap.exec</span></code> (collation- and ICU-aware) and propagates <code class="docutils literal notranslate"><span class="pre">child.dataType</span></code>. Comet ignores collation; 3.x divergences persist plus collation/ICU mismatches (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4496">#4496</a>).</p></li>
</ul>
</section>
<section id="instr">
@@ -656,7 +656,7 @@ under the License.
<ul class="simple">
<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">StringInstr(str,</span> <span class="pre">substr)</span> <span class="pre">-></span> <span class="pre">IntegerType</span></code>; returns <code class="docutils literal notranslate"><span class="pre">string.indexOf(sub,</span> <span class="pre">0)</span> <span class="pre">+</span> <span class="pre">1</span></code> (1-based, 0 when not found, 1 on empty substring). Resolves to [...]
-<li><p>Spark 4.0.1 (audited 2026-05-27): routes through <code class="docutils literal notranslate"><span class="pre">CollationSupport.StringInstr.exec</span></code>; semantics unchanged for <code class="docutils literal notranslate"><span class="pre">UTF8_BINARY</span></code>. Non-default collations not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4496).</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): routes through <code class="docutils literal notranslate"><span class="pre">CollationSupport.StringInstr.exec</span></code>; semantics unchanged for <code class="docutils literal notranslate"><span class="pre">UTF8_BINARY</span></code>. Non-default collations not honoured by Comet (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4496">#4496</a>).</p></li>
</ul>
</section>
<section id="lcase">
@@ -672,7 +672,7 @@ under the License.
<ul class="simple">
<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">RuntimeReplaceable</span></code> with <code class="docutils literal notranslate"><span class="pre">replacement</span> <span class="pre">=</span> <span class="pre">Substring(str,</span> <span class="pre">Literal(1),</span> <span class="pre">len)</span></code>; accepts <code class="docutils literal notranslate"><span class="pre">StringType</span></code> or <code class="docutils l [...]
-<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code> widened with <code class="docutils literal notranslate"><span class="pre">StringTypeWithCollation</span></code>; behaviour unchanged for <code class="docutils literal notranslate"><span class="pre">UTF8_BINARY</span></code>. Non-default collations not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4496).</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code> widened with <code class="docutils literal notranslate"><span class="pre">StringTypeWithCollation</span></code>; behaviour unchanged for <code class="docutils literal notranslate"><span class="pre">UTF8_BINARY</span></code>. Non-default collations not honoured by Comet (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4496">#4 [...]
</ul>
</section>
<section id="len">
@@ -688,7 +688,7 @@ under the License.
<ul class="simple">
<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">(StringType|BinaryType)</span> <span class="pre">-></span> <span class="pre">IntegerType</span></code>; eval returns <code class="docutils literal notranslate"><span class="pre">numChars</span></code> for strings and <code class="docutils literal notranslate"><span class="pre">.length</span></code> for binary. <code class="docutils literal notranslate"><span class="pre">Bina [...]
-<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code> widened to <code class="docutils literal notranslate"><span class="pre">StringTypeWithCollation(supportsTrimCollation</span> <span class="pre">=</span> <span class="pre">true)</span></code>; semantics unchanged. Non-default collations not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4496).</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code> widened to <code class="docutils literal notranslate"><span class="pre">StringTypeWithCollation(supportsTrimCollation</span> <span class="pre">=</span> <span class="pre">true)</span></code>; semantics unchanged. Non-default collations not honoured by Comet (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4496">#4496</a>).</p></li>
</ul>
</section>
<section id="lower">
@@ -696,7 +696,7 @@ under the License.
<ul class="simple">
<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. JVM default-locale <code class="docutils literal notranslate"><span class="pre">toLowerCase</span></code> on <code class="docutils literal notranslate"><span class="pre">UTF8String</span></code>. Comet routes to DataFusion <code class="docutils literal notranslate"><span class="pre">lower</span></code> (Rust Unicode default case mapping, no locale awareness) and is unconditionally <code class="docutils literal notranslate"><span class="p [...]
-<li><p>Spark 4.0.1 (audited 2026-05-27): routes through <code class="docutils literal notranslate"><span class="pre">CollationSupport.Lower.exec(v,</span> <span class="pre">collationId,</span> <span class="pre">useICU)</span></code> with <code class="docutils literal notranslate"><span class="pre">SQLConf.ICU_CASE_MAPPINGS_ENABLED</span></code>; <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code> widened to <code class="docutils literal notranslate"><spa [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): routes through <code class="docutils literal notranslate"><span class="pre">CollationSupport.Lower.exec(v,</span> <span class="pre">collationId,</span> <span class="pre">useICU)</span></code> with <code class="docutils literal notranslate"><span class="pre">SQLConf.ICU_CASE_MAPPINGS_ENABLED</span></code>; <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code> widened to <code class="docutils literal notranslate"><spa [...]
</ul>
</section>
<section id="lpad">
@@ -704,7 +704,7 @@ under the License.
<ul class="simple">
<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">StringLPad(str,</span> <span class="pre">len,</span> <span class="pre">pad)</span> <span class="pre">-></span> <span class="pre">StringType</span></code>; <code class="docutils literal notranslate"><span class="pre">len</span> <span class="pre"><=</span> <span class="pre">0</span></code> returns the empty string, empty <code class="docutils literal notranslate"><span clas [...]
-<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">NullIntolerant</span></code> trait replaced by <code class="docutils literal notranslate"><span class="pre">override</span> <span class="pre">def</span> <span class="pre">nullIntolerant:</span> <span class="pre">Boolean</span> <span class="pre">=</span> <span class="pre">true</span></code>; <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code> widened to < [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">NullIntolerant</span></code> trait replaced by <code class="docutils literal notranslate"><span class="pre">override</span> <span class="pre">def</span> <span class="pre">nullIntolerant:</span> <span class="pre">Boolean</span> <span class="pre">=</span> <span class="pre">true</span></code>; <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code> widened to < [...]
<li><p>Known limitation: <code class="docutils literal notranslate"><span class="pre">lpad(<binary>,</span> <span class="pre">...)</span></code> is rewritten by Spark to <code class="docutils literal notranslate"><span class="pre">BinaryPad</span> <span class="pre">/</span> <span class="pre">StaticInvoke(ByteArray.lpad)</span></code> before serde runs and always falls back to Spark.</p></li>
</ul>
</section>
@@ -713,7 +713,7 @@ under the License.
<ul class="simple">
<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">StringTrimLeft</span></code> extends <code class="docutils literal notranslate"><span class="pre">String2TrimExpression</span></code>; no-arg form strips ASCII space <code class="docutils literal notranslate"><span class="pre">0x20</span></code> only. The two-arg parser form <code class="docutils literal notranslate"><span class="pre">ltrim(trimStr,</span> <span class="pre">src [...]
-<li><p>Spark 4.0.1 (audited 2026-05-27): routes through <code class="docutils literal notranslate"><span class="pre">CollationSupport.StringTrimLeft.exec</span></code>; semantics unchanged for <code class="docutils literal notranslate"><span class="pre">UTF8_BINARY</span></code>. Non-default collations not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4496).</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): routes through <code class="docutils literal notranslate"><span class="pre">CollationSupport.StringTrimLeft.exec</span></code>; semantics unchanged for <code class="docutils literal notranslate"><span class="pre">UTF8_BINARY</span></code>. Non-default collations not honoured by Comet (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4496">#4496</a>).</p></li>
</ul>
</section>
<section id="octet-length">
@@ -721,7 +721,7 @@ under the License.
<ul class="simple">
<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">(StringType|BinaryType)</span> <span class="pre">-></span> <span class="pre">IntegerType</span></code>; eval returns <code class="docutils literal notranslate"><span class="pre">numBytes</span></code> for strings and <code class="docutils literal notranslate"><span class="pre">.length</span></code> for binary.</p></li>
-<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code> widened to <code class="docutils literal notranslate"><span class="pre">StringTypeWithCollation</span></code>; semantics unchanged for <code class="docutils literal notranslate"><span class="pre">UTF8_BINARY</span></code>. Non-default collations not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4496).</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code> widened to <code class="docutils literal notranslate"><span class="pre">StringTypeWithCollation</span></code>; semantics unchanged for <code class="docutils literal notranslate"><span class="pre">UTF8_BINARY</span></code>. Non-default collations not honoured by Comet (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4496">#449 [...]
<li><p><code class="docutils literal notranslate"><span class="pre">BinaryType</span></code> input is reported <code class="docutils literal notranslate"><span class="pre">Unsupported</span></code> in <code class="docutils literal notranslate"><span class="pre">getSupportLevel</span></code>, so <code class="docutils literal notranslate"><span class="pre">octet_length(<binary>)</span></code> falls back to Spark cleanly (DataFusion’s <code class="docutils literal notranslate"><span c [...]
</ul>
</section>
@@ -739,15 +739,15 @@ under the License.
<ul class="simple">
<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">StringRepeat(str,</span> <span class="pre">times)</span></code> with <code class="docutils literal notranslate"><span class="pre">nullSafeEval(s,</span> <span class="pre">n)</span> <span class="pre">=</span> <span class="pre">s.repeat(n)</span></code>; <code class="docutils literal notranslate"><span class="pre">UTF8String.repeat</span></code> returns the empty string for <code [...]
-<li><p>Spark 4.0.1 (audited 2026-05-27): adds <code class="docutils literal notranslate"><span class="pre">nullIntolerant:</span> <span class="pre">Boolean</span></code> field; <code class="docutils literal notranslate"><span class="pre">dataType</span></code> becomes <code class="docutils literal notranslate"><span class="pre">str.dataType</span></code> (collation-tracking). Semantics unchanged for <code class="docutils literal notranslate"><span class="pre">UTF8_BINARY</span></code>. N [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): adds <code class="docutils literal notranslate"><span class="pre">nullIntolerant:</span> <span class="pre">Boolean</span></code> field; <code class="docutils literal notranslate"><span class="pre">dataType</span></code> becomes <code class="docutils literal notranslate"><span class="pre">str.dataType</span></code> (collation-tracking). Semantics unchanged for <code class="docutils literal notranslate"><span class="pre">UTF8_BINARY</span></code>. N [...]
</ul>
</section>
<section id="replace">
<h2>replace<a class="headerlink" href="#replace" title="Link to this heading">#</a></h2>
<ul class="simple">
<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
-<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">StringReplace(src,</span> <span class="pre">search,</span> <span class="pre">replace)</span></code>; when <code class="docutils literal notranslate"><span class="pre">search</span></code> is empty, Spark returns <code class="docutils literal notranslate"><span class="pre">src</span></code> unchanged (short-circuit on <code class="docutils literal notranslate"><span class="pre"> [...]
-<li><p>Spark 4.0.1 (audited 2026-05-27): routes through <code class="docutils literal notranslate"><span class="pre">CollationSupport.StringReplace.exec</span></code>; semantics unchanged for <code class="docutils literal notranslate"><span class="pre">UTF8_BINARY</span></code>. Non-default collations not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4496).</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">StringReplace(src,</span> <span class="pre">search,</span> <span class="pre">replace)</span></code>; when <code class="docutils literal notranslate"><span class="pre">search</span></code> is empty, Spark returns <code class="docutils literal notranslate"><span class="pre">src</span></code> unchanged (short-circuit on <code class="docutils literal notranslate"><span class="pre"> [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): routes through <code class="docutils literal notranslate"><span class="pre">CollationSupport.StringReplace.exec</span></code>; semantics unchanged for <code class="docutils literal notranslate"><span class="pre">UTF8_BINARY</span></code>. Non-default collations not honoured by Comet (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4496">#4496</a>).</p></li>
</ul>
</section>
<section id="right">
@@ -755,7 +755,7 @@ under the License.
<ul class="simple">
<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">RuntimeReplaceable</span></code> with <code class="docutils literal notranslate"><span class="pre">replacement</span> <span class="pre">=</span> <span class="pre">If(IsNull(str),</span> <span class="pre">null,</span> <span class="pre">If(len</span> <span class="pre"><=</span> <span class="pre">0,</span> <span class="pre">"",</span> <span class="pre">Substring(str,< [...]
-<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code> widened with collation; uses <code class="docutils literal notranslate"><span class="pre">UnaryMinus(len,</span> <span class="pre">failOnError</span> <span class="pre">=</span> <span class="pre">false)</span></code> to avoid integer-overflow exceptions on <code class="docutils literal notranslate"><span class="pre">len</span> <span class="pre">=</span> <span clas [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code> widened with collation; uses <code class="docutils literal notranslate"><span class="pre">UnaryMinus(len,</span> <span class="pre">failOnError</span> <span class="pre">=</span> <span class="pre">false)</span></code> to avoid integer-overflow exceptions on <code class="docutils literal notranslate"><span class="pre">len</span> <span class="pre">=</span> <span clas [...]
</ul>
</section>
<section id="rpad">
@@ -763,7 +763,7 @@ under the License.
<ul class="simple">
<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">StringRPad(str,</span> <span class="pre">len,</span> <span class="pre">pad)</span> <span class="pre">-></span> <span class="pre">StringType</span></code>; same edge-case behaviour as <code class="docutils literal notranslate"><span class="pre">lpad</span></code> (negative len, empty pad, NULL propagation). Comet serde requires column <code class="docutils literal notranslate [...]
-<li><p>Spark 4.0.1 (audited 2026-05-27): same evolution as <code class="docutils literal notranslate"><span class="pre">lpad</span></code>; default-pad literal type tightened; semantics unchanged for <code class="docutils literal notranslate"><span class="pre">UTF8_BINARY</span></code>. Non-default collations not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4496).</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): same evolution as <code class="docutils literal notranslate"><span class="pre">lpad</span></code>; default-pad literal type tightened; semantics unchanged for <code class="docutils literal notranslate"><span class="pre">UTF8_BINARY</span></code>. Non-default collations not honoured by Comet (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4496">#4496</a>).</p></li>
<li><p>Known limitation: same <code class="docutils literal notranslate"><span class="pre">BinaryPad</span> <span class="pre">/</span> <span class="pre">StaticInvoke</span></code> rewrite as <code class="docutils literal notranslate"><span class="pre">lpad</span></code> causes <code class="docutils literal notranslate"><span class="pre">rpad(<binary>,</span> <span class="pre">...)</span></code> to fall back.</p></li>
</ul>
</section>
@@ -772,7 +772,7 @@ under the License.
<ul class="simple">
<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">StringTrimRight</span></code> extends <code class="docutils literal notranslate"><span class="pre">String2TrimExpression</span></code>; semantically symmetric to <code class="docutils literal notranslate"><span class="pre">ltrim</span></code>.</p></li>
-<li><p>Spark 4.0.1 (audited 2026-05-27): routes through <code class="docutils literal notranslate"><span class="pre">CollationSupport.StringTrimRight.exec</span></code>; semantics unchanged for <code class="docutils literal notranslate"><span class="pre">UTF8_BINARY</span></code>. Non-default collations not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4496).</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): routes through <code class="docutils literal notranslate"><span class="pre">CollationSupport.StringTrimRight.exec</span></code>; semantics unchanged for <code class="docutils literal notranslate"><span class="pre">UTF8_BINARY</span></code>. Non-default collations not honoured by Comet (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4496">#4496</a>).</p></li>
</ul>
</section>
<section id="space">
@@ -788,7 +788,7 @@ under the License.
<ul class="simple">
<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">StringSplit(str,</span> <span class="pre">regex,</span> <span class="pre">limit)</span></code>; <code class="docutils literal notranslate"><span class="pre">limit</span> <span class="pre">></span> <span class="pre">0</span></code> permits at most <code class="docutils literal notranslate"><span class="pre">limit-1</span></code> splits, <code class="docutils literal notransla [...]
-<li><p>Spark 4.0.1 (audited 2026-05-27): wraps the regex via <code class="docutils literal notranslate"><span class="pre">CollationSupport.collationAwareRegex</span></code> and changes <code class="docutils literal notranslate"><span class="pre">dataType</span></code> to <code class="docutils literal notranslate"><span class="pre">ArrayType(str.dataType,</span> <span class="pre">...)</span></code>. Comet does not honour collation flags (https://github.com/apache/datafusion-comet/issues/4 [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): wraps the regex via <code class="docutils literal notranslate"><span class="pre">CollationSupport.collationAwareRegex</span></code> and changes <code class="docutils literal notranslate"><span class="pre">dataType</span></code> to <code class="docutils literal notranslate"><span class="pre">ArrayType(str.dataType,</span> <span class="pre">...)</span></code>. Comet does not honour collation flags (<a class="reference external" href="https://github. [...]
</ul>
</section>
<section id="startswith">
@@ -796,7 +796,7 @@ under the License.
<ul class="simple">
<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">UTF8String.startsWith</span></code> on <code class="docutils literal notranslate"><span class="pre">StringType</span></code>; binary form routed to <code class="docutils literal notranslate"><span class="pre">BinaryPredicate</span></code>.</p></li>
-<li><p>Spark 4.0.1 (audited 2026-05-27): routes through <code class="docutils literal notranslate"><span class="pre">CollationSupport.StartsWith.exec</span></code>; semantics unchanged for <code class="docutils literal notranslate"><span class="pre">UTF8_BINARY</span></code>. Non-default collations not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4496).</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): routes through <code class="docutils literal notranslate"><span class="pre">CollationSupport.StartsWith.exec</span></code>; semantics unchanged for <code class="docutils literal notranslate"><span class="pre">UTF8_BINARY</span></code>. Non-default collations not honoured by Comet (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4496">#4496</a>).</p></li>
</ul>
</section>
<section id="substr">
@@ -812,7 +812,7 @@ under the License.
<ul class="simple">
<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">TernaryExpression</span></code>; two-arg form defaults <code class="docutils literal notranslate"><span class="pre">len</span> <span class="pre">=</span> <span class="pre">Integer.MAX_VALUE</span></code>; supports <code class="docutils literal notranslate"><span class="pre">StringType</span></code> and <code class="docutils literal notranslate"><span class="pre">BinaryType</spa [...]
-<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code> widened with <code class="docutils literal notranslate"><span class="pre">StringTypeWithCollation</span></code>; semantics unchanged for <code class="docutils literal notranslate"><span class="pre">UTF8_BINARY</span></code>. <code class="docutils literal notranslate"><span class="pre">SparkSubstring</span></code> implements Spark’s 1-indexed, negative-start, and [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code> widened with <code class="docutils literal notranslate"><span class="pre">StringTypeWithCollation</span></code>; semantics unchanged for <code class="docutils literal notranslate"><span class="pre">UTF8_BINARY</span></code>. <code class="docutils literal notranslate"><span class="pre">SparkSubstring</span></code> implements Spark’s 1-indexed, negative-start, and [...]
</ul>
</section>
<section id="substring-index">
@@ -820,7 +820,7 @@ under the License.
<ul class="simple">
<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">TernaryExpression(StringType,</span> <span class="pre">StringType,</span> <span class="pre">IntegerType)</span> <span class="pre">-></span> <span class="pre">StringType</span></code>. Comet casts <code class="docutils literal notranslate"><span class="pre">count</span></code> to <code class="docutils literal notranslate"><span class="pre">LongType</span></code> and delegates [...]
-<li><p>Spark 4.0.1 (audited 2026-05-27): routes through <code class="docutils literal notranslate"><span class="pre">CollationSupport.SubstringIndex.exec</span></code> and propagates <code class="docutils literal notranslate"><span class="pre">strExpr.dataType</span></code>; semantics unchanged for <code class="docutils literal notranslate"><span class="pre">UTF8_BINARY</span></code>. Non-default collations not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4496).</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): routes through <code class="docutils literal notranslate"><span class="pre">CollationSupport.SubstringIndex.exec</span></code> and propagates <code class="docutils literal notranslate"><span class="pre">strExpr.dataType</span></code>; semantics unchanged for <code class="docutils literal notranslate"><span class="pre">UTF8_BINARY</span></code>. Non-default collations not honoured by Comet (<a class="reference external" href="https://github.com/apa [...]
</ul>
</section>
<section id="translate">
@@ -828,7 +828,7 @@ under the License.
<ul class="simple">
<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">StringTranslate(src,</span> <span class="pre">from,</span> <span class="pre">to)</span></code>; <code class="docutils literal notranslate"><span class="pre">UTF8String.translate(dict)</span></code> is code-point based, and any character mapped explicitly to U+0000 in <code class="docutils literal notranslate"><span class="pre">to</span></code> is also deleted.</p></li>
-<li><p>Spark 4.0.1 (audited 2026-05-27): routes through <code class="docutils literal notranslate"><span class="pre">CollationSupport.StringTranslate.exec</span></code>; semantics unchanged for <code class="docutils literal notranslate"><span class="pre">UTF8_BINARY</span></code>. Non-default collations not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4496).</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): routes through <code class="docutils literal notranslate"><span class="pre">CollationSupport.StringTranslate.exec</span></code>; semantics unchanged for <code class="docutils literal notranslate"><span class="pre">UTF8_BINARY</span></code>. Non-default collations not honoured by Comet (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4496">#4496</a>).</p></li>
<li><p>Marked <code class="docutils literal notranslate"><span class="pre">Incompatible</span></code>: DataFusion’s <code class="docutils literal notranslate"><span class="pre">translate</span></code> iterates over Unicode graphemes (Spark uses code points) and substitutes U+0000 instead of treating it as a deletion sentinel. It falls back to Spark by default and runs natively only when incompatible expressions are explicitly allowed.</p></li>
</ul>
</section>
@@ -837,7 +837,7 @@ under the License.
<ul class="simple">
<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">StringTrim</span></code> no-arg form strips ASCII space <code class="docutils literal notranslate"><span class="pre">0x20</span></code> only (matches DataFusion <code class="docutils literal notranslate"><span class="pre">btrim</span></code>’s default); two-arg form’s children are <code class="docutils literal notranslate"><span class="pre">(srcStr,</span> <span class="pre">tri [...]
-<li><p>Spark 4.0.1 (audited 2026-05-27): routes through <code class="docutils literal notranslate"><span class="pre">CollationSupport.StringTrim.exec</span></code> and uses <code class="docutils literal notranslate"><span class="pre">StringTypeNonCSAICollation</span></code>; semantics unchanged for <code class="docutils literal notranslate"><span class="pre">UTF8_BINARY</span></code>. Non-default collations not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4496).</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): routes through <code class="docutils literal notranslate"><span class="pre">CollationSupport.StringTrim.exec</span></code> and uses <code class="docutils literal notranslate"><span class="pre">StringTypeNonCSAICollation</span></code>; semantics unchanged for <code class="docutils literal notranslate"><span class="pre">UTF8_BINARY</span></code>. Non-default collations not honoured by Comet (<a class="reference external" href="https://github.com/apa [...]
</ul>
</section>
<section id="ucase">
@@ -853,7 +853,7 @@ under the License.
<ul class="simple">
<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. JVM default-locale <code class="docutils literal notranslate"><span class="pre">toUpperCase</span></code> on <code class="docutils literal notranslate"><span class="pre">UTF8String</span></code>. Comet routes to DataFusion <code class="docutils literal notranslate"><span class="pre">upper</span></code> (Rust Unicode default case mapping, no locale awareness) and is unconditionally <code class="docutils literal notranslate"><span class="p [...]
-<li><p>Spark 4.0.1 (audited 2026-05-27): routes through <code class="docutils literal notranslate"><span class="pre">CollationSupport.Upper.exec(v,</span> <span class="pre">collationId,</span> <span class="pre">useICU)</span></code> with <code class="docutils literal notranslate"><span class="pre">SQLConf.ICU_CASE_MAPPINGS_ENABLED</span></code>. Comet does not propagate collation or ICU mode; non-default collations or <code class="docutils literal notranslate"><span class="pre">ICU_CASE_ [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): routes through <code class="docutils literal notranslate"><span class="pre">CollationSupport.Upper.exec(v,</span> <span class="pre">collationId,</span> <span class="pre">useICU)</span></code> with <code class="docutils literal notranslate"><span class="pre">SQLConf.ICU_CASE_MAPPINGS_ENABLED</span></code>. Comet does not propagate collation or ICU mode; non-default collations or <code class="docutils literal notranslate"><span class="pre">ICU_CASE_ [...]
</ul>
</section>
</section>
diff --git a/contributor-guide/expression-audits/url_funcs.html b/contributor-guide/expression-audits/url_funcs.html
index 21037ef940..77ec111836 100644
--- a/contributor-guide/expression-audits/url_funcs.html
+++ b/contributor-guide/expression-audits/url_funcs.html
@@ -555,7 +555,7 @@ under the License.
<section id="parse-url">
<h2>parse_url<a class="headerlink" href="#parse-url" title="Link to this heading">#</a></h2>
<ul class="simple">
-<li><p>Performance (tuned 2026-07-11, PR #4893): cache the compiled query regex in a thread-local single-slot cache keyed by the query key, eliminating per-row <code class="docutils literal notranslate"><span class="pre">Regex::new()</span></code> recompilation in the QUERY-with-key path (the key is constant across a batch in practice). Up to 50x faster. Benchmark: <code class="docutils literal notranslate"><span class="pre">benches/parse_url.rs</span></code>.</p></li>
+<li><p>Performance (tuned 2026-07-11, PR <a class="reference external" href="https://github.com/apache/datafusion-comet/pull/4893">#4893</a>): cache the compiled query regex in a thread-local single-slot cache keyed by the query key, eliminating per-row <code class="docutils literal notranslate"><span class="pre">Regex::new()</span></code> recompilation in the QUERY-with-key path (the key is constant across a batch in practice). Up to 50x faster. Benchmark: <code class="docutils literal [...]
</ul>
</section>
<section id="try-url-decode">
diff --git a/contributor-guide/expression-audits/window_funcs.html b/contributor-guide/expression-audits/window_funcs.html
index 05090c0841..872441a5a9 100644
--- a/contributor-guide/expression-audits/window_funcs.html
+++ b/contributor-guide/expression-audits/window_funcs.html
@@ -555,7 +555,7 @@ under the License.
<section id="avg-window">
<h2>avg (window)<a class="headerlink" href="#avg-window" title="Link to this heading">#</a></h2>
<ul class="simple">
-<li><p>4.1.1, audited 2026-06-25: <code class="docutils literal notranslate"><span class="pre">AVG(<decimal>)</span></code> over a window falls back to Spark on Spark 4.x. Spark wraps the result as <code class="docutils literal notranslate"><span class="pre">Cast(avg(UnscaledValue(v))</span> <span class="pre">/</span> <span class="pre">pow10</span> <span class="pre">as</span> <span class="pre">decimal)</span></code>, a shape <code class="docutils literal notranslate"><span class="p [...]
+<li><p>4.1.1, audited 2026-06-25: <code class="docutils literal notranslate"><span class="pre">AVG(<decimal>)</span></code> over a window runs natively on Spark 4.x. Spark wraps the result as <code class="docutils literal notranslate"><span class="pre">Cast(avg(UnscaledValue(v))</span> <span class="pre">/</span> <span class="pre">pow10</span> <span class="pre">as</span> <span class="pre">decimal)</span></code>; <code class="docutils literal notranslate"><span class="pre">CometWindo [...]
</ul>
</section>
<section id="lag">
diff --git a/contributor-guide/roadmap.html b/contributor-guide/roadmap.html
index 348c2a352d..b28df0e4dc 100644
--- a/contributor-guide/roadmap.html
+++ b/contributor-guide/roadmap.html
@@ -578,14 +578,14 @@ Iceberg native scans support non-AQE DPP (<a class="reference external" href="ht
<h2>TPC-H and TPC-DS Performance<a class="headerlink" href="#tpc-h-and-tpc-ds-performance" title="Link to this heading">#</a></h2>
<p>We regularly publish benchmark results derived from TPC-H and TPC-DS to track performance against Spark. Closing
the remaining gaps and increasing the speedup on both benchmark suites is an ongoing focus, tracked under <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/2004">#2004</a>
-(TPC-H), <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/858">#858</a> (TPC-DS), and <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/3799">#3799</a> (improving the awslabs published TPC-DS results).</p>
+(TPC-H) and <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/858">#858</a> (TPC-DS).</p>
</section>
<section id="upstream-work-in-datafusion">
<h2>Upstream Work in DataFusion<a class="headerlink" href="#upstream-work-in-datafusion" title="Link to this heading">#</a></h2>
<p>A growing number of Spark-compatible expressions live in the <code class="docutils literal notranslate"><span class="pre">datafusion-spark</span></code> crate in the core DataFusion
repository. Comet is migrating its expression implementations to that crate so that they can be shared by other
-DataFusion-based projects, tracked in <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/2084">#2084</a>. Improvements to core DataFusion operators (joins, aggregates,
-window) made in support of Comet also benefit the wider ecosystem.</p>
+DataFusion-based projects, and is wiring up the functions that crate already provides (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4150">#4150</a>). Improvements to
+core DataFusion operators (joins, aggregates, window) made in support of Comet also benefit the wider ecosystem.</p>
</section>
<section id="spillable-hash-join">
<h2>Spillable Hash Join<a class="headerlink" href="#spillable-hash-join" title="Link to this heading">#</a></h2>
diff --git a/contributor-guide/s3-credential-provider-design.html b/contributor-guide/s3-credential-provider-design.html
index b5bb2e6017..7cec38c972 100644
--- a/contributor-guide/s3-credential-provider-design.html
+++ b/contributor-guide/s3-credential-provider-design.html
@@ -566,7 +566,7 @@ under the License.
<li><p>ServiceLoader makes activation implicit on classpath presence. A vendor JAR drifting onto a cluster could silently change S3 auth behavior. The config key makes activation explicit.</p></li>
<li><p>The activation key (<code class="docutils literal notranslate"><span class="pre">fs.s3a.comet.credential.provider.class</span></code>, with per-bucket override) follows the same shape as <code class="docutils literal notranslate"><span class="pre">fs.s3a.bucket.<name>.aws.credentials.provider</span></code>, so operators do not learn a new pattern.</p></li>
</ul>
-<p>Activation is modeled on <code class="docutils literal notranslate"><span class="pre">parquet.crypto.factory.class</span></code> (Parquet Modular Encryption KMS, see Comet #2447): the user names a single vendor class and the vendor dispatches across multiple credential backends inside that class if they need to. This mirrors how Iceberg’s <code class="docutils literal notranslate"><span class="pre">DecryptionPropertiesFactory</span></code> already behaves for Parquet keys.</p>
+<p>Activation is modeled on <code class="docutils literal notranslate"><span class="pre">parquet.crypto.factory.class</span></code> (Parquet Modular Encryption KMS, see Comet <a class="reference external" href="https://github.com/apache/datafusion-comet/pull/2447">#2447</a>): the user names a single vendor class and the vendor dispatches across multiple credential backends inside that class if they need to. This mirrors how Iceberg’s <code class="docutils literal notranslate"><span class [...]
</section>
<section id="why-fqcn-dispatchkey-catalogproperties-keying">
<h2>Why <code class="docutils literal notranslate"><span class="pre">(FQCN,</span> <span class="pre">dispatchKey,</span> <span class="pre">catalogProperties)</span></code> keying<a class="headerlink" href="#why-fqcn-dispatchkey-catalogproperties-keying" title="Link to this heading">#</a></h2>
diff --git a/searchindex.js b/searchindex.js
index 2f93b3b006..a7211f6d28 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[54, "id1"]], "%": [[52, "id1"]], "&": [[42, "id1"]], "*": [[52, "id2"]], "+": [[52, "id3"]], "-": [[52, "id4"]], "/": [[52, "id5"]], "1. Format Your Code": [[39, "format-your-code"]], "1. Install Comet": [[60, "install-comet"], [70, "install-comet"]], "1. Native Operators (nativeExecs map)": [[27, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[39, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[60, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[54, "id1"]], "%": [[52, "id1"]], "&": [[42, "id1"]], "*": [[52, "id2"]], "+": [[52, "id3"]], "-": [[52, "id4"]], "/": [[52, "id5"]], "1. Format Your Code": [[39, "format-your-code"]], "1. Install Comet": [[60, "install-comet"], [70, "install-comet"]], "1. Native Operators (nativeExecs map)": [[27, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[39, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[60, "clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/compatibility/scans.html b/user-guide/latest/compatibility/scans.html
index bb59615c8e..f4e04f3eef 100644
--- a/user-guide/latest/compatibility/scans.html
+++ b/user-guide/latest/compatibility/scans.html
@@ -793,7 +793,7 @@ instants that cannot be safely reinterpreted as timezone-free values, and Comet
rejecting the read. This applies to all LTZ physical encodings (INT96, TIMESTAMP_MICROS,
TIMESTAMP_MILLIS). On Spark 4.0+, this read is permitted
(<a class="reference external" href="https://issues.apache.org/jira/browse/SPARK-47447">SPARK-47447</a>) and Comet matches Spark’s
-behavior. See <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4219">#4219</a>.</p></li>
+behavior.</p></li>
</ul>
<section id="schema-mismatch-handling">
<h3>Schema Mismatch Handling<a class="headerlink" href="#schema-mismatch-handling" title="Link to this heading">#</a></h3>
diff --git a/user-guide/latest/datasources.html b/user-guide/latest/datasources.html
index 99b0530659..1b0ca4d839 100644
--- a/user-guide/latest/datasources.html
+++ b/user-guide/latest/datasources.html
@@ -1004,7 +1004,7 @@ installed.</p>
<p>Comet’s S3 support has the following limitations:</p>
<ol class="arabic simple">
<li><p><strong>Partial Hadoop S3A configuration support</strong>: Not all Hadoop S3A configurations are currently supported. Only the configurations listed in the tables above are translated and applied to the underlying <code class="docutils literal notranslate"><span class="pre">object_store</span></code> crate.</p></li>
-<li><p><strong>Custom credential providers</strong>: Custom implementations of AWS credential providers are not supported. The implementation only supports the standard credential providers listed in the table above. We are planning to add support for custom credential providers through a JNI-based adapter that will allow calling Java credential providers from Rust code. See <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/1829">issue #1829</a> for mo [...]
+<li><p><strong>Custom credential providers</strong>: Custom implementations of AWS credential providers are not supported. The implementation only supports the standard credential providers listed in the table above. We are planning to add support for custom credential providers through a JNI-based adapter that will allow calling Java credential providers from Rust code. See <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/1829">#1829</a> for more det [...]
</ol>
</section>
</section>
diff --git a/user-guide/latest/datatypes.html b/user-guide/latest/datatypes.html
index adcbb33b87..3259048dc2 100644
--- a/user-guide/latest/datatypes.html
+++ b/user-guide/latest/datatypes.html
@@ -899,8 +899,12 @@ the tables below and may be reconsidered based on demand:</p>
</section>
<section id="interval">
<h2>Interval<a class="headerlink" href="#interval" title="Link to this heading">#</a></h2>
-<p>Interval types fall back to Spark today. Native acceleration is tracked by
-<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4540">#4540</a>.</p>
+<p>All three interval types are mapped to Arrow and flow through serde, native shuffle, and the
+codegen dispatcher, so interval columns and the interval-producing expressions run natively.
+Several operators still gate on the type and fall back: Parquet scans of ANSI interval columns,
+single-column sorts, hash aggregates (<code class="docutils literal notranslate"><span class="pre">min</span></code> / <code class="docutils literal notranslate"><span class="pre">max</span></code> / <code class="docutils literal notranslate"><span class="pre">sum</span></code> / <code class="docutils literal notranslate"><span class="pre">avg</span></code>), <code class="docutils literal notranslate"><span class="pre">GROUP</span> <span class="pre">BY</span></code>, window
+functions, and hashing a <code class="docutils literal notranslate"><span class="pre">CalendarInterval</span></code>. Remaining work is tracked by
+<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/5061">#5061</a>.</p>
<div class="pst-scrollable-table-container"><table class="table">
<thead>
<tr class="row-odd"><th class="head"><p>Type</p></th>
@@ -910,16 +914,16 @@ the tables below and may be reconsidered based on demand:</p>
</thead>
<tbody>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">YearMonthIntervalType</span></code></p></td>
-<td><p>🔜</p></td>
-<td><p>Tracked by #4540.</p></td>
+<td><p>⚠️</p></td>
+<td><p>Parquet scan, single-column sort, aggregate, <code class="docutils literal notranslate"><span class="pre">GROUP</span> <span class="pre">BY</span></code>, and window operators fall back.</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">DayTimeIntervalType</span></code></p></td>
-<td><p>🔜</p></td>
-<td><p>Tracked by #4540.</p></td>
+<td><p>⚠️</p></td>
+<td><p>Parquet scan, single-column sort, aggregate, <code class="docutils literal notranslate"><span class="pre">GROUP</span> <span class="pre">BY</span></code>, and window operators fall back.</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">CalendarIntervalType</span></code></p></td>
-<td><p>🔜</p></td>
-<td><p>Tracked by #4540.</p></td>
+<td><p>⚠️</p></td>
+<td><p>As above, plus <code class="docutils literal notranslate"><span class="pre">hash</span></code> / <code class="docutils literal notranslate"><span class="pre">xxhash64</span></code> of a <code class="docutils literal notranslate"><span class="pre">CalendarInterval</span></code> fails rather than falling back.</p></td>
</tr>
</tbody>
</table>
diff --git a/user-guide/latest/expressions.html b/user-guide/latest/expressions.html
index 8705becb5a..f88c025204 100644
--- a/user-guide/latest/expressions.html
+++ b/user-guide/latest/expressions.html
@@ -948,7 +948,7 @@ expressions. The following function families are <strong>not currently planned</
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">kurtosis</span></code></p></td>
<td><p>🔜</p></td>
<td><p>—</p></td>
-<td><p>tracking <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4098">#4098</a></p></td>
+<td><p>Not yet implemented natively</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">last</span></code></p></td>
<td><p>✅</p></td>
@@ -1018,17 +1018,17 @@ expressions. The following function families are <strong>not currently planned</
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">regr_avgx</span></code></p></td>
<td><p>✅</p></td>
<td><p>—</p></td>
-<td><p>Native: Spark rewrites to <code class="docutils literal notranslate"><span class="pre">Average</span></code> (tests in <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4551">#4551</a>)</p></td>
+<td><p>Native: Spark rewrites to <code class="docutils literal notranslate"><span class="pre">Average</span></code> (tests in <a class="reference external" href="https://github.com/apache/datafusion-comet/pull/4551">#4551</a>)</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">regr_avgy</span></code></p></td>
<td><p>✅</p></td>
<td><p>—</p></td>
-<td><p>Native: Spark rewrites to <code class="docutils literal notranslate"><span class="pre">Average</span></code> (tests in <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4551">#4551</a>)</p></td>
+<td><p>Native: Spark rewrites to <code class="docutils literal notranslate"><span class="pre">Average</span></code> (tests in <a class="reference external" href="https://github.com/apache/datafusion-comet/pull/4551">#4551</a>)</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">regr_count</span></code></p></td>
<td><p>✅</p></td>
<td><p>—</p></td>
-<td><p>Native: Spark rewrites to <code class="docutils literal notranslate"><span class="pre">Count</span></code> (tests in <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4551">#4551</a>)</p></td>
+<td><p>Native: Spark rewrites to <code class="docutils literal notranslate"><span class="pre">Count</span></code> (tests in <a class="reference external" href="https://github.com/apache/datafusion-comet/pull/4551">#4551</a>)</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">regr_intercept</span></code></p></td>
<td><p>🔜</p></td>
@@ -1063,7 +1063,7 @@ expressions. The following function families are <strong>not currently planned</
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">skewness</span></code></p></td>
<td><p>🔜</p></td>
<td><p>—</p></td>
-<td><p>tracking <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4098">#4098</a></p></td>
+<td><p>Not yet implemented natively</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">some</span></code></p></td>
<td><p>✅</p></td>
@@ -1259,7 +1259,7 @@ expressions. The following function families are <strong>not currently planned</
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">slice</span></code></p></td>
<td><p>✅</p></td>
<td><p>Native</p></td>
-<td><p>Native (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4149">#4149</a>)</p></td>
+<td><p>Native (<a class="reference external" href="https://github.com/apache/datafusion-comet/pull/4149">#4149</a>)</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">sort_array</span></code></p></td>
<td><p>✅</p></td>
@@ -1671,7 +1671,7 @@ expressions. The following function families are <strong>not currently planned</
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">make_interval</span></code></p></td>
<td><p>🔜</p></td>
<td><p>—</p></td>
-<td><p>Produces legacy CalendarInterval; tracked by <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4540">#4540</a></p></td>
+<td><p>Produces legacy CalendarInterval; tracked by <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/5061">#5061</a></p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">make_time</span></code></p></td>
<td><p>🔜</p></td>
@@ -1811,7 +1811,7 @@ expressions. The following function families are <strong>not currently planned</
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">try_make_interval</span></code></p></td>
<td><p>🔜</p></td>
<td><p>—</p></td>
-<td><p>Produces legacy CalendarInterval; tracked by <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4540">#4540</a></p></td>
+<td><p>Produces legacy CalendarInterval; tracked by <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/5061">#5061</a></p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">try_make_timestamp</span></code></p></td>
<td><p>✅</p></td>
@@ -2611,7 +2611,7 @@ expression-level). The <code class="docutils literal notranslate"><span class="p
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">is_variant_null</span></code></p></td>
<td><p>🔜</p></td>
<td><p>—</p></td>
-<td><p>tracking <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4098">#4098</a></p></td>
+<td><p>Requires <code class="docutils literal notranslate"><span class="pre">VariantType</span></code> support</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">monotonically_increasing_id</span></code></p></td>
<td><p>✅</p></td>
@@ -2621,7 +2621,7 @@ expression-level). The <code class="docutils literal notranslate"><span class="p
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">parse_json</span></code></p></td>
<td><p>🔜</p></td>
<td><p>—</p></td>
-<td><p>tracking <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4098">#4098</a></p></td>
+<td><p>Requires <code class="docutils literal notranslate"><span class="pre">VariantType</span></code> support</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">raise_error</span></code></p></td>
<td><p>🔜</p></td>
@@ -2641,12 +2641,12 @@ expression-level). The <code class="docutils literal notranslate"><span class="p
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">schema_of_variant</span></code></p></td>
<td><p>🔜</p></td>
<td><p>—</p></td>
-<td><p>tracking <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4098">#4098</a></p></td>
+<td><p>Requires <code class="docutils literal notranslate"><span class="pre">VariantType</span></code> support</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">schema_of_variant_agg</span></code></p></td>
<td><p>🔜</p></td>
<td><p>—</p></td>
-<td><p>tracking <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4098">#4098</a></p></td>
+<td><p>Requires <code class="docutils literal notranslate"><span class="pre">VariantType</span></code> support</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">session_user</span></code></p></td>
<td><p>✅</p></td>
@@ -2661,7 +2661,7 @@ expression-level). The <code class="docutils literal notranslate"><span class="p
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">to_variant_object</span></code></p></td>
<td><p>🔜</p></td>
<td><p>—</p></td>
-<td><p>tracking <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4098">#4098</a></p></td>
+<td><p>Requires <code class="docutils literal notranslate"><span class="pre">VariantType</span></code> support</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">try_aes_decrypt</span></code></p></td>
<td><p>✅</p></td>
@@ -2671,12 +2671,12 @@ expression-level). The <code class="docutils literal notranslate"><span class="p
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">try_parse_json</span></code></p></td>
<td><p>🔜</p></td>
<td><p>—</p></td>
-<td><p>tracking <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4098">#4098</a></p></td>
+<td><p>Requires <code class="docutils literal notranslate"><span class="pre">VariantType</span></code> support</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">try_variant_get</span></code></p></td>
<td><p>🔜</p></td>
<td><p>—</p></td>
-<td><p>tracking <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4098">#4098</a></p></td>
+<td><p>Requires <code class="docutils literal notranslate"><span class="pre">VariantType</span></code> support</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">typeof</span></code></p></td>
<td><p>✅</p></td>
@@ -2696,7 +2696,7 @@ expression-level). The <code class="docutils literal notranslate"><span class="p
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">variant_get</span></code></p></td>
<td><p>🔜</p></td>
<td><p>—</p></td>
-<td><p>tracking <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4098">#4098</a></p></td>
+<td><p>Requires <code class="docutils literal notranslate"><span class="pre">VariantType</span></code> support</p></td>
</tr>
</tbody>
</table>
@@ -3133,7 +3133,7 @@ expression-level). The <code class="docutils literal notranslate"><span class="p
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">translate</span></code></p></td>
<td><p>✅</p></td>
<td><p>Native</p></td>
-<td><p>Falls back by default; opt-in via allowIncompatible (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4463">#4463</a>)</p></td>
+<td><p>DataFusion’s <code class="docutils literal notranslate"><span class="pre">translate</span></code> iterates over Unicode graphemes (Spark uses code points) and substitutes U+0000 instead of treating it as a deletion sentinel, so the native path is opt-in via allowIncompatible</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">trim</span></code></p></td>
<td><p>✅</p></td>
diff --git a/user-guide/latest/operators.html b/user-guide/latest/operators.html
index db3609bcd3..b3c9a80782 100644
--- a/user-guide/latest/operators.html
+++ b/user-guide/latest/operators.html
@@ -779,7 +779,7 @@ omitted from the tables below and may be reconsidered based on demand:</p>
<li><p><strong>Structured Streaming operators</strong> (<code class="docutils literal notranslate"><span class="pre">StateStoreSaveExec</span></code>, <code class="docutils literal notranslate"><span class="pre">StateStoreRestoreExec</span></code>, <code class="docutils literal notranslate"><span class="pre">StreamingSymmetricHashJoinExec</span></code>, and similar): Comet targets batch execution.</p></li>
<li><p><strong>Cartesian / cross joins</strong> (<code class="docutils literal notranslate"><span class="pre">CartesianProductExec</span></code>): rare and expensive, with little acceleration benefit.</p></li>
<li><p><strong>Sampling and range generation</strong> (<code class="docutils literal notranslate"><span class="pre">SampleExec</span></code>, <code class="docutils literal notranslate"><span class="pre">RangeExec</span></code>): niche leaf operators.</p></li>
-<li><p><strong>Pickled (non-Arrow) Python UDFs</strong> (<code class="docutils literal notranslate"><span class="pre">BatchEvalPythonExec</span></code>): Comet accelerates Arrow-based Python UDFs only (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4234">#4234</a>).</p></li>
+<li><p><strong>Pickled (non-Arrow) Python UDFs</strong> (<code class="docutils literal notranslate"><span class="pre">BatchEvalPythonExec</span></code>): Comet accelerates Arrow-based Python UDFs only (<a class="reference external" href="https://github.com/apache/datafusion-comet/pull/4234">#4234</a>).</p></li>
</ul>
</section>
<section id="scans">
@@ -802,7 +802,7 @@ omitted from the tables below and may be reconsidered based on demand:</p>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">LocalTableScanExec</span></code></p></td>
<td><p>⚠️</p></td>
-<td><p>Disabled by default; there is no acceleration advantage and this operator is typically only used in test code. Can be opted into via config (#4393).</p></td>
+<td><p>Disabled by default; there is no acceleration advantage and this operator is typically only used in test code. Can be opted into via config (<a class="reference external" href="https://github.com/apache/datafusion-comet/pull/4393">#4393</a>).</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">InMemoryTableScanExec</span></code></p></td>
<td><p>🔜</p></td>
@@ -918,7 +918,7 @@ omitted from the tables below and may be reconsidered based on demand:</p>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">BroadcastNestedLoopJoinExec</span></code></p></td>
<td><p>✅</p></td>
-<td><p>Falls back to Spark when the preserved side is broadcast (for example LEFT OUTER with BROADCAST on the left) (#4429).</p></td>
+<td><p>Falls back to Spark when the preserved side is broadcast (for example LEFT OUTER with BROADCAST on the left) (<a class="reference external" href="https://github.com/apache/datafusion-comet/pull/4429">#4429</a>).</p></td>
</tr>
</tbody>
</table>
@@ -1028,7 +1028,7 @@ omitted from the tables below and may be reconsidered based on demand:</p>
<tbody>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">ArrowEvalPythonExec</span></code>, <code class="docutils literal notranslate"><span class="pre">MapInArrowExec</span></code>, <code class="docutils literal notranslate"><span class="pre">MapInPandasExec</span></code>, <code class="docutils literal notranslate"><span class="pre">FlatMapGroupsInPandasExec</span></code></p></td>
<td><p>🔜</p></td>
-<td><p>Experimental accelerated PyArrow UDF support is in progress (#4234).</p></td>
+<td><p>Experimental accelerated PyArrow UDF support is in progress (<a class="reference external" href="https://github.com/apache/datafusion-comet/pull/4234">#4234</a>).</p></td>
</tr>
</tbody>
</table>
diff --git a/user-guide/latest/pyarrow-udfs.html b/user-guide/latest/pyarrow-udfs.html
index e2ae2e0c47..f7a7bf7812 100644
--- a/user-guide/latest/pyarrow-udfs.html
+++ b/user-guide/latest/pyarrow-udfs.html
@@ -768,9 +768,10 @@ operators in the physical plan and replaces them with <code class="docutils lite
<li><p>Keeps the Python output in columnar format for downstream operators</p></li>
</ul>
<p>This eliminates the ColumnarToRow transition and the output row conversion, reducing CPU overhead
-and memory allocations. The internal row-to-Arrow IPC re-encoding inside Spark’s
-<code class="docutils literal notranslate"><span class="pre">ArrowPythonRunner</span></code> is unchanged in this version; full round-trip elimination is tracked in
-<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4240">#4240</a>.</p>
+and memory allocations. The row-to-Arrow re-encoding that Spark’s <code class="docutils literal notranslate"><span class="pre">ArrowPythonRunner</span></code> performed on
+the input side is also gone: <code class="docutils literal notranslate"><span class="pre">CometArrowPythonRunner</span></code> consumes <code class="docutils literal notranslate"><span class="pre">ColumnarBatch</span></code> directly, so batches
+are written straight from Comet’s vectors into the IPC root. See <a class="reference internal" href="#limitations">Limitations</a> for the
+copies that remain.</p>
<section id="plan-flow">
<h3>Plan flow<a class="headerlink" href="#plan-flow" title="Link to this heading">#</a></h3>
<p>Without Comet’s optimization:</p>
diff --git a/user-guide/latest/tuning.html b/user-guide/latest/tuning.html
index 502c788c88..d3f8156c47 100644
--- a/user-guide/latest/tuning.html
+++ b/user-guide/latest/tuning.html
@@ -907,7 +907,7 @@ is left at the 128 MB default, most splits will contain at most one row group bo
mismatch is amplified; setting <code class="docutils literal notranslate"><span class="pre">spark.sql.files.maxPartitionBytes</span></code> below 120 MB (for example, 64 MB)
distributes row groups across more splits and reduces the number of idle tasks. Smaller values produce
more splits overall, so some idle tasks may remain — tune the value against your file layout.</p>
-<p>See <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/3817#issuecomment-4193279630">issue #3817</a> for a
+<p>See <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/3817#issuecomment-4193279630">#3817</a> for a
worked example and further discussion.</p>
</section>
</section>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org