You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/06/02 20:24:00 UTC
(datafusion-comet) branch asf-site updated: Publish built docs triggered by 13f5d7bbc86cea121de72ee4689edfd19380b026
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new cbc71e3717 Publish built docs triggered by 13f5d7bbc86cea121de72ee4689edfd19380b026
cbc71e3717 is described below
commit cbc71e3717ca8aca4b6410c062007185f628f6b1
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Tue Jun 2 20:23:55 2026 +0000
Publish built docs triggered by 13f5d7bbc86cea121de72ee4689edfd19380b026
---
.../adding_a_new_expression.md.txt | 5 +-
.../expression-audits/agg_funcs.md.txt | 42 +
.../expression-audits/array_funcs.md.txt | 177 ++
.../expression-audits/bitwise_funcs.md.txt | 101 +
.../expression-audits/collection_funcs.md.txt | 52 +
.../expression-audits/conditional_funcs.md.txt | 73 +
.../expression-audits/conversion_funcs.md.txt | 39 +
.../expression-audits/datetime_funcs.md.txt | 90 +
.../expression-audits/generator_funcs.md.txt | 40 +
.../expression-audits/hash_funcs.md.txt | 74 +
.../expression-audits/index.md.txt | 46 +
.../expression-audits/json_funcs.md.txt | 40 +
.../expression-audits/map_funcs.md.txt | 81 +
.../expression-audits/math_funcs.md.txt | 274 +++
.../expression-audits/misc_funcs.md.txt | 85 +
.../expression-audits/predicate_funcs.md.txt | 151 ++
.../expression-audits/struct_funcs.md.txt | 39 +
.../expression-audits/url_funcs.md.txt | 40 +
.../expression-audits/window_funcs.md.txt | 32 +
_sources/contributor-guide/index.md.txt | 2 +-
_sources/contributor-guide/release_process.md.txt | 5 +-
.../spark_expressions_support.md.txt | 1091 ----------
_sources/user-guide/latest/expressions.md.txt | 502 ++---
contributor-guide/adding_a_new_expression.html | 26 +-
contributor-guide/adding_a_new_operator.html | 21 +-
contributor-guide/adding_a_new_spark_version.html | 21 +-
contributor-guide/benchmark-results/tpc-ds.html | 21 +-
contributor-guide/benchmark-results/tpc-h.html | 21 +-
contributor-guide/benchmarking.html | 21 +-
contributor-guide/benchmarking_aws_ec2.html | 21 +-
contributor-guide/benchmarking_macos.html | 21 +-
contributor-guide/benchmarking_spark_sql_perf.html | 21 +-
contributor-guide/bug_triage.html | 21 +-
contributor-guide/contributing.html | 21 +-
contributor-guide/debugging.html | 21 +-
contributor-guide/development.html | 21 +-
.../agg_funcs.html} | 155 +-
.../expression-audits/array_funcs.html | 811 +++++++
.../bitwise_funcs.html} | 230 +-
.../collection_funcs.html} | 167 +-
.../conditional_funcs.html} | 194 +-
.../conversion_funcs.html} | 148 +-
.../datetime_funcs.html} | 227 +-
.../generator_funcs.html} | 155 +-
.../hash_funcs.html} | 190 +-
.../tpc-h.html => expression-audits/index.html} | 156 +-
.../json_funcs.html} | 153 +-
.../map_funcs.html} | 199 +-
.../expression-audits/math_funcs.html | 988 +++++++++
.../misc_funcs.html} | 214 +-
.../expression-audits/predicate_funcs.html | 781 +++++++
.../struct_funcs.html} | 145 +-
.../url_funcs.html} | 148 +-
.../window_funcs.html} | 138 +-
contributor-guide/ffi.html | 21 +-
contributor-guide/iceberg-spark-tests.html | 21 +-
contributor-guide/index.html | 21 +-
contributor-guide/jvm_shuffle.html | 21 +-
contributor-guide/native_shuffle.html | 21 +-
contributor-guide/plugin_overview.html | 21 +-
contributor-guide/profiling.html | 21 +-
contributor-guide/release_process.html | 26 +-
contributor-guide/roadmap.html | 21 +-
.../s3-credential-provider-design.html | 21 +-
contributor-guide/spark-sql-tests.html | 21 +-
contributor-guide/spark_configs_support.html | 27 +-
contributor-guide/spark_expressions_support.html | 2259 --------------------
contributor-guide/sql-file-tests.html | 21 +-
contributor-guide/sql_error_propagation.html | 21 +-
contributor-guide/tracing.html | 27 +-
genindex.html | 21 +-
index.html | 21 +-
objects.inv | Bin 2468 -> 2718 bytes
search.html | 21 +-
searchindex.js | 2 +-
user-guide/latest/expressions.html | 182 +-
76 files changed, 6531 insertions(+), 4865 deletions(-)
diff --git a/_sources/contributor-guide/adding_a_new_expression.md.txt b/_sources/contributor-guide/adding_a_new_expression.md.txt
index 81752b1a9a..f08226380d 100644
--- a/_sources/contributor-guide/adding_a_new_expression.md.txt
+++ b/_sources/contributor-guide/adding_a_new_expression.md.txt
@@ -25,10 +25,9 @@ Before you start, have a look through [these slides](https://docs.google.com/pre
## Finding an Expression to Add
-You may have a specific expression in mind that you'd like to add, but if not, you can review the [expression coverage document](spark_expressions_support.md) to see which expressions are not yet supported.
+You may have a specific expression in mind that you'd like to add, but if not, you can review [Supported Spark Expressions](../user-guide/latest/expressions.md) in the user guide to see which expressions are not yet supported. For deep-dive audit notes on expressions that are already supported, see the [Expression Audits](expression-audits/index.md) section.
-When you add or change an expression, update **both** the coverage checklist
-(`spark_expressions_support.md`) and the user-facing status in
+When you add or change an expression, update its status in
[Supported Spark Expressions](../user-guide/latest/expressions.md).
## Implementing the Expression
diff --git a/_sources/contributor-guide/expression-audits/agg_funcs.md.txt b/_sources/contributor-guide/expression-audits/agg_funcs.md.txt
new file mode 100644
index 0000000000..8025f31b73
--- /dev/null
+++ b/_sources/contributor-guide/expression-audits/agg_funcs.md.txt
@@ -0,0 +1,42 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# agg_funcs Expression Audits
+
+> Audit notes for expressions in this category that have been audited. Absence of an entry means the expression has not been audited yet, not that it is unsupported. See the user guide [Spark Expression Support] for current support status.
+
+## any
+
+- Spark 3.4.3 (audited 2026-05-26): registered as a SQL alias of `BoolOr`, which extends `RuntimeReplaceableAggregate` with `replacement = Max(child)`. Catalyst rewrites `any(x)` to `max(x)` before Comet sees the plan, so `any` is served by `CometMax` on a `BooleanType` column.
+- Spark 3.5.8 (audited 2026-05-26): identical to 3.4.3.
+- Spark 4.0.1 (audited 2026-05-26): identical to 3.4.3.
+
+## avg
+
+- Spark 3.4.3 (2026-05-26)
+- Spark 3.5.8 (2026-05-26): aggregate logic identical to 3.4.3
+- Spark 4.0.1 (2026-05-26): aggregate logic identical to 3.5.8; only `QueryContext` import path differs. `YearMonthIntervalType` and `DayTimeIntervalType` inputs (supported by Spark) fall back to Spark in Comet.
+
+## bit_and
+
+- Spark 3.4.3 (2026-05-26)
+- Spark 3.5.8 (2026-05-26)
+- Spark 4.0.1 (2026-05-26)
+
+[Spark Expression Support]: ../../user-guide/latest/expressions.md
diff --git a/_sources/contributor-guide/expression-audits/array_funcs.md.txt b/_sources/contributor-guide/expression-audits/array_funcs.md.txt
new file mode 100644
index 0000000000..280aa2215e
--- /dev/null
+++ b/_sources/contributor-guide/expression-audits/array_funcs.md.txt
@@ -0,0 +1,177 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# array_funcs Expression Audits
+
+> Audit notes for expressions in this category that have been audited. Absence of an entry means the expression has not been audited yet, not that it is unsupported. See the user guide [Spark Expression Support] for current support status.
+
+## array
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `CreateArray(children, useStringTypeWhenEmpty)`; element type is the common type of children. Comet routes via `CometCreateArray` (native `make_array`) and special-cases the empty-array case to dodge a known DataFusion `coerce_types` issue (#3338).
+- Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
+- Spark 4.1.1 (audited 2026-05-27): adds `contextIndependentFoldable` override; runtime semantics unchanged.
+
+## array_append
+
+- Spark 3.4.3 (audited 2026-05-27): standalone `BinaryExpression`, evaluated directly. Comet routes via `CometArrayAppend`.
+- Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.
+- Spark 4.0.1 (audited 2026-05-27): now `RuntimeReplaceable` and rewritten to `ArrayInsert(arr, Literal(-1), elem)`. `CometArrayAppend` is therefore unreachable; dispatch goes through `CometArrayInsert` (which carries its own `Incompatible` notes documented at the `array_insert` entry).
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## array_compact
+
+- Spark 3.4.3 (audited 2026-05-27): `RuntimeReplaceable` -> `ArrayFilter(arr, IsNotNull(lambda))`. Comet receives the rewritten form, dispatches through `CometArrayFilter`, which delegates back to `CometArrayCompact.convert` for the actual proto emission. The native path uses Comet's `spark_array_compact` UDF rather than DataFusion's `array_remove_all` because DataFusion 53 changed `array_remove_all`'s NULL semantics.
+- Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.
+- Spark 4.0.1 (audited 2026-05-27): the replacement is wrapped in `KnownNotContainsNull(...)` (analysis-only hint, no semantic change).
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## array_contains
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `ArrayContains(left, right) extends BinaryExpression with NullIntolerant with Predicate`; `inputTypes` uses `findWiderTypeWithoutStringPromotionForTwo`. Wired as `CometScalarFunction("array_contains")`.
+- Spark 4.0.1 (audited 2026-05-27): `NullIntolerant` trait replaced by `nullIntolerant: Boolean`; `checkInputDataTypes` adopts `DataTypeUtils.sameType` (collation-aware in 4.x).
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+- Known limitation: no NaN-canonicalization guard in `getSupportLevel`. For `Float`/`Double` arrays containing NaN, Spark's `SQLOrderingUtil` may produce different results than DataFusion's IEEE comparison (https://github.com/apache/datafusion-comet/issues/4481).
+
+## array_distinct
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `ArrayDistinct(child)` over `ArraySetLike`; uses `SQLOpenHashSet` so NaN and `+0.0`/`-0.0` are canonicalized. Wired as `CometScalarFunction("array_distinct")`.
+- Spark 4.0.1 (audited 2026-05-27): `NullIntolerant` -> `nullIntolerant` field refactor.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+- Known divergence: DataFusion `array_distinct` uses hash-based equality without NaN/signed-zero canonicalization, so float/double arrays may produce different results (https://github.com/apache/datafusion-comet/issues/4481).
+
+## array_except
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `ArrayExcept(left, right) extends ArrayBinaryLike with ComplexTypeMergingExpression`; result preserves left-side first occurrences not present in right. Comet routes via `CometArrayExcept` and unconditionally flags `Incompatible` ("Null handling and ordering may differ from Spark"); also falls back for `BinaryType` / `StructType` element types.
+- Spark 4.0.1 (audited 2026-05-27): `nullIntolerant = true` moves into `ArrayBinaryLike`; the overflow path uses `arrayFunctionWithElementsExceedLimitError`.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+- Known divergence: same NaN/signed-zero canonicalization gap as `array_distinct` for float/double arrays (https://github.com/apache/datafusion-comet/issues/4481).
+
+## array_insert
+
+- Spark 3.4.3 audited 2026-04-02
+- Spark 3.5.8 audited 2026-04-02
+- Spark 4.0.1 audited 2026-04-02 (pos=0 error message differs from Spark)
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## array_intersect
+
+- Spark 3.4.3 audited 2026-04-24 (result element order may differ from Spark when the right array is longer than the left; DataFusion probes the longer side)
+- Spark 3.5.8 audited 2026-04-24 (same ordering incompatibility as 3.4.3)
+- Spark 4.0.1 audited 2026-04-24 (ordering incompatibility as above; collated strings now fall back to Spark)
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## array_join
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `ArrayJoin(array, delimiter, nullReplacement)`. Comet routes via `CometArrayJoin` to DataFusion's `array_to_string` and is unconditionally flagged `Incompatible` ("Null handling may differ from Spark", #3178).
+- Spark 4.0.1 (audited 2026-05-27): `inputTypes` widened to `AbstractArrayType(StringTypeWithCollation(supportsTrimCollation = true))`; non-binary collations not propagated (https://github.com/apache/datafusion-comet/issues/2190).
+- Spark 4.1.1 (audited 2026-05-27): adds `contextIndependentFoldable` override; runtime unchanged.
+
+## array_max
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `ArrayMax(child) extends UnaryExpression with ImplicitCastInputTypes`; skips NULL elements; for float/double Spark's `SQLOrderingUtil` treats NaN as greater than any non-NaN. Wired as `CometScalarFunction("array_max")`.
+- Spark 4.0.1 (audited 2026-05-27): `NullIntolerant` -> `nullIntolerant` field refactor.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+- Known divergence: DataFusion's `array_max` uses Arrow `partial_cmp`-based ordering, so float/double arrays containing NaN may produce different results (https://github.com/apache/datafusion-comet/issues/4482).
+
+## array_min
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): mirror of `ArrayMax` with `evalInternal` returning the minimum. Same NULL-skip and NaN-ordering semantics. Wired as `CometScalarFunction("array_min")`.
+- Spark 4.0.1 (audited 2026-05-27): same trait refactor as `array_max`.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+- Known divergence: same NaN-handling gap as `array_max` (https://github.com/apache/datafusion-comet/issues/4482).
+
+## array_position
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `ArrayPosition(left, right)`; returns 1-based `LongType` position, 0 if not found, NULL if either input is NULL. `CometArrayPosition` falls back for all-foldable args (constant folding handles those) and for unsupported element types (binary/struct/map/null).
+- Spark 4.0.1 (audited 2026-05-27): `NullIntolerant` -> `nullIntolerant` field refactor.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## array_remove
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `ArrayRemove(left, right)`; removes all occurrences equal to `right`. Wired as `CometScalarFunction("array_remove")`. Falls back via `ArraysBase.isTypeSupported` for binary/struct/map/null child types.
+- Spark 4.0.1 (audited 2026-05-27): `NullIntolerant` -> `nullIntolerant` field refactor.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## array_repeat
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `ArrayRepeat(left, right) extends BinaryExpression with ExpectsInputTypes`; `inputTypes = Seq(AnyDataType, IntegerType)`. NULL count yields NULL; count <= 0 yields empty array; count > `MAX_ROUNDED_ARRAY_LENGTH` throws at runtime. Comet wraps the call in `CaseWhen(IsNotNull(right), array_repeat(...), null)`.
+- Spark 4.0.1 (audited 2026-05-27): error message uses `createArrayWithElementsExceedLimitError(prettyName, count)`; semantics unchanged.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## array_union
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `ArrayUnion(left, right) extends ArrayBinaryLike with ComplexTypeMergingExpression`; result is left-side distinct elements followed by new right-side elements. Wired as `CometScalarFunction("array_union")`.
+- Spark 4.0.1 (audited 2026-05-27): `nullIntolerant = true` moves into `ArrayBinaryLike`; overflow path uses `arrayFunctionWithElementsExceedLimitError`.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+- Known divergence: same NaN/signed-zero canonicalization gap as `array_distinct` (https://github.com/apache/datafusion-comet/issues/4481). Result ordering versus DataFusion is also unverified; compare the `array_intersect` ordering caveat.
+
+## arrays_overlap
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `ArraysOverlap(left, right)`; three-valued logic (TRUE if any common non-null element, NULL if a null is present and no overlap is found in non-nulls, FALSE otherwise). Comet routes via `CometArraysOverlap` to the native `spark_arrays_overlap` UDF, which implements the same three-valued logic.
+- Spark 4.0.1 (audited 2026-05-27): `NullIntolerant` -> `nullIntolerant` field refactor.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## arrays_zip
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `ArraysZip(children, names)`; returns an array of structs, padding shorter inputs with NULL. Comet routes via `CometArraysZip` and rejects unsupported child element types (anything outside primitives, decimals, dates/timestamps, strings, binary, and nested arrays/structs of those).
+- Spark 4.0.1 (audited 2026-05-27): the length-mismatch error switches from `IllegalArgumentException` to `SparkIllegalArgumentException("_LEGACY_ERROR_TEMP_3235")`; runtime unchanged.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## element_at
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `ElementAt(left, right, defaultValueOutOfBound, failOnError)`; group label `map_funcs`. Comet supports only `ArrayType` input; `MapType` input falls back.
+- Spark 4.0.1 (audited 2026-05-27): `NullIntolerant` -> `nullIntolerant` field refactor; group label changes to `collection_funcs`; ANSI default flips to `true` so out-of-bound throws by default. Comet wires `failOnError` through to native `ListExtract`.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## flatten
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `Flatten(child) extends UnaryExpression`; returns NULL if any inner sub-array is NULL. Comet routes via `CometFlatten` and falls back for child types containing `BinaryType` / `StructType` / `MapType` (limitation of `ArraysBase.isTypeSupported`).
+- Spark 4.0.1 (audited 2026-05-27): `NullIntolerant` -> `nullIntolerant` field refactor.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## get
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `GetArrayItem(child, ordinal, failOnError)`; `inputTypes = Seq(AnyDataType, IntegralType)`. Comet routes via `CometGetArrayItem`, wiring `failOnError` through to the proto.
+- Spark 4.0.1 (audited 2026-05-27): semantics unchanged; ANSI default flips to `true`.
+- Spark 4.1.1 (audited 2026-05-27): `inputTypes` tightened to `Seq(ArrayType, IntegralType)` (analysis-time only); runtime unchanged.
+
+## sort_array
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `SortArray(base, ascendingOrder) extends BinaryExpression with ArraySortLike`; the second arg must be a `Literal(_: Boolean, BooleanType)`. Comet `CometSortArray` flags `Incompatible` under strict floating-point and falls back for nested arrays whose innermost element is `Struct` or `Null`.
+- Spark 4.0.1 (audited 2026-05-27): trait set changes substantively: `ArraySortLike` and `NullIntolerant` are removed, `nullIntolerant = true` becomes an override, and `ascendingOrder` is widened to accept any foldable boolean (not just `Literal`). Comet's `CometSortArray` still requires a `Literal`, so the new foldable form falls back at convert time.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+[Spark Expression Support]: ../../user-guide/latest/expressions.md
diff --git a/_sources/contributor-guide/expression-audits/bitwise_funcs.md.txt b/_sources/contributor-guide/expression-audits/bitwise_funcs.md.txt
new file mode 100644
index 0000000000..56967b60fd
--- /dev/null
+++ b/_sources/contributor-guide/expression-audits/bitwise_funcs.md.txt
@@ -0,0 +1,101 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# bitwise_funcs Expression Audits
+
+> Audit notes for expressions in this category that have been audited. Absence of an entry means the expression has not been audited yet, not that it is unsupported. See the user guide [Spark Expression Support] for current support status.
+
+## `&`
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `BitwiseAnd(left, right) extends BinaryArithmetic` over `IntegralType`. Comet routes via `CometBitwiseAnd` to the proto's `bitwise_and` binary expression.
+- Spark 4.0.1 (audited 2026-05-27): semantics unchanged; `NullIntolerant` trait replaced by `nullIntolerant: Boolean`.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## `<<`
+
+- Spark 3.4.3 (audited 2026-05-27): only the function form `shiftleft(...)` exists; the `<<` operator alias is added in Spark 4.0.
+- Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.
+- Spark 4.0.1 (audited 2026-05-27): `<<` parses to `ShiftLeft` (the same `case class` as `shiftleft`) via a parser alias. `ShiftLeft` now extends a shared `BitShiftOperation` trait; eval semantics unchanged. Comet `CometShiftLeft` casts the `right` operand to `LongType` when `left.dataType == LongType` because DataFusion's bitwise shift requires matching operand types.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## `>>`
+
+- Spark 3.4.3 (audited 2026-05-27): same as `<<` history — only the function form exists pre-4.0.
+- Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.
+- Spark 4.0.1 (audited 2026-05-27): `>>` parses to `ShiftRight`. Comet `CometShiftRight` mirrors the same operand-cast logic as `CometShiftLeft`.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## `^`
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `BitwiseXor(left, right) extends BinaryArithmetic` over `IntegralType`. Comet routes via `CometBitwiseXor` to the proto's `bitwise_xor` binary expression.
+- Spark 4.0.1 (audited 2026-05-27): semantics unchanged; `nullIntolerant` field refactor.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## bit_count
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `BitwiseCount(child) extends UnaryExpression`; accepts `IntegralType` or `BooleanType` and returns the population count as `IntegerType`. Wired as `CometScalarFunction("bit_count")`.
+- Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## bit_get
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `BitwiseGet(left, right) extends BinaryExpression`; `inputTypes = (IntegralType, IntegerType) -> ByteType`. The position is bounds-checked at runtime via `BitwiseGetUtil.checkPosition`. Comet routes via `CometBitwiseGet` to the native `bit_get` scalar.
+- Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## getbit
+
+- Spark 3.4.3 (audited 2026-05-27): registry alias of `BitwiseGet`. Same support as `bit_get`.
+- Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.
+- Spark 4.0.1 (audited 2026-05-27): identical to 3.4.3.
+- Spark 4.1.1 (audited 2026-05-27): identical to 3.4.3.
+
+## shiftright
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. Function name for `ShiftRight`. Same support as the `>>` operator alias added in 4.0.
+- Spark 4.0.1 (audited 2026-05-27): identical to 3.5.8 (the `>>` operator is the only addition).
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## shiftrightunsigned
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `ShiftRightUnsigned(left, right) extends BinaryExpression`; uses Java's `>>>` (zero-fill) semantics. Comet wires as `CometScalarFunction("shiftrightunsigned")` (added in #4375).
+- Spark 4.0.1 (audited 2026-05-27): now extends the shared `BitShiftOperation` trait; eval semantics unchanged. The `>>>` operator alias is added at the parser level.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## `|`
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `BitwiseOr(left, right) extends BinaryArithmetic` over `IntegralType`. Comet routes via `CometBitwiseOr` to the proto's `bitwise_or` binary expression.
+- Spark 4.0.1 (audited 2026-05-27): semantics unchanged; `nullIntolerant` field refactor.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## `~`
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `BitwiseNot(child) extends UnaryExpression`; accepts `IntegralType` and returns the bitwise complement of the same type. Comet routes via `CometBitwiseNot` to the native `bitwise_not` scalar.
+- Spark 4.0.1 (audited 2026-05-27): semantics unchanged; `nullIntolerant` field refactor.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+[Spark Expression Support]: ../../user-guide/latest/expressions.md
diff --git a/_sources/contributor-guide/expression-audits/collection_funcs.md.txt b/_sources/contributor-guide/expression-audits/collection_funcs.md.txt
new file mode 100644
index 0000000000..2499d9572f
--- /dev/null
+++ b/_sources/contributor-guide/expression-audits/collection_funcs.md.txt
@@ -0,0 +1,52 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# collection_funcs Expression Audits
+
+> Audit notes for expressions in this category that have been audited. Absence of an entry means the expression has not been audited yet, not that it is unsupported. See the user guide [Spark Expression Support] for current support status.
+
+## array_size
+
+- Native via `size`; returns -1 instead of NULL for NULL input (https://github.com/apache/datafusion-comet/issues/4560).
+
+## concat
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `Concat(children) extends ComplexTypeMergingExpression with QueryErrorsBase`; `allowedTypes = Seq(StringType, BinaryType, ArrayType)`; result type is the merged child type. Empty children is allowed and returns the empty string of the result type.
+- Spark 4.0.1 (audited 2026-05-27): `allowedTypes` widens `StringType` to `StringTypeWithCollation(supportsTrimCollation = true)`. Error-formatting helper changes from `paramIndex` to `ordinalNumber`. Runtime semantics unchanged for `UTF8_BINARY`.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+- Known limitation: Comet only supports `StringType` children natively; `BinaryType` and `ArrayType` inputs fall back to Spark (https://github.com/apache/datafusion-comet/issues/4471). Non-default Spark 4.0 string collations are not propagated (https://github.com/apache/datafusion-comet/issues/2190).
+
+## reverse
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `Reverse(child) extends UnaryExpression with ImplicitCastInputTypes with NullIntolerant`; `inputTypes = Seq(TypeCollection(StringType, ArrayType))`; `dataType = child.dataType`. For string, calls `UTF8String.reverse()`; for array, reverses element order in-place via `GenericArrayData`.
+- Spark 4.0.1 (audited 2026-05-27): `NullIntolerant` trait replaced by `override def nullIntolerant: Boolean = true`; `inputTypes` widened to `Seq(TypeCollection(StringTypeWithCollation(supportsTrimCollation = true), ArrayType))`. Semantics unchanged for `UTF8_BINARY`.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+- Known limitation: `Reverse` on an array containing `BinaryType` elements is reported as `Incompatible` and falls back unless explicitly enabled (https://github.com/apache/datafusion-comet/issues/2763).
+
+## size
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `Size(child, legacySizeOfNull) extends UnaryExpression with ExpectsInputTypes`; `inputTypes = Seq(TypeCollection(ArrayType, MapType)) -> IntegerType`. `legacySizeOfNull=true` returns `-1` for NULL input; `false` returns NULL. Comet routes via `CometSize`, which emits a `CaseWhen(isNotNull(child), size_scalar(child), Literal(legacySizeOfNull))`.
+- Spark 4.0.1 (audited 2026-05-27): byte-for-byte identical to 3.5.8.
+- Spark 4.1.1 (audited 2026-05-27): byte-for-byte identical to 3.5.8.
+- Known limitation: `Size` over `MapType` falls back to Spark (https://github.com/apache/datafusion-comet/issues/4472).
+
+[Spark Expression Support]: ../../user-guide/latest/expressions.md
diff --git a/_sources/contributor-guide/expression-audits/conditional_funcs.md.txt b/_sources/contributor-guide/expression-audits/conditional_funcs.md.txt
new file mode 100644
index 0000000000..d24d51dbaa
--- /dev/null
+++ b/_sources/contributor-guide/expression-audits/conditional_funcs.md.txt
@@ -0,0 +1,73 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# conditional_funcs Expression Audits
+
+> Audit notes for expressions in this category that have been audited. Absence of an entry means the expression has not been audited yet, not that it is unsupported. See the user guide [Spark Expression Support] for current support status.
+
+## coalesce
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `Coalesce(children) extends Expression with ComplexTypeMergingExpression`; returns the first non-null child, evaluated left-to-right with short-circuit. Result type is the merged child type. Comet routes via `CometCoalesce`, which serialises as nested `CaseWhen(IsNotNull(c1) -> c1, IsNotNull(c2) -> c2, ..., else cN)` so the native engine preserves the short-circuit semantics.
+- Spark 4.0.1 (audited 2026-05-27): byte-for-byte identical to 3.5.8.
+- Spark 4.1.1 (audited 2026-05-27): byte-for-byte identical to 3.5.8.
+
+## if
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `If(predicate, trueValue, falseValue) extends ComplexTypeMergingExpression`; standard ternary semantics with short-circuit, predicate must be `BooleanType`. Comet routes via `CometIf` to the native `IfExpr` proto.
+- Spark 4.0.1 (audited 2026-05-27): adds an `override def withNewAlwaysEvaluatedInputs(...)` hook for the new optimizer phase; semantics unchanged. Error messages reformatted (`paramIndex` -> `ordinalNumber`).
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## ifnull
+
+- Spark 3.4.3 (audited 2026-05-27): registry alias of `Nvl` (`expression[Nvl]("ifnull", setAlias = true)`). Same `RuntimeReplaceable` lowering as `nvl`.
+- Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.
+- Spark 4.0.1 (audited 2026-05-27): identical to 3.4.3.
+- Spark 4.1.1 (audited 2026-05-27): identical to 3.4.3.
+
+## nullif
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): `NullIf(left, right, replacement) extends RuntimeReplaceable with InheritAnalysisRules`; the analyzer rewrites to `If(EqualTo(left, right), Literal(null, left.dataType), left)`. Comet handles via `CometIf` plus `CometEqualTo`.
+- Spark 4.0.1 (audited 2026-05-27): identical to 3.5.8.
+- Spark 4.1.1 (audited 2026-05-27): identical to 3.5.8.
+
+## nvl
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): `Nvl(left, right, replacement) extends RuntimeReplaceable`; analyzer rewrites to `Coalesce(Seq(left, right))`. Comet handles via `CometCoalesce`.
+- Spark 4.0.1 (audited 2026-05-27): identical to 3.5.8.
+- Spark 4.1.1 (audited 2026-05-27): identical to 3.5.8.
+
+## nvl2
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): `Nvl2(expr1, expr2, expr3, replacement) extends RuntimeReplaceable`; analyzer rewrites to `If(IsNotNull(expr1), expr2, expr3)`. Comet handles via `CometIf`.
+- Spark 4.0.1 (audited 2026-05-27): identical to 3.5.8.
+- Spark 4.1.1 (audited 2026-05-27): identical to 3.5.8.
+
+## when
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): the `CASE WHEN ... THEN ...` SQL form lowers to `CaseWhen(branches: Seq[(Expression, Expression)], elseValue: Option[Expression])`. Spark evaluates left-to-right with short-circuit; result type is the merged branch type. Comet routes via `CometCaseWhen` to the native `CaseWhen` proto.
+- Spark 4.0.1 (audited 2026-05-27): adds the `withNewAlwaysEvaluatedInputs` optimizer hook; semantics unchanged.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+[Spark Expression Support]: ../../user-guide/latest/expressions.md
diff --git a/_sources/contributor-guide/expression-audits/conversion_funcs.md.txt b/_sources/contributor-guide/expression-audits/conversion_funcs.md.txt
new file mode 100644
index 0000000000..45b6a8f03b
--- /dev/null
+++ b/_sources/contributor-guide/expression-audits/conversion_funcs.md.txt
@@ -0,0 +1,39 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# conversion_funcs Expression Audits
+
+> Audit notes for expressions in this category that have been audited. Absence of an entry means the expression has not been audited yet, not that it is unsupported. See the user guide [Spark Expression Support] for current support status.
+
+## cast
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8 modulo `Cast.canUpCast` refactored to delegate to `UpCastRule.canUpCast`.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `Cast(child, dataType, timeZoneId, evalMode)`; eval modes are `LEGACY`, `ANSI`, `TRY`. The legacy `Cast.canCast` matrix and the `Cast.canAnsiCast` matrix decide acceptance per type pair. Comet routes via `CometCast` (`spark/src/main/scala/org/apache/comet/expressions/CometCast.scala`) using a per-source-type support matrix that returns `Compatible`, `Incompatible(reason)`, or `Unsupported(reason)`; literal children are short-circuited to `Com [...]
+- Spark 4.0.1 (audited 2026-05-27): `VariantType` added; `StringType` literals replaced with `_: StringType` to accommodate collated strings. `(TimestampType, ByteType|ShortType|IntegerType)` added to `canAnsiCast`. `NullIntolerant` -> `nullIntolerant: Boolean` refactor. New `ToPrettyString.BinaryFormatter` semantics for `Binary -> String` are replicated natively via `spark_binary_formatter`. Numeric-to-numeric matrix unchanged.
+- Spark 4.1.1 (audited 2026-05-27): `TimeType` added; many `TimeType` arms in `canCast`/`canAnsiCast`. Geospatial `GeographyType` / `GeometryType` types added with their own conversion rules. Numeric-to-numeric matrix unchanged.
+- Known divergences and gaps:
+ - `CAST(<binary> AS STRING)` uses `unsafe { String::from_utf8_unchecked }` in native code, which is undefined behaviour for non-UTF8 inputs (https://github.com/apache/datafusion-comet/issues/4488).
+ - Spark 4.0 collated `StringType` is not explicitly guarded; pattern equality is expected to keep collated-string casts falling back, but there is no test (https://github.com/apache/datafusion-comet/issues/4489; umbrella #2190).
+ - Spark 4.1 `TimeType` casts have no explicit `Unsupported` arm; they fall back implicitly but do not appear in the auto-generated compatibility doc (https://github.com/apache/datafusion-comet/issues/4490).
+ - `CAST(<map> AS <map>)` falls back to Spark even though native `cast_map_to_map` exists (https://github.com/apache/datafusion-comet/issues/4491).
+ - `spark.sql.legacy.castComplexTypesToString.enabled=true` is not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4492).
+ - `CAST(<float|double> AS DECIMAL)` rounding may differ from Spark (`Incompatible`, gated by `spark.comet.expression.Cast.allowIncompatible`, tracked at https://github.com/apache/datafusion-comet/issues/1371).
+- Spark registers the type-name conversion functions (`bigint`, `binary`, `boolean`, `date`, `decimal`, `double`, `float`, `int`, `smallint`, `string`, `timestamp`, `tinyint`) as cast aliases. Each lowers to the same `Cast` node, so Comet handles it via the `cast` implementation with the same compatibility profile.
+
+[Spark Expression Support]: ../../user-guide/latest/expressions.md
diff --git a/_sources/contributor-guide/expression-audits/datetime_funcs.md.txt b/_sources/contributor-guide/expression-audits/datetime_funcs.md.txt
new file mode 100644
index 0000000000..e2958b2eea
--- /dev/null
+++ b/_sources/contributor-guide/expression-audits/datetime_funcs.md.txt
@@ -0,0 +1,90 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# datetime_funcs Expression Audits
+
+> Audit notes for expressions in this category that have been audited. Absence of an entry means the expression has not been audited yet, not that it is unsupported. See the user guide [Spark Expression Support] for current support status.
+
+## curdate
+
+- Alias of `current_date`; constant-folded to a literal by Spark's `ComputeCurrentTime` rule before Comet sees the plan.
+
+## current_date
+
+- Constant-folded to a literal by Spark's `ComputeCurrentTime` rule before Comet sees the plan.
+
+## current_timestamp
+
+- Constant-folded to a literal by Spark's `ComputeCurrentTime` rule before Comet sees the plan.
+
+## dayname
+
+- Spark 4.0+. Implemented natively: maps a `DateType` value to a fixed US-English abbreviated day name (`DayOfWeek.getDisplayName(TextStyle.SHORT, Locale.US)`), with no session-locale or timezone dependence.
+
+## from_utc_timestamp
+
+- Spark 3.4.3 (audited 2026-05-12): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-12): baseline.
+- Spark 4.0.1 (audited 2026-05-12): `inputTypes` widened to `StringTypeWithCollation`; behaviour unchanged for ASCII timezone strings.
+- Known divergence: Comet's native timezone parser does not accept Spark's legacy zone forms (`GMT+1`, `UTC+1`, three-letter abbreviations like `PST`). Such timezones throw a native parse error at execution.
+
+## make_timestamp_ltz
+
+- The 6-argument form rewrites to `MakeTimestamp` and runs via the codegen dispatcher. The 2-argument `(date, time)` form requires the Spark 4.1 TIME type and falls back.
+
+## make_timestamp_ntz
+
+- The 6-argument form rewrites to `MakeTimestamp` and runs via the codegen dispatcher. The 2-argument `(date, time)` form requires the Spark 4.1 TIME type and falls back.
+
+## monthname
+
+- Spark 4.0+. Implemented natively: maps a `DateType` value to a fixed US-English abbreviated month name (`Month.getDisplayName(TextStyle.SHORT, Locale.US)`), with no session-locale or timezone dependence.
+
+## now
+
+- Alias of `current_timestamp`; constant-folded to a literal by Spark's `ComputeCurrentTime` rule before Comet sees the plan.
+
+## to_date
+
+- Rewrites to `Cast` (no format, native) or `Cast(GetTimestamp(...))` (with format, via the codegen dispatcher) before Comet sees the plan.
+
+## to_timestamp
+
+- Rewrites to `Cast` (no format, native) or `GetTimestamp` (with format, via the codegen dispatcher) before Comet sees the plan.
+
+## to_timestamp_ltz
+
+- Rewrites to `to_timestamp` with `TimestampType`; same support as `to_timestamp`.
+
+## to_timestamp_ntz
+
+- Rewrites to `to_timestamp` with `TimestampNTZType`; same support as `to_timestamp`.
+
+## to_utc_timestamp
+
+- Spark 3.4.3 (audited 2026-05-12): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-12): baseline.
+- Spark 4.0.1 (audited 2026-05-12): `inputTypes` widened to `StringTypeWithCollation`; behaviour unchanged for ASCII timezone strings.
+- Known divergence: Comet's native timezone parser does not accept Spark's legacy zone forms (`GMT+1`, `UTC+1`, three-letter abbreviations like `PST`). Such timezones throw a native parse error at execution.
+
+## try_make_timestamp
+
+- Native for valid inputs; returns wrong values for invalid inputs instead of NULL (https://github.com/apache/datafusion-comet/issues/4554).
+
+[Spark Expression Support]: ../../user-guide/latest/expressions.md
diff --git a/_sources/contributor-guide/expression-audits/generator_funcs.md.txt b/_sources/contributor-guide/expression-audits/generator_funcs.md.txt
new file mode 100644
index 0000000000..3d797e83b3
--- /dev/null
+++ b/_sources/contributor-guide/expression-audits/generator_funcs.md.txt
@@ -0,0 +1,40 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# generator_funcs Expression Audits
+
+> Audit notes for expressions in this category that have been audited. Absence of an entry means the expression has not been audited yet, not that it is unsupported. See the user guide [Spark Expression Support] for current support status.
+
+## explode
+
+- Handled at the operator level as a `GenerateExec` (`CometExplodeExec`), not via the expression serde maps, so it is not auto-detected by the function-registry checkbox logic. Compatible for array inputs; map inputs fall back ([#2837](https://github.com/apache/datafusion-comet/issues/2837)).
+
+## explode_outer
+
+- Same `CometExplodeExec` path as `explode`, but the `outer=true` case is `Incompatible` (empty arrays are not preserved as null outputs) and falls back unless `spark.comet.expr.allowIncompatible=true` ([datafusion#19053](https://github.com/apache/datafusion/issues/19053)).
+
+## posexplode
+
+- Handled at the operator level as a `GenerateExec` (`CometExplodeExec`), like `explode`. Compatible for array inputs; map inputs fall back ([#2837](https://github.com/apache/datafusion-comet/issues/2837)).
+
+## posexplode_outer
+
+- Same `CometExplodeExec` path as `posexplode`, but the `outer=true` case is `Incompatible` and falls back unless `spark.comet.expr.allowIncompatible=true` ([datafusion#19053](https://github.com/apache/datafusion/issues/19053)).
+
+[Spark Expression Support]: ../../user-guide/latest/expressions.md
diff --git a/_sources/contributor-guide/expression-audits/hash_funcs.md.txt b/_sources/contributor-guide/expression-audits/hash_funcs.md.txt
new file mode 100644
index 0000000000..35337b8fe8
--- /dev/null
+++ b/_sources/contributor-guide/expression-audits/hash_funcs.md.txt
@@ -0,0 +1,74 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# hash_funcs Expression Audits
+
+> Audit notes for expressions in this category that have been audited. Absence of an entry means the expression has not been audited yet, not that it is unsupported. See the user guide [Spark Expression Support] for current support status.
+
+## crc32
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `Crc32(child) extends UnaryExpression`; `inputTypes = Seq(BinaryType) -> LongType`. Wired as `CometScalarFunction("crc32")`.
+- Spark 4.0.1 (audited 2026-05-27): semantics unchanged; `NullIntolerant` trait replaced by `nullIntolerant: Boolean` override.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## hash
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `Murmur3Hash(children, seed) extends HashExpression[Int]`; produces a Murmur3 hash with a configurable Int seed and `IntegerType` result. Comet routes via `CometMurmur3Hash` to the native `murmur3_hash` UDF.
+- Spark 4.0.1 (audited 2026-05-27): semantics unchanged; some inner helper refactors only.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+- Known limitation: `DecimalType` children with precision > 18 fall back because Spark hashes them through Java `BigDecimal`; `TimeType` (Spark 4.0+) is also unsupported. The same limitations apply to `xxhash64`, `sha1`, `sha2` through the shared `HashUtils`.
+
+## md5
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `Md5(child) extends UnaryExpression`; `inputTypes = Seq(BinaryType) -> StringType`. Wired as `CometScalarFunction("md5")`.
+- Spark 4.0.1 (audited 2026-05-27): semantics unchanged; trait set gains `DefaultStringProducingExpression` and the `nullIntolerant: Boolean` refactor.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## sha
+
+- Spark 3.4.3 (audited 2026-05-27): registry alias of `Sha1`. Same support as `sha1`.
+- Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.
+- Spark 4.0.1 (audited 2026-05-27): identical to 3.4.3.
+- Spark 4.1.1 (audited 2026-05-27): identical to 3.4.3.
+
+## sha1
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `Sha1(child) extends UnaryExpression with NullIntolerant`; `inputTypes = Seq(BinaryType) -> StringType`. Comet routes via `CometSha1` to the native `sha1` UDF.
+- Spark 4.0.1 (audited 2026-05-27): trait set gains `DefaultStringProducingExpression` and `NullIntolerant` is replaced by `nullIntolerant: Boolean`. Runtime unchanged.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## sha2
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `Sha2(left, right) extends BinaryExpression`; `inputTypes = Seq(BinaryType, IntegerType) -> StringType`. The `numBits` argument selects SHA-224/256/384/512 (0 is treated as 256); other values return NULL. Comet routes via `CometSha2` to the native `sha2` UDF; non-foldable `numBits` falls back to Spark.
+- Spark 4.0.1 (audited 2026-05-27): trait set gains `DefaultStringProducingExpression` and the `nullIntolerant: Boolean` refactor. Runtime unchanged.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## xxhash64
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `XxHash64(children, seed) extends HashExpression[Long]`; produces an xxHash64 hash with a configurable Long seed and `LongType` result. Comet routes via `CometXxHash64` to the native `xxhash64` UDF.
+- Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+[Spark Expression Support]: ../../user-guide/latest/expressions.md
diff --git a/_sources/contributor-guide/expression-audits/index.md.txt b/_sources/contributor-guide/expression-audits/index.md.txt
new file mode 100644
index 0000000000..c891a77b84
--- /dev/null
+++ b/_sources/contributor-guide/expression-audits/index.md.txt
@@ -0,0 +1,46 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Expression Audits
+
+This section records per-version audit notes for Spark expressions that have been audited for Comet compatibility. These are findings from auditing the Spark implementation across versions (3.4.3, 3.5.8, 4.0.1), not a statement of support status.
+
+For the authoritative list of which expressions Comet supports, see the user guide [Spark Expression Support](../../user-guide/latest/expressions.md). To audit an expression, use the `audit-comet-expression` skill, which appends its findings to the relevant category page below.
+
+```{toctree}
+:maxdepth: 1
+
+agg_funcs <agg_funcs>
+array_funcs <array_funcs>
+bitwise_funcs <bitwise_funcs>
+collection_funcs <collection_funcs>
+conditional_funcs <conditional_funcs>
+conversion_funcs <conversion_funcs>
+datetime_funcs <datetime_funcs>
+generator_funcs <generator_funcs>
+hash_funcs <hash_funcs>
+json_funcs <json_funcs>
+map_funcs <map_funcs>
+math_funcs <math_funcs>
+misc_funcs <misc_funcs>
+predicate_funcs <predicate_funcs>
+struct_funcs <struct_funcs>
+url_funcs <url_funcs>
+window_funcs <window_funcs>
+```
diff --git a/_sources/contributor-guide/expression-audits/json_funcs.md.txt b/_sources/contributor-guide/expression-audits/json_funcs.md.txt
new file mode 100644
index 0000000000..4184ceac22
--- /dev/null
+++ b/_sources/contributor-guide/expression-audits/json_funcs.md.txt
@@ -0,0 +1,40 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# json_funcs Expression Audits
+
+> Audit notes for expressions in this category that have been audited. Absence of an entry means the expression has not been audited yet, not that it is unsupported. See the user guide [Spark Expression Support] for current support status.
+
+## from_json
+
+- Partial native support, marked `Incompatible` (requires explicit schema).
+
+## get_json_object
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `BinaryExpression with ExpectsInputTypes with CodegenFallback`; `inputTypes = Seq(StringType, StringType) -> StringType`. Eval is inline and uses Jackson with `RawStyle` output. Foldable paths are parsed once. Returns NULL for invalid JSON, missing paths, or `JsonProcessingException`.
+- Spark 4.0.1 (audited 2026-05-27): the eval is extracted into a `GetJsonObjectEvaluator` helper (no behaviour change). The trait set now mixes in `DefaultStringProducingExpression`, and `inputTypes` is widened to `StringTypeWithCollation(supportsTrimCollation = true)` for both arguments.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+- Known incompatibility: Spark accepts single-quoted JSON and unescaped control characters; Comet's native parser (built on `serde_json`) rejects both, so those inputs require `spark.comet.expression.GetJsonObject.allowIncompatible=true` and may still produce different results. Non-default Spark 4.0 string collations are not propagated (https://github.com/apache/datafusion-comet/issues/2190).
+
+## to_json
+
+- Partial native support; options and map/array inputs fall back.
+
+[Spark Expression Support]: ../../user-guide/latest/expressions.md
diff --git a/_sources/contributor-guide/expression-audits/map_funcs.md.txt b/_sources/contributor-guide/expression-audits/map_funcs.md.txt
new file mode 100644
index 0000000000..87de704956
--- /dev/null
+++ b/_sources/contributor-guide/expression-audits/map_funcs.md.txt
@@ -0,0 +1,81 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# map_funcs Expression Audits
+
+> Audit notes for expressions in this category that have been audited. Absence of an entry means the expression has not been audited yet, not that it is unsupported. See the user guide [Spark Expression Support] for current support status.
+
+## element_at
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `ElementAt(left, right, defaultValueOutOfBound, failOnError) extends GetMapValueUtil`; the parser routes `element_at(<array>, ...)` to one overload and `element_at(<map>, ...)` to another. Comet `CometElementAt` only supports `ArrayType` input; `MapType` input falls back.
+- Spark 4.0.1 (audited 2026-05-27): adds `nullIntolerant: Boolean` field; semantics unchanged.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## map_contains_key
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `MapContainsKey(left, right) extends RuntimeReplaceable with InheritAnalysisRules`; the analyzer rewrites to `ArrayContains(MapKeys(left), right)`. Comet routes via `CometMapContainsKey` which emits the equivalent `array_has(map_keys(map), key)`.
+- Spark 4.0.1 (audited 2026-05-27): semantics unchanged; minor trait refactors.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## map_entries
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `MapEntries(child)` returns an array of structs `<key, value>`. Wired to native `map_entries`.
+- Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## map_from_arrays
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `MapFromArrays(left, right) extends BinaryExpression with NullIntolerant`; Spark uses `ArrayBasedMapBuilder` to detect duplicate keys (subject to `spark.sql.mapKeyDedupPolicy`) and rejects null keys with `RuntimeException("Cannot use null as map key")`. Comet `CometMapFromArrays` wraps the inputs in `CaseWhen(IsNotNull(left) AND IsNotNull(right), map(left, right), null)` so NULL-array inputs return NULL rather than triggering the previously r [...]
+- Spark 4.0.1 (audited 2026-05-27): semantics unchanged; `NullIntolerant` trait replaced by `nullIntolerant: Boolean`.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## map_from_entries
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `MapFromEntries(child) extends UnaryExpression with NullIntolerant`; expects an array of structs and produces a map. Wired as `CometScalarFunction("map_from_entries")`.
+- Spark 4.0.1 (audited 2026-05-27): semantics unchanged; trait refactor.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+- Known limitation: input arrays where the struct's key or value type contains `BinaryType` are marked `Incompatible` and fall back unless `spark.comet.expression.MapFromEntries.allowIncompatible=true`.
+
+## map_keys
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `MapKeys(child)` returns the map's keys as an array. Wired to native `map_keys`.
+- Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## map_values
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `MapValues(child)` returns the map's values as an array. Wired to native `map_values`.
+- Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## str_to_map
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `StringToMap(text, pairDelim, keyValueDelim) extends TernaryExpression`; splits `text` on `pairDelim`, then each pair on `keyValueDelim` (default `","` and `":"`). Uses `ArrayBasedMapBuilder` for duplicate-key handling. Wired as `CometScalarFunction("str_to_map")`.
+- Spark 4.0.1 (audited 2026-05-27): `inputTypes` widened to `StringTypeNonCSAICollation`; uses `CollationAwareUTF8String.splitSQL` with a `collationId`. Runtime unchanged for `UTF8_BINARY`.
+- Spark 4.1.1 (audited 2026-05-27): adds the `legacySplitTruncate` flag (driven by `spark.sql.legacy.truncateForEmptyRegexSplit`) to both `splitSQL` calls (https://github.com/apache/datafusion-comet/issues/4477). The Comet native impl does not honour this flag; behaviour matches the non-legacy default.
+
+[Spark Expression Support]: ../../user-guide/latest/expressions.md
diff --git a/_sources/contributor-guide/expression-audits/math_funcs.md.txt b/_sources/contributor-guide/expression-audits/math_funcs.md.txt
new file mode 100644
index 0000000000..0edb9fac09
--- /dev/null
+++ b/_sources/contributor-guide/expression-audits/math_funcs.md.txt
@@ -0,0 +1,274 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# math_funcs Expression Audits
+
+> Audit notes for expressions in this category that have been audited. Absence of an entry means the expression has not been audited yet, not that it is unsupported. See the user guide [Spark Expression Support] for current support status.
+
+## `%`
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `Remainder(left, right, evalMode)` signature identical across versions. Native path uses Rust `spark_modulo` UDF; non-ANSI returns NULL on divide-by-zero, ANSI raises `DIVIDE_BY_ZERO` / `REMAINDER_BY_ZERO`. `CometRemainder` rejects `EvalMode.TRY`, so `try_mod` (Spark 4.0+) falls back to Spark (https://github.com/apache/datafusion-comet/issues/4484).
+
+## `*`
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `Multiply(left, right, evalMode)` signature identical. Decimal results exceeding `DECIMAL128_MAX_PRECISION` go through `WideDecimalBinaryExpr` (Decimal256 intermediate); smaller decimals and primitives use DataFusion `BinaryExpr`. ANSI integer overflow uses Rust `checked_mul`. Interval multiplication falls back.
+
+## `+`
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `Add(left, right, evalMode)` with the same Decimal / ANSI plumbing as `*`. `Date + Int8/16/32` dispatches to the Rust `date_add` UDF to work around DataFusion's Date32 + Interval-only kernel.
+
+## `-`
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `Subtract(left, right, evalMode)` mirrors `+`. `Date - Int8/16/32` uses the Rust `date_sub` UDF.
+
+## `/`
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `Divide(left, right, evalMode)`. Non-ANSI mode wraps the divisor in `If(EqualTo(right, 0), null, right)` so DataFusion never throws. Decimal output is wrapped in `CheckOverflow(failOnError = ANSI)`; ANSI surfaces `NUMERIC_VALUE_OUT_OF_RANGE`, non-ANSI returns NULL.
+
+## abs
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `Abs(child, failOnError)` over `NumericType` plus the two interval types. `failOnError` (ANSI) is propagated to the native `abs` UDF, which throws `ARITHMETIC_OVERFLOW` on `Int.MinValue` / `Long.MinValue` / Decimal MIN. `DayTimeIntervalType` and `YearMonthIntervalType` fall back to Spark. Spark 4.0 / 4.1 do the `NullIntolerant` -> `nullIntolerant: Boolean` refactor; behaviour unchanged.
+
+## acos
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `UnaryMathExpression(math.acos, "ACOS")` unchanged across versions; wired as `CometScalarFunction("acos")` to DataFusion's `acos` UDF. NaN for `|x| > 1`.
+
+## acosh
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): custom `StrictMath.log(x + sqrt(x*x - 1))` unchanged across versions. NaN for `x < 1`. Routes to DataFusion's `acosh`.
+
+## asin
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `UnaryMathExpression(math.asin, "ASIN")` unchanged. NaN for `|x| > 1`.
+
+## asinh
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): special-cases `Double.NegativeInfinity` to avoid `log(NaN)`, otherwise `StrictMath.log(x + sqrt(x*x + 1))`. Identical across versions.
+
+## atan
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `UnaryMathExpression(math.atan, "ATAN")` unchanged.
+
+## atan2
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `BinaryMathExpression(math.atan2, "ATAN2")` with both inputs adjusted by `+0.0` to flip `-0.0` to `+0.0`. `CometAtan2` reproduces this by wrapping each child in `Add(child, Literal.default(child.dataType))` before dispatching to DataFusion's `atan2`.
+
+## atanh
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): custom `0.5 * (log1p(x) - log1p(-x))` (SPARK-28519). NaN for `|x| > 1`, +/-Infinity for `x = +/-1`.
+
+## bin
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `Bin(child)` over `LongType -> StringType`. Spark 4.x gains `DefaultStringProducingExpression` and the `nullIntolerant: Boolean` refactor; no behaviour change. Routes to datafusion-spark `SparkBin`.
+
+## cbrt
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): passthrough to DataFusion `cbrt`.
+
+## ceil
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): one-arg `ceil(expr)` supported (`LongType` / `DoubleType` / `DecimalType` with scale >= 0). Decimal with negative scale falls back at convert time. The two-arg `ceil(expr, scale)` form (`RoundCeil`) is not wired and falls back to Spark.
+
+## ceiling
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): registry alias for `Ceil`. Same support as `ceil`.
+
+## cos
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `UnaryMathExpression(math.cos, "COS")` unchanged across versions.
+
+## cosh
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `UnaryMathExpression(math.cosh, "COSH")` unchanged.
+
+## cot
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): custom `1 / math.tan(x)`. DataFusion's `cot` is also `1.0 / tan(x)`, so the result matches.
+
+## csc
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): custom `1 / math.sin(x)`. Routed to datafusion-spark's `SparkCsc` (registered in `jni_api.rs`).
+
+## degrees
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `UnaryMathExpression(math.toDegrees, "DEGREES")` unchanged across versions.
+
+## div
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `IntegralDivide(left, right, evalMode)`. Non-decimal operands are cast to `DecimalType(19, 0)`; result is recomputed per `IntegralDivide.resultDecimalType`, wrapped in `CheckOverflow`, then cast to `Long`. ANSI overflow for `Long.MinValue div -1` and decimal-overflow ANSI cases are covered by existing tests.
+
+## e
+
+- Foldable; rewritten to a literal by ConstantFolding (like `pi`).
+
+## exp
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `UnaryMathExpression(StrictMath.exp, "EXP")` unchanged. ULP-level differences vs DataFusion `exp` are possible but unflagged.
+
+## expm1
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `UnaryMathExpression(StrictMath.expm1, "EXPM1")` unchanged.
+
+## factorial
+
+- 3.4.3 (audited 2026-05-15): identical to v3.5.8.
+- 3.5.8 (audited 2026-05-15): canonical reference; `extends UnaryExpression with ImplicitCastInputTypes with NullIntolerant`. Returns NULL for NULL input or values outside `[0, 20]`.
+- 4.0.1 (audited 2026-05-15): `NullIntolerant` trait replaced by `nullIntolerant: Boolean` method override; behavior unchanged.
+- 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## floor
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): mirror of `ceil`. Two-arg `floor(expr, scale)` form (`RoundFloor`) falls back to Spark.
+
+## greatest
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): NULL-skipping variadic. Wired as `CometScalarFunction("greatest")` to DataFusion's `GreatestFunc`. Comet does not gate input types, so interval inputs and other Spark-only orderings rely on the native UDF accepting them; no explicit fallback path.
+
+## hex
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): accepts `LongType` / `BinaryType` / `StringType`. Spark 4.x widens `StringType` to `StringTypeWithCollation` and preserves collation in `dataType`; `CometHex` passes `expr.dataType` to native `SparkHex`, which always returns `Utf8` -- collation propagation may diverge on Spark 4.x.
+
+## least
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): mirror of `greatest`; same caveats. Spark 4.1.1 adds `contextIndependentFoldable` (no Comet impact).
+
+## ln
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): registry alias for `Log`. Comet wires through `CometLog` to DataFusion `ln` with a `nullIfNegative` rewrite to match Spark's NULL behaviour for `x <= 0`.
+
+## log
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): one-arg `log(x)` -> `CometLog` (DataFusion `ln`); two-arg `log(base, x)` -> `CometLogarithm` (custom `spark_log` UDF, returns NULL when `base <= 0` or `x <= 0` to match `Logarithm.nullSafeEval`).
+
+## log10
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `UnaryLogExpression(StrictMath.log10, "LOG10")`; returns NULL for `x <= 0`. Possible ULP differences from `StrictMath`.
+
+## log2
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `UnaryLogExpression(StrictMath.log(x) / StrictMath.log(2), "LOG2")`; returns NULL for `x <= 0`.
+
+## mod
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): registry alias for `Remainder`. Same support as `%`.
+
+## negative
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `UnaryMinus(child, failOnError)` -> Rust `NegativeExpr`. ANSI overflow is detected for `Int8/Int16/Int32/Int64` and `IntervalYearMonth/IntervalDayTime`. Float / Double / Decimal cannot overflow on negate. Spark 4.0 `NullIntolerant` -> `nullIntolerant: Boolean` refactor; no impact.
+
+## pi
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `LeafMathExpression(math.Pi, "PI")`; foldable, so Spark `ConstantFolding` rewrites it to a `Literal` before Comet sees the plan. The `CometScalarFunction("pi")` registration is exercised only when `ConstantFolding` is excluded.
+
+## positive
+
+- Spark 3.4.3, 3.5.8 (audited 2026-05-27): `UnaryPositive(child)` is a regular expression. There is no Comet serde for `UnaryPositive`, so projections containing `+col` silently disable Comet for the projection on 3.4/3.5.
+- Spark 4.0.1, 4.1.1 (audited 2026-05-27): `UnaryPositive` is `RuntimeReplaceable` with `replacement = child`; the optimizer removes it before Comet sees the plan, so the gap is transparent on 4.x.
+
+## pow
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `Pow(left, right) extends BinaryMathExpression(StrictMath.pow, "POWER")`; routes to DataFusion `pow`. ULP-level differences possible.
+
+## power
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): registry alias for `Pow`. Same support as `pow`.
+
+## radians
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `UnaryMathExpression(math.toRadians, "RADIANS")` unchanged across versions.
+
+## rand
+
+- See `misc_funcs / rand`.
+
+## randn
+
+- See `misc_funcs / randn`.
+
+## rint
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `UnaryMathExpression(math.rint, "ROUND")` with `funcName = "rint"`. Passthrough to DataFusion `rint` (round-half-to-even).
+
+## round
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): HALF_UP rounding for integer / decimal. Float / Double child types always fall back to Spark because `BigDecimal`-via-`toString` rounding cannot be precisely matched (documented inline in `CometRound`). ANSI `failOnError` is propagated for integer overflow. `BRound` (HALF_EVEN) is not wired.
+
+## sec
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): custom `1 / math.cos(x)`. Routed to datafusion-spark's `SparkSec`.
+
+## shiftleft
+
+- See `bitwise_funcs / <<` (audited in PR #4479). Same support as the operator alias added in 4.0.
+
+## sign
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): registry alias for `Signum`. Same support as `signum`.
+
+## signum
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `Signum(child)` over `DoubleType`. Spark also restricts to the two interval types via `inputTypes`; Comet handles only the `Double` case via DataFusion `signum`.
+
+## sin
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `UnaryMathExpression(math.sin, "SIN")` unchanged.
+
+## sinh
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `UnaryMathExpression(math.sinh, "SINH")` unchanged.
+
+## sqrt
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `UnaryMathExpression(math.sqrt, "SQRT")` unchanged.
+
+## tan
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `UnaryMathExpression(math.tan, "TAN")` unchanged.
+
+## tanh
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `UnaryMathExpression(math.tanh, "TANH")` unchanged.
+
+## try_add
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `TryAdd` is `RuntimeReplaceable` and rewrites to `Add(.., EvalMode.TRY)` for numeric inputs (datetime / interval go through `TryEval(Add(.., ANSI))` and fall back). Numeric path uses the Rust `checked_add` UDF, returning NULL on overflow. Decimal goes through `WideDecimalBinaryExpr` with `EvalMode.Try`.
+
+## try_divide
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `TryDivide` rewrites to `Divide(.., EvalMode.TRY)`. The `nullIfWhenPrimitive` wrapper swaps zero divisors to NULL; integer / float divide uses `checked_div`; decimal uses `decimal_div` + `CheckOverflow(failOnError = false)` returning NULL.
+
+## try_multiply
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): rewrites to `Multiply(.., EvalMode.TRY)`. Integer path uses `checked_mul`; decimal uses `WideDecimalBinaryExpr` with `EvalMode.Try`, returning NULL on overflow.
+
+## try_subtract
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): rewrites to `Subtract(.., EvalMode.TRY)`. Integer path uses `checked_sub`; decimal uses `WideDecimalBinaryExpr` as needed.
+
+## unhex
+
+- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `Unhex(child, failOnError)`. Spark 4.x widens input to `StringTypeWithCollation` and wraps the inner call in try/catch; Comet `CometUnhex` forwards `failOnError` to native `spark_unhex` but does not gate on collation.
+
+## width_bucket
+
+- Spark 3.5.8 (audited 2026-05-27): introduced; not available in 3.4.3.
+- Spark 4.0.1, 4.1.1 (audited 2026-05-27): same semantics; `NullIntolerant` -> `nullIntolerant: Boolean` refactor.
+- Known limitation: wired via per-version `CometExprShim` rather than a `CometExpressionSerde`, so it bypasses the support-level framework and the auto-generated compatibility doc (https://github.com/apache/datafusion-comet/issues/4485). Native path uses datafusion-spark `SparkWidthBucket`; interval input types are not exercised by Comet tests.
+
+[Spark Expression Support]: ../../user-guide/latest/expressions.md
diff --git a/_sources/contributor-guide/expression-audits/misc_funcs.md.txt b/_sources/contributor-guide/expression-audits/misc_funcs.md.txt
new file mode 100644
index 0000000000..6fbefe1d24
--- /dev/null
+++ b/_sources/contributor-guide/expression-audits/misc_funcs.md.txt
@@ -0,0 +1,85 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# misc_funcs Expression Audits
+
+> Audit notes for expressions in this category that have been audited. Absence of an entry means the expression has not been audited yet, not that it is unsupported. See the user guide [Spark Expression Support] for current support status.
+
+## current_catalog
+
+- Resolved to a literal by the analyzer (`ReplaceCurrentLike`).
+
+## current_database
+
+- Resolved to a literal by the analyzer (`ReplaceCurrentLike`).
+
+## current_schema
+
+- Alias of `current_database`; resolved to a literal by the analyzer.
+
+## current_user
+
+- Resolved to a literal by the analyzer; same as `user`.
+
+## monotonically_increasing_id
+
+- Spark 3.4.3 (audited 2026-05-27): byte-for-byte identical to 4.1.1. `MonotonicallyIncreasingID() extends LeafExpression with Stateful`; produces a Long that encodes the partition id in the upper 31 bits and a per-partition row counter in the lower 33 bits. Comet emits an empty `MonotonicallyIncreasingId` proto and the native side produces the same encoding.
+- Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.
+- Spark 4.0.1 (audited 2026-05-27): identical to 3.4.3.
+- Spark 4.1.1 (audited 2026-05-27): identical to 3.4.3.
+
+## rand
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `Rand(child, hideSeed) extends RDG` (an `UnaryExpression with ExpectsInputTypes with Nondeterministic with ExpressionWithRandomSeed`); `child` is the seed expression, coerced to `IntegerType` or `LongType` via `ImplicitCastInputTypes`. Uses `XORShiftRandom(seed + partitionIndex)` per partition and returns `nextDouble()` in `[0, 1)`. NULL seed evaluates to `0L` (via `null.asInstanceOf[Long]`).
+- Spark 4.0.1 (audited 2026-05-27): `RDG` is refactored from an `abstract class` into a trait, and `Rand` now extends a new `NondeterministicUnaryRDG` base. `ExpressionWithRandomSeed.expressionToSeed` is hoisted as a shared helper and throws `QueryCompilationErrors.invalidRandomSeedParameter` for non-literal seeds at analysis time. Runtime semantics unchanged.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+- Comet limitation: the seed argument must be a literal (column-reference seeds are rejected via `getSupportLevel`). Pre-4.0 Spark would otherwise silently fail at runtime; 4.0+ rejects at analysis time before the expression reaches Comet.
+
+## randn
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): same base as `Rand`; differs only in the eval body (`nextGaussian()` instead of `nextDouble()`), producing values from the standard normal distribution.
+- Spark 4.0.1 (audited 2026-05-27): same refactor as `Rand`; runtime unchanged.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+- Comet limitation: same as `rand` — the seed argument must be a literal.
+
+## session_user
+
+- Alias of `current_user`; resolved to a literal by the analyzer.
+
+## spark_partition_id
+
+- Spark 3.4.3 (audited 2026-05-27): byte-for-byte identical to 4.1.1. `SparkPartitionID() extends LeafExpression with Nondeterministic`; returns the integer index of the partition being processed. Comet emits an empty `SparkPartitionId` proto.
+- Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.
+- Spark 4.0.1 (audited 2026-05-27): identical to 3.4.3.
+- Spark 4.1.1 (audited 2026-05-27): identical to 3.4.3.
+
+## typeof
+
+- Foldable; resolved to a literal before Comet sees the plan.
+
+## user
+
+- Spark 3.4.3 (audited 2026-05-27): `CurrentUser() extends LeafExpression with Unevaluable`; the analyzer's `ResolveCurrentLike` rule replaces it with a `StringType` literal of the current user name before Comet sees the plan. No Comet serde needed; the literal flows through `CometLiteral`.
+- Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.
+- Spark 4.0.1 (audited 2026-05-27): identical to 3.4.3 except the resulting literal carries the default string collation.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+[Spark Expression Support]: ../../user-guide/latest/expressions.md
diff --git a/_sources/contributor-guide/expression-audits/predicate_funcs.md.txt b/_sources/contributor-guide/expression-audits/predicate_funcs.md.txt
new file mode 100644
index 0000000000..0731abf7f7
--- /dev/null
+++ b/_sources/contributor-guide/expression-audits/predicate_funcs.md.txt
@@ -0,0 +1,151 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# predicate_funcs Expression Audits
+
+> Audit notes for expressions in this category that have been audited. Absence of an entry means the expression has not been audited yet, not that it is unsupported. See the user guide [Spark Expression Support] for current support status.
+
+## `!`
+
+- Spark 3.4.3 (audited 2026-05-27): registry alias of `Not`. Same support as `not`.
+- Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.
+- Spark 4.0.1 (audited 2026-05-27): identical to 3.4.3.
+- Spark 4.1.1 (audited 2026-05-27): identical to 3.4.3.
+
+## `<`
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `LessThan(left, right) extends BinaryComparison`. Comet routes via `CometLessThan` to the proto's `lt` binary expression.
+- Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## `<=`
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `LessThanOrEqual(left, right) extends BinaryComparison`. Comet routes via `CometLessThanOrEqual` to the proto's `lt_eq` binary expression.
+- Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## `<=>`
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `EqualNullSafe(left, right) extends BinaryComparison`; treats two NULLs as equal and a NULL with a non-NULL as not equal. Comet routes via `CometEqualNullSafe` to the proto's `eq_null_safe` binary expression.
+- Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## `=`
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `EqualTo(left, right) extends BinaryComparison`. Comet routes via `CometEqualTo` to the proto's `eq` binary expression.
+- Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## `==`
+
+- Spark 3.4.3 (audited 2026-05-27): registry alias of `EqualTo`. Same support as `=`.
+- Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.
+- Spark 4.0.1 (audited 2026-05-27): identical to 3.4.3.
+- Spark 4.1.1 (audited 2026-05-27): identical to 3.4.3.
+
+## `>`
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `GreaterThan(left, right) extends BinaryComparison`. Comet routes via `CometGreaterThan` to the proto's `gt` binary expression.
+- Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## `>=`
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `GreaterThanOrEqual(left, right) extends BinaryComparison`. Comet routes via `CometGreaterThanOrEqual` to the proto's `gt_eq` binary expression.
+- Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## and
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `And(left, right) extends BinaryOperator with Predicate`; short-circuit left-to-right. Comet routes via `CometAnd` to the proto's `and` binary expression.
+- Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## between
+
+- Spark 3.4.3 (audited 2026-05-27): the SQL form `expr BETWEEN low AND high` is rewritten at the parser level to `expr >= low AND expr <= high`. Comet sees only the resulting `And(GreaterThanOrEqual, LessThanOrEqual)` and routes via `CometAnd` + `CometGreaterThanOrEqual` + `CometLessThanOrEqual`.
+- Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.
+- Spark 4.0.1 (audited 2026-05-27): identical to 3.4.3.
+- Spark 4.1.1 (audited 2026-05-27): identical to 3.4.3.
+
+## ilike
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `ILike(left, right, escapeChar) extends RuntimeReplaceable`; the analyzer rewrites to `Like(Lower(left), Lower(right), escapeChar)`. Comet handles via `CometLike` and `CometLower` (case-conversion path, gated by `spark.comet.caseConversion.enabled=false` by default).
+- Spark 4.0.1 (audited 2026-05-27): identical to 3.5.8.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## in
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `In(value, list: Seq[Expression]) extends Predicate`; NULL-sensitive set membership (NULL in any element of the list yields NULL when no match found). Comet routes via `CometIn` to the proto's `In` expression with `negated = false`; `InSet` is rewritten to `In` so the native side can perform its own set-lookup optimization.
+- Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## isnan
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `IsNaN(child) extends UnaryExpression`; returns `BooleanType`; `false` for NULL or non-float types. Comet routes via `CometIsNaN` to the native `isnan` scalar.
+- Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## isnotnull
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `IsNotNull(child) extends UnaryExpression with Predicate`. Comet routes via `CometIsNotNull` to the proto's `is_not_null` unary expression.
+- Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## isnull
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `IsNull(child) extends UnaryExpression with Predicate`. Comet routes via `CometIsNull` to the proto's `is_null` unary expression.
+- Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## like
+
+- See `string_funcs / like` (audited in PR #4461). `CometLike` only supports the default `\\` escape character.
+
+## not
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `Not(child) extends UnaryExpression with Predicate`. Comet routes via `CometNot`, which optimizes a few special cases: `Not(EqualTo)` -> proto `neq`, `Not(EqualNullSafe)` -> proto `neq_null_safe`, `Not(In)` -> proto `In(negated = true)`.
+- Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## or
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `Or(left, right) extends BinaryOperator with Predicate`; short-circuit left-to-right. Comet routes via `CometOr` to the proto's `or` binary expression.
+- Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
+- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+
+## rlike
+
+- See `string_funcs / regexp_replace` and the `CometRLike` notes (audited in PR #4461). Uses the Rust `regex` crate, which differs from Java's `Pattern` engine; requires `spark.comet.expression.regexp.allowIncompatible=true`.
+
+[Spark Expression Support]: ../../user-guide/latest/expressions.md
diff --git a/_sources/contributor-guide/expression-audits/struct_funcs.md.txt b/_sources/contributor-guide/expression-audits/struct_funcs.md.txt
new file mode 100644
index 0000000000..fc0ab6f13a
--- /dev/null
+++ b/_sources/contributor-guide/expression-audits/struct_funcs.md.txt
@@ -0,0 +1,39 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# struct_funcs Expression Audits
+
+> Audit notes for expressions in this category that have been audited. Absence of an entry means the expression has not been audited yet, not that it is unsupported. See the user guide [Spark Expression Support] for current support status.
+
+## named_struct
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): baseline. `CreateNamedStruct(children)` with `NoThrow`; `children` is `Seq(name1, val1, name2, val2, ...)`. Names must be foldable `StringType` and distinct; non-null. `dataType` is the `StructType` of the resulting fields. Comet routes via `CometCreateNamedStruct`, builds a `CreateNamedStruct` proto, and the native `CreateNamedStruct` expression (in `native/spark-expr/src/struct_funcs/create_named_struct.rs`) emits a `StructArray`.
+- Spark 4.0.1 (audited 2026-05-27): semantics unchanged; an `override def stateful: Boolean = true` flag is added for the optimizer. Parser-level Alias unwrapping (`case (a: Alias, _) => Seq(Literal(a.name), a)`) is transparent to Comet.
+- Spark 4.1.1 (audited 2026-05-27): semantics unchanged; adds `override def contextIndependentFoldable: Boolean = children.forall(_.contextIndependentFoldable)` for the new constant-folding pass; no impact on Comet conversion.
+- Comet limitation: a `CreateNamedStruct` whose names contain duplicates falls back to Spark. (Spark's analyzer also tolerates duplicates at the column-name level, but the proto layer would lose them, so the fallback is the safe choice.)
+
+## struct
+
+- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-05-27): the SQL function `struct(a, b, c)` is built by `CreateStruct.create`, which lowers to a `CreateNamedStruct` with synthetic field names `col1`, `col2`, etc. Comet handles both `struct` and `named_struct` via the same `CometCreateNamedStruct` path; the synthetic-name and explicit-name cases behave identically.
+- Spark 4.0.1 (audited 2026-05-27): same lowering; identical to 3.5.8.
+- Spark 4.1.1 (audited 2026-05-27): same lowering; identical to 3.5.8.
+
+[Spark Expression Support]: ../../user-guide/latest/expressions.md
diff --git a/_sources/contributor-guide/expression-audits/url_funcs.md.txt b/_sources/contributor-guide/expression-audits/url_funcs.md.txt
new file mode 100644
index 0000000000..c262762ef6
--- /dev/null
+++ b/_sources/contributor-guide/expression-audits/url_funcs.md.txt
@@ -0,0 +1,40 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# url_funcs Expression Audits
+
+> Audit notes for expressions in this category that have been audited. Absence of an entry means the expression has not been audited yet, not that it is unsupported. See the user guide [Spark Expression Support] for current support status.
+
+## try_url_decode
+
+- 4.0.1, 2026-05-05
+
+## url_decode
+
+- 3.4.3, 2026-04-29
+- 3.5.8, 2026-04-29
+- 4.0.1, 2026-04-29
+
+## url_encode
+
+- 3.4.3, 2026-04-29
+- 3.5.8, 2026-04-29
+- 4.0.1, 2026-04-29
+
+[Spark Expression Support]: ../../user-guide/latest/expressions.md
diff --git a/_sources/contributor-guide/expression-audits/window_funcs.md.txt b/_sources/contributor-guide/expression-audits/window_funcs.md.txt
new file mode 100644
index 0000000000..290337c15a
--- /dev/null
+++ b/_sources/contributor-guide/expression-audits/window_funcs.md.txt
@@ -0,0 +1,32 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# window_funcs Expression Audits
+
+> Audit notes for expressions in this category that have been audited. Absence of an entry means the expression has not been audited yet, not that it is unsupported. See the user guide [Spark Expression Support] for current support status.
+
+## lag
+
+- Supported via `CometWindowExec` (operator level), not the expression serde.
+
+## lead
+
+- Supported via `CometWindowExec` (operator level), not the expression serde.
+
+[Spark Expression Support]: ../../user-guide/latest/expressions.md
diff --git a/_sources/contributor-guide/index.md.txt b/_sources/contributor-guide/index.md.txt
index 55b6552371..5236d03b9d 100644
--- a/_sources/contributor-guide/index.md.txt
+++ b/_sources/contributor-guide/index.md.txt
@@ -88,7 +88,7 @@ Tracing <tracing>
:caption: Reference
:hidden:
-Supported Spark Expressions <spark_expressions_support>
+Expression Audits <expression-audits/index>
Supported Spark Configurations <spark_configs_support>
```
diff --git a/_sources/contributor-guide/release_process.md.txt b/_sources/contributor-guide/release_process.md.txt
index 8490ec84cf..0daf4416b6 100644
--- a/_sources/contributor-guide/release_process.md.txt
+++ b/_sources/contributor-guide/release_process.md.txt
@@ -57,8 +57,9 @@ instructions on each step.
Before starting the release process, review the user guide to ensure it accurately reflects the current state of the
project:
-- Review the supported expressions and operators lists in the user guide. Verify that any expressions added since
- the last release are included and that their support status is accurate.
+- Review the [Spark Expression Support](../user-guide/latest/expressions.md) page and the supported operators list
+ in the user guide. The expression page is the source of truth for expression support status, so verify that any
+ expressions added or changed since the last release appear there with an accurate status (✅ / ⚠️ / 🔜 / 💤).
- Spot-check the support status of individual expressions by running tests or queries to confirm they work as
documented.
- Look for any expressions that may have regressed or changed behavior since the last release and update the
diff --git a/_sources/contributor-guide/spark_expressions_support.md.txt b/_sources/contributor-guide/spark_expressions_support.md.txt
deleted file mode 100644
index abef2c76f1..0000000000
--- a/_sources/contributor-guide/spark_expressions_support.md.txt
+++ /dev/null
@@ -1,1091 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-
-# Supported Spark Expressions
-
-## How to Read This Document
-
-- A function marked with `[x]` is implemented in Comet and can be evaluated without falling back to Spark. Comet serves such a function through one of two paths:
- - **Native:** a Rust/DataFusion implementation that runs entirely in native execution. This is the path used for most functions.
- - **Codegen dispatch:** a JVM kernel that runs Spark's own generated code over Arrow batches, used for some functions that do not have a native implementation. This path is gated by `spark.comet.exec.scalaUDF.codegen.enabled`; when it is disabled, the enclosing operator falls back to Spark.
- - A few entries (such as `explode` / `posexplode`) are instead handled at the operator level rather than as expressions; these are annotated inline.
-- A function marked with `[ ]` has no Comet implementation and always falls back to Spark.
-
-> **Note:** Some functions listed as supported may still be incompatible with Spark in
-> certain cases (data types, modes, edge values) and fall back to Spark at runtime. Full
-> per-function details are documented in the
-> [Compatibility Guide](https://datafusion.apache.org/comet/user-guide/latest/compatibility/expressions/index.html).
-
-### agg_funcs
-
-- [x] any
- - Spark 3.4.3 (audited 2026-05-26): registered as a SQL alias of `BoolOr`, which extends `RuntimeReplaceableAggregate` with `replacement = Max(child)`. Catalyst rewrites `any(x)` to `max(x)` before Comet sees the plan, so `any` is served by `CometMax` on a `BooleanType` column.
- - Spark 3.5.8 (audited 2026-05-26): identical to 3.4.3.
- - Spark 4.0.1 (audited 2026-05-26): identical to 3.4.3.
-- [x] any_value
-- [ ] approx_count_distinct
-- [ ] approx_percentile
-- [ ] approx_top_k
-- [ ] approx_top_k_accumulate
-- [ ] approx_top_k_combine
-- [ ] array_agg
-- [x] avg
- - Spark 3.4.3 (2026-05-26)
- - Spark 3.5.8 (2026-05-26): aggregate logic identical to 3.4.3
- - Spark 4.0.1 (2026-05-26): aggregate logic identical to 3.5.8; only `QueryContext` import path differs. `YearMonthIntervalType` and `DayTimeIntervalType` inputs (supported by Spark) fall back to Spark in Comet.
-- [x] bit_and
- - Spark 3.4.3 (2026-05-26)
- - Spark 3.5.8 (2026-05-26)
- - Spark 4.0.1 (2026-05-26)
-- [x] bit_or
-- [x] bit_xor
-- [x] bool_and
-- [x] bool_or
-- [ ] collect_list
-- [x] collect_set
-- [x] corr
-- [x] count
-- [x] count_if
-- [ ] count_min_sketch
-- [x] covar_pop
-- [x] covar_samp
-- [x] every
-- [x] first
-- [x] first_value
-- [ ] grouping
-- [ ] grouping_id
-- [ ] histogram_numeric
-- [ ] hll_sketch_agg
-- [ ] hll_union_agg
-- [ ] kll_sketch_agg_bigint
-- [ ] kll_sketch_agg_double
-- [ ] kll_sketch_agg_float
-- [ ] kll_sketch_get_n_bigint
-- [ ] kll_sketch_get_n_double
-- [ ] kll_sketch_get_n_float
-- [ ] kll_sketch_get_quantile_bigint
-- [ ] kll_sketch_get_quantile_double
-- [ ] kll_sketch_get_quantile_float
-- [ ] kll_sketch_get_rank_bigint
-- [ ] kll_sketch_get_rank_double
-- [ ] kll_sketch_get_rank_float
-- [ ] kll_sketch_merge_bigint
-- [ ] kll_sketch_merge_double
-- [ ] kll_sketch_merge_float
-- [ ] kll_sketch_to_string_bigint
-- [ ] kll_sketch_to_string_double
-- [ ] kll_sketch_to_string_float
-- [ ] kurtosis
-- [x] last
-- [x] last_value
-- [ ] listagg
-- [x] max
-- [ ] max_by
-- [x] mean
-- [ ] median
-- [x] min
-- [ ] min_by
-- [ ] mode
-- [ ] percentile
-- [ ] percentile_approx
-- [ ] percentile_cont
-- [ ] percentile_disc
-- [x] regr_avgx
-- [x] regr_avgy
-- [x] regr_count
-- [ ] regr_intercept
-- [ ] regr_r2
-- [ ] regr_slope
-- [ ] regr_sxx
-- [ ] regr_sxy
-- [ ] regr_syy
-- [ ] skewness
-- [x] some
-- [x] std
-- [x] stddev
-- [x] stddev_pop
-- [x] stddev_samp
-- [ ] string_agg
-- [x] sum
-- [ ] theta_intersection_agg
-- [ ] theta_sketch_agg
-- [ ] theta_union_agg
-- [ ] try_avg
-- [ ] try_sum
-- [x] var_pop
-- [x] var_samp
-- [x] variance
-
-### array_funcs
-
-- [x] array
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `CreateArray(children, useStringTypeWhenEmpty)`; element type is the common type of children. Comet routes via `CometCreateArray` (native `make_array`) and special-cases the empty-array case to dodge a known DataFusion `coerce_types` issue (#3338).
- - Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
- - Spark 4.1.1 (audited 2026-05-27): adds `contextIndependentFoldable` override; runtime semantics unchanged.
-- [x] array_append
- - Spark 3.4.3 (audited 2026-05-27): standalone `BinaryExpression`, evaluated directly. Comet routes via `CometArrayAppend`.
- - Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.
- - Spark 4.0.1 (audited 2026-05-27): now `RuntimeReplaceable` and rewritten to `ArrayInsert(arr, Literal(-1), elem)`. `CometArrayAppend` is therefore unreachable; dispatch goes through `CometArrayInsert` (which carries its own `Incompatible` notes documented at the `array_insert` entry).
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] array_compact
- - Spark 3.4.3 (audited 2026-05-27): `RuntimeReplaceable` -> `ArrayFilter(arr, IsNotNull(lambda))`. Comet receives the rewritten form, dispatches through `CometArrayFilter`, which delegates back to `CometArrayCompact.convert` for the actual proto emission. The native path uses Comet's `spark_array_compact` UDF rather than DataFusion's `array_remove_all` because DataFusion 53 changed `array_remove_all`'s NULL semantics.
- - Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.
- - Spark 4.0.1 (audited 2026-05-27): the replacement is wrapped in `KnownNotContainsNull(...)` (analysis-only hint, no semantic change).
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] array_contains
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `ArrayContains(left, right) extends BinaryExpression with NullIntolerant with Predicate`; `inputTypes` uses `findWiderTypeWithoutStringPromotionForTwo`. Wired as `CometScalarFunction("array_contains")`.
- - Spark 4.0.1 (audited 2026-05-27): `NullIntolerant` trait replaced by `nullIntolerant: Boolean`; `checkInputDataTypes` adopts `DataTypeUtils.sameType` (collation-aware in 4.x).
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
- - Known limitation: no NaN-canonicalization guard in `getSupportLevel`. For `Float`/`Double` arrays containing NaN, Spark's `SQLOrderingUtil` may produce different results than DataFusion's IEEE comparison (https://github.com/apache/datafusion-comet/issues/4481).
-- [x] array_distinct
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `ArrayDistinct(child)` over `ArraySetLike`; uses `SQLOpenHashSet` so NaN and `+0.0`/`-0.0` are canonicalized. Wired as `CometScalarFunction("array_distinct")`.
- - Spark 4.0.1 (audited 2026-05-27): `NullIntolerant` -> `nullIntolerant` field refactor.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
- - Known divergence: DataFusion `array_distinct` uses hash-based equality without NaN/signed-zero canonicalization, so float/double arrays may produce different results (https://github.com/apache/datafusion-comet/issues/4481).
-- [x] array_except
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `ArrayExcept(left, right) extends ArrayBinaryLike with ComplexTypeMergingExpression`; result preserves left-side first occurrences not present in right. Comet routes via `CometArrayExcept` and unconditionally flags `Incompatible` ("Null handling and ordering may differ from Spark"); also falls back for `BinaryType` / `StructType` element types.
- - Spark 4.0.1 (audited 2026-05-27): `nullIntolerant = true` moves into `ArrayBinaryLike`; the overflow path uses `arrayFunctionWithElementsExceedLimitError`.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
- - Known divergence: same NaN/signed-zero canonicalization gap as `array_distinct` for float/double arrays (https://github.com/apache/datafusion-comet/issues/4481).
-- [x] array_insert
- - Spark 3.4.3 audited 2026-04-02
- - Spark 3.5.8 audited 2026-04-02
- - Spark 4.0.1 audited 2026-04-02 (pos=0 error message differs from Spark)
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] array_intersect
- - Spark 3.4.3 audited 2026-04-24 (result element order may differ from Spark when the right array is longer than the left; DataFusion probes the longer side)
- - Spark 3.5.8 audited 2026-04-24 (same ordering incompatibility as 3.4.3)
- - Spark 4.0.1 audited 2026-04-24 (ordering incompatibility as above; collated strings now fall back to Spark)
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] array_join
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `ArrayJoin(array, delimiter, nullReplacement)`. Comet routes via `CometArrayJoin` to DataFusion's `array_to_string` and is unconditionally flagged `Incompatible` ("Null handling may differ from Spark", #3178).
- - Spark 4.0.1 (audited 2026-05-27): `inputTypes` widened to `AbstractArrayType(StringTypeWithCollation(supportsTrimCollation = true))`; non-binary collations not propagated (https://github.com/apache/datafusion-comet/issues/2190).
- - Spark 4.1.1 (audited 2026-05-27): adds `contextIndependentFoldable` override; runtime unchanged.
-- [x] array_max
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `ArrayMax(child) extends UnaryExpression with ImplicitCastInputTypes`; skips NULL elements; for float/double Spark's `SQLOrderingUtil` treats NaN as greater than any non-NaN. Wired as `CometScalarFunction("array_max")`.
- - Spark 4.0.1 (audited 2026-05-27): `NullIntolerant` -> `nullIntolerant` field refactor.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
- - Known divergence: DataFusion's `array_max` uses Arrow `partial_cmp`-based ordering, so float/double arrays containing NaN may produce different results (https://github.com/apache/datafusion-comet/issues/4482).
-- [x] array_min
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): mirror of `ArrayMax` with `evalInternal` returning the minimum. Same NULL-skip and NaN-ordering semantics. Wired as `CometScalarFunction("array_min")`.
- - Spark 4.0.1 (audited 2026-05-27): same trait refactor as `array_max`.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
- - Known divergence: same NaN-handling gap as `array_max` (https://github.com/apache/datafusion-comet/issues/4482).
-- [x] array_position
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `ArrayPosition(left, right)`; returns 1-based `LongType` position, 0 if not found, NULL if either input is NULL. `CometArrayPosition` falls back for all-foldable args (constant folding handles those) and for unsupported element types (binary/struct/map/null).
- - Spark 4.0.1 (audited 2026-05-27): `NullIntolerant` -> `nullIntolerant` field refactor.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [ ] array_prepend
-- [x] array_remove
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `ArrayRemove(left, right)`; removes all occurrences equal to `right`. Wired as `CometScalarFunction("array_remove")`. Falls back via `ArraysBase.isTypeSupported` for binary/struct/map/null child types.
- - Spark 4.0.1 (audited 2026-05-27): `NullIntolerant` -> `nullIntolerant` field refactor.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] array_repeat
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `ArrayRepeat(left, right) extends BinaryExpression with ExpectsInputTypes`; `inputTypes = Seq(AnyDataType, IntegerType)`. NULL count yields NULL; count <= 0 yields empty array; count > `MAX_ROUNDED_ARRAY_LENGTH` throws at runtime. Comet wraps the call in `CaseWhen(IsNotNull(right), array_repeat(...), null)`.
- - Spark 4.0.1 (audited 2026-05-27): error message uses `createArrayWithElementsExceedLimitError(prettyName, count)`; semantics unchanged.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] array_union
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `ArrayUnion(left, right) extends ArrayBinaryLike with ComplexTypeMergingExpression`; result is left-side distinct elements followed by new right-side elements. Wired as `CometScalarFunction("array_union")`.
- - Spark 4.0.1 (audited 2026-05-27): `nullIntolerant = true` moves into `ArrayBinaryLike`; overflow path uses `arrayFunctionWithElementsExceedLimitError`.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
- - Known divergence: same NaN/signed-zero canonicalization gap as `array_distinct` (https://github.com/apache/datafusion-comet/issues/4481). Result ordering versus DataFusion is also unverified; compare the `array_intersect` ordering caveat.
-- [x] arrays_overlap
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `ArraysOverlap(left, right)`; three-valued logic (TRUE if any common non-null element, NULL if a null is present and no overlap is found in non-nulls, FALSE otherwise). Comet routes via `CometArraysOverlap` to the native `spark_arrays_overlap` UDF, which implements the same three-valued logic.
- - Spark 4.0.1 (audited 2026-05-27): `NullIntolerant` -> `nullIntolerant` field refactor.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] arrays_zip
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `ArraysZip(children, names)`; returns an array of structs, padding shorter inputs with NULL. Comet routes via `CometArraysZip` and rejects unsupported child element types (anything outside primitives, decimals, dates/timestamps, strings, binary, and nested arrays/structs of those).
- - Spark 4.0.1 (audited 2026-05-27): the length-mismatch error switches from `IllegalArgumentException` to `SparkIllegalArgumentException("_LEGACY_ERROR_TEMP_3235")`; runtime unchanged.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] element_at
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `ElementAt(left, right, defaultValueOutOfBound, failOnError)`; group label `map_funcs`. Comet supports only `ArrayType` input; `MapType` input falls back.
- - Spark 4.0.1 (audited 2026-05-27): `NullIntolerant` -> `nullIntolerant` field refactor; group label changes to `collection_funcs`; ANSI default flips to `true` so out-of-bound throws by default. Comet wires `failOnError` through to native `ListExtract`.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] flatten
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `Flatten(child) extends UnaryExpression`; returns NULL if any inner sub-array is NULL. Comet routes via `CometFlatten` and falls back for child types containing `BinaryType` / `StructType` / `MapType` (limitation of `ArraysBase.isTypeSupported`).
- - Spark 4.0.1 (audited 2026-05-27): `NullIntolerant` -> `nullIntolerant` field refactor.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] get
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `GetArrayItem(child, ordinal, failOnError)`; `inputTypes = Seq(AnyDataType, IntegralType)`. Comet routes via `CometGetArrayItem`, wiring `failOnError` through to the proto.
- - Spark 4.0.1 (audited 2026-05-27): semantics unchanged; ANSI default flips to `true`.
- - Spark 4.1.1 (audited 2026-05-27): `inputTypes` tightened to `Seq(ArrayType, IntegralType)` (analysis-time only); runtime unchanged.
-- [ ] sequence
-- [ ] shuffle
-- [x] slice
-- [x] sort_array
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `SortArray(base, ascendingOrder) extends BinaryExpression with ArraySortLike`; the second arg must be a `Literal(_: Boolean, BooleanType)`. Comet `CometSortArray` flags `Incompatible` under strict floating-point and falls back for nested arrays whose innermost element is `Struct` or `Null`.
- - Spark 4.0.1 (audited 2026-05-27): trait set changes substantively: `ArraySortLike` and `NullIntolerant` are removed, `nullIntolerant = true` becomes an override, and `ascendingOrder` is widened to accept any foldable boolean (not just `Literal`). Comet's `CometSortArray` still requires a `Literal`, so the new foldable form falls back at convert time.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-
-### bitwise_funcs
-
-- [x] `&`
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `BitwiseAnd(left, right) extends BinaryArithmetic` over `IntegralType`. Comet routes via `CometBitwiseAnd` to the proto's `bitwise_and` binary expression.
- - Spark 4.0.1 (audited 2026-05-27): semantics unchanged; `NullIntolerant` trait replaced by `nullIntolerant: Boolean`.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] `<<`
- - Spark 3.4.3 (audited 2026-05-27): only the function form `shiftleft(...)` exists; the `<<` operator alias is added in Spark 4.0.
- - Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.
- - Spark 4.0.1 (audited 2026-05-27): `<<` parses to `ShiftLeft` (the same `case class` as `shiftleft`) via a parser alias. `ShiftLeft` now extends a shared `BitShiftOperation` trait; eval semantics unchanged. Comet `CometShiftLeft` casts the `right` operand to `LongType` when `left.dataType == LongType` because DataFusion's bitwise shift requires matching operand types.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] `>>`
- - Spark 3.4.3 (audited 2026-05-27): same as `<<` history — only the function form exists pre-4.0.
- - Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.
- - Spark 4.0.1 (audited 2026-05-27): `>>` parses to `ShiftRight`. Comet `CometShiftRight` mirrors the same operand-cast logic as `CometShiftLeft`.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] `>>>`
-- [x] `^`
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `BitwiseXor(left, right) extends BinaryArithmetic` over `IntegralType`. Comet routes via `CometBitwiseXor` to the proto's `bitwise_xor` binary expression.
- - Spark 4.0.1 (audited 2026-05-27): semantics unchanged; `nullIntolerant` field refactor.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] bit_count
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `BitwiseCount(child) extends UnaryExpression`; accepts `IntegralType` or `BooleanType` and returns the population count as `IntegerType`. Wired as `CometScalarFunction("bit_count")`.
- - Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] bit_get
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `BitwiseGet(left, right) extends BinaryExpression`; `inputTypes = (IntegralType, IntegerType) -> ByteType`. The position is bounds-checked at runtime via `BitwiseGetUtil.checkPosition`. Comet routes via `CometBitwiseGet` to the native `bit_get` scalar.
- - Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] getbit
- - Spark 3.4.3 (audited 2026-05-27): registry alias of `BitwiseGet`. Same support as `bit_get`.
- - Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.
- - Spark 4.0.1 (audited 2026-05-27): identical to 3.4.3.
- - Spark 4.1.1 (audited 2026-05-27): identical to 3.4.3.
-- [x] shiftright
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. Function name for `ShiftRight`. Same support as the `>>` operator alias added in 4.0.
- - Spark 4.0.1 (audited 2026-05-27): identical to 3.5.8 (the `>>` operator is the only addition).
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] shiftrightunsigned
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `ShiftRightUnsigned(left, right) extends BinaryExpression`; uses Java's `>>>` (zero-fill) semantics. Comet wires as `CometScalarFunction("shiftrightunsigned")` (added in #4375).
- - Spark 4.0.1 (audited 2026-05-27): now extends the shared `BitShiftOperation` trait; eval semantics unchanged. The `>>>` operator alias is added at the parser level.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] `|`
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `BitwiseOr(left, right) extends BinaryArithmetic` over `IntegralType`. Comet routes via `CometBitwiseOr` to the proto's `bitwise_or` binary expression.
- - Spark 4.0.1 (audited 2026-05-27): semantics unchanged; `nullIntolerant` field refactor.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] `~`
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `BitwiseNot(child) extends UnaryExpression`; accepts `IntegralType` and returns the bitwise complement of the same type. Comet routes via `CometBitwiseNot` to the native `bitwise_not` scalar.
- - Spark 4.0.1 (audited 2026-05-27): semantics unchanged; `nullIntolerant` field refactor.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-
-### collection_funcs
-
-- [x] array_size
- - Native via `size`; returns -1 instead of NULL for NULL input (https://github.com/apache/datafusion-comet/issues/4560).
-- [x] cardinality
-- [x] concat
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `Concat(children) extends ComplexTypeMergingExpression with QueryErrorsBase`; `allowedTypes = Seq(StringType, BinaryType, ArrayType)`; result type is the merged child type. Empty children is allowed and returns the empty string of the result type.
- - Spark 4.0.1 (audited 2026-05-27): `allowedTypes` widens `StringType` to `StringTypeWithCollation(supportsTrimCollation = true)`. Error-formatting helper changes from `paramIndex` to `ordinalNumber`. Runtime semantics unchanged for `UTF8_BINARY`.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
- - Known limitation: Comet only supports `StringType` children natively; `BinaryType` and `ArrayType` inputs fall back to Spark (https://github.com/apache/datafusion-comet/issues/4471). Non-default Spark 4.0 string collations are not propagated (https://github.com/apache/datafusion-comet/issues/2190).
-- [x] reverse
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `Reverse(child) extends UnaryExpression with ImplicitCastInputTypes with NullIntolerant`; `inputTypes = Seq(TypeCollection(StringType, ArrayType))`; `dataType = child.dataType`. For string, calls `UTF8String.reverse()`; for array, reverses element order in-place via `GenericArrayData`.
- - Spark 4.0.1 (audited 2026-05-27): `NullIntolerant` trait replaced by `override def nullIntolerant: Boolean = true`; `inputTypes` widened to `Seq(TypeCollection(StringTypeWithCollation(supportsTrimCollation = true), ArrayType))`. Semantics unchanged for `UTF8_BINARY`.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
- - Known limitation: `Reverse` on an array containing `BinaryType` elements is reported as `Incompatible` and falls back unless explicitly enabled (https://github.com/apache/datafusion-comet/issues/2763).
-- [x] size
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `Size(child, legacySizeOfNull) extends UnaryExpression with ExpectsInputTypes`; `inputTypes = Seq(TypeCollection(ArrayType, MapType)) -> IntegerType`. `legacySizeOfNull=true` returns `-1` for NULL input; `false` returns NULL. Comet routes via `CometSize`, which emits a `CaseWhen(isNotNull(child), size_scalar(child), Literal(legacySizeOfNull))`.
- - Spark 4.0.1 (audited 2026-05-27): byte-for-byte identical to 3.5.8.
- - Spark 4.1.1 (audited 2026-05-27): byte-for-byte identical to 3.5.8.
- - Known limitation: `Size` over `MapType` falls back to Spark (https://github.com/apache/datafusion-comet/issues/4472).
-
-### conditional_funcs
-
-- [x] coalesce
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `Coalesce(children) extends Expression with ComplexTypeMergingExpression`; returns the first non-null child, evaluated left-to-right with short-circuit. Result type is the merged child type. Comet routes via `CometCoalesce`, which serialises as nested `CaseWhen(IsNotNull(c1) -> c1, IsNotNull(c2) -> c2, ..., else cN)` so the native engine preserves the short-circuit semantics.
- - Spark 4.0.1 (audited 2026-05-27): byte-for-byte identical to 3.5.8.
- - Spark 4.1.1 (audited 2026-05-27): byte-for-byte identical to 3.5.8.
-- [x] if
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `If(predicate, trueValue, falseValue) extends ComplexTypeMergingExpression`; standard ternary semantics with short-circuit, predicate must be `BooleanType`. Comet routes via `CometIf` to the native `IfExpr` proto.
- - Spark 4.0.1 (audited 2026-05-27): adds an `override def withNewAlwaysEvaluatedInputs(...)` hook for the new optimizer phase; semantics unchanged. Error messages reformatted (`paramIndex` -> `ordinalNumber`).
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] ifnull
- - Spark 3.4.3 (audited 2026-05-27): registry alias of `Nvl` (`expression[Nvl]("ifnull", setAlias = true)`). Same `RuntimeReplaceable` lowering as `nvl`.
- - Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.
- - Spark 4.0.1 (audited 2026-05-27): identical to 3.4.3.
- - Spark 4.1.1 (audited 2026-05-27): identical to 3.4.3.
-- [ ] nanvl
-- [x] nullif
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): `NullIf(left, right, replacement) extends RuntimeReplaceable with InheritAnalysisRules`; the analyzer rewrites to `If(EqualTo(left, right), Literal(null, left.dataType), left)`. Comet handles via `CometIf` plus `CometEqualTo`.
- - Spark 4.0.1 (audited 2026-05-27): identical to 3.5.8.
- - Spark 4.1.1 (audited 2026-05-27): identical to 3.5.8.
-- [x] nullifzero
-- [x] nvl
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): `Nvl(left, right, replacement) extends RuntimeReplaceable`; analyzer rewrites to `Coalesce(Seq(left, right))`. Comet handles via `CometCoalesce`.
- - Spark 4.0.1 (audited 2026-05-27): identical to 3.5.8.
- - Spark 4.1.1 (audited 2026-05-27): identical to 3.5.8.
-- [x] nvl2
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): `Nvl2(expr1, expr2, expr3, replacement) extends RuntimeReplaceable`; analyzer rewrites to `If(IsNotNull(expr1), expr2, expr3)`. Comet handles via `CometIf`.
- - Spark 4.0.1 (audited 2026-05-27): identical to 3.5.8.
- - Spark 4.1.1 (audited 2026-05-27): identical to 3.5.8.
-- [x] when
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): the `CASE WHEN ... THEN ...` SQL form lowers to `CaseWhen(branches: Seq[(Expression, Expression)], elseValue: Option[Expression])`. Spark evaluates left-to-right with short-circuit; result type is the merged branch type. Comet routes via `CometCaseWhen` to the native `CaseWhen` proto.
- - Spark 4.0.1 (audited 2026-05-27): adds the `withNewAlwaysEvaluatedInputs` optimizer hook; semantics unchanged.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] zeroifnull
-
-### conversion_funcs
-
-- [x] bigint
-- [x] binary
-- [x] boolean
-- [x] cast
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8 modulo `Cast.canUpCast` refactored to delegate to `UpCastRule.canUpCast`.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `Cast(child, dataType, timeZoneId, evalMode)`; eval modes are `LEGACY`, `ANSI`, `TRY`. The legacy `Cast.canCast` matrix and the `Cast.canAnsiCast` matrix decide acceptance per type pair. Comet routes via `CometCast` (`spark/src/main/scala/org/apache/comet/expressions/CometCast.scala`) using a per-source-type support matrix that returns `Compatible`, `Incompatible(reason)`, or `Unsupported(reason)`; literal children are short-circuited to `C [...]
- - Spark 4.0.1 (audited 2026-05-27): `VariantType` added; `StringType` literals replaced with `_: StringType` to accommodate collated strings. `(TimestampType, ByteType|ShortType|IntegerType)` added to `canAnsiCast`. `NullIntolerant` -> `nullIntolerant: Boolean` refactor. New `ToPrettyString.BinaryFormatter` semantics for `Binary -> String` are replicated natively via `spark_binary_formatter`. Numeric-to-numeric matrix unchanged.
- - Spark 4.1.1 (audited 2026-05-27): `TimeType` added; many `TimeType` arms in `canCast`/`canAnsiCast`. Geospatial `GeographyType` / `GeometryType` types added with their own conversion rules. Numeric-to-numeric matrix unchanged.
- - Known divergences and gaps:
- - `CAST(<binary> AS STRING)` uses `unsafe { String::from_utf8_unchecked }` in native code, which is undefined behaviour for non-UTF8 inputs (https://github.com/apache/datafusion-comet/issues/4488).
- - Spark 4.0 collated `StringType` is not explicitly guarded; pattern equality is expected to keep collated-string casts falling back, but there is no test (https://github.com/apache/datafusion-comet/issues/4489; umbrella #2190).
- - Spark 4.1 `TimeType` casts have no explicit `Unsupported` arm; they fall back implicitly but do not appear in the auto-generated compatibility doc (https://github.com/apache/datafusion-comet/issues/4490).
- - `CAST(<map> AS <map>)` falls back to Spark even though native `cast_map_to_map` exists (https://github.com/apache/datafusion-comet/issues/4491).
- - `spark.sql.legacy.castComplexTypesToString.enabled=true` is not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4492).
- - `CAST(<float|double> AS DECIMAL)` rounding may differ from Spark (`Incompatible`, gated by `spark.comet.expression.Cast.allowIncompatible`, tracked at https://github.com/apache/datafusion-comet/issues/1371).
- - Spark registers the type-name conversion functions (`bigint`, `binary`, `boolean`, `date`, `decimal`, `double`, `float`, `int`, `smallint`, `string`, `timestamp`, `tinyint`) as cast aliases. Each lowers to the same `Cast` node, so Comet handles it via the `cast` implementation with the same compatibility profile.
-- [x] date
-- [x] decimal
-- [x] double
-- [x] float
-- [x] int
-- [x] smallint
-- [x] string
-- [x] timestamp
-- [x] tinyint
-
-### csv_funcs
-
-- [ ] from_csv
-- [ ] schema_of_csv
-- [ ] to_csv
-
-### datetime_funcs
-
-- [x] add_months
-- [x] convert_timezone
-- [x] curdate
- - Alias of `current_date`; constant-folded to a literal by Spark's `ComputeCurrentTime` rule before Comet sees the plan.
-- [x] current_date
- - Constant-folded to a literal by Spark's `ComputeCurrentTime` rule before Comet sees the plan.
-- [ ] current_time
-- [x] current_timestamp
- - Constant-folded to a literal by Spark's `ComputeCurrentTime` rule before Comet sees the plan.
-- [x] current_timezone
-- [x] date_add
-- [x] date_diff
-- [x] date_format
-- [x] date_from_unix_date
-- [x] date_part
-- [x] date_sub
-- [x] date_trunc
-- [x] dateadd
-- [x] datediff
-- [x] datepart
-- [x] day
-- [x] dayname
- - Spark 4.0+. Implemented natively: maps a `DateType` value to a fixed US-English abbreviated day name (`DayOfWeek.getDisplayName(TextStyle.SHORT, Locale.US)`), with no session-locale or timezone dependence.
-- [x] dayofmonth
-- [x] dayofweek
-- [x] dayofyear
-- [x] extract
-- [x] from_unixtime
-- [x] from_utc_timestamp
- - Spark 3.4.3 (audited 2026-05-12): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-12): baseline.
- - Spark 4.0.1 (audited 2026-05-12): `inputTypes` widened to `StringTypeWithCollation`; behaviour unchanged for ASCII timezone strings.
- - Known divergence: Comet's native timezone parser does not accept Spark's legacy zone forms (`GMT+1`, `UTC+1`, three-letter abbreviations like `PST`). Such timezones throw a native parse error at execution.
-- [x] hour
-- [x] last_day
-- [x] localtimestamp
-- [x] make_date
-- [ ] make_dt_interval
-- [ ] make_interval
-- [ ] make_time
-- [x] make_timestamp
-- [x] make_timestamp_ltz
- - The 6-argument form rewrites to `MakeTimestamp` and runs via the codegen dispatcher. The 2-argument `(date, time)` form requires the Spark 4.1 TIME type and falls back.
-- [x] make_timestamp_ntz
- - The 6-argument form rewrites to `MakeTimestamp` and runs via the codegen dispatcher. The 2-argument `(date, time)` form requires the Spark 4.1 TIME type and falls back.
-- [ ] make_ym_interval
-- [x] minute
-- [x] month
-- [x] monthname
- - Spark 4.0+. Implemented natively: maps a `DateType` value to a fixed US-English abbreviated month name (`Month.getDisplayName(TextStyle.SHORT, Locale.US)`), with no session-locale or timezone dependence.
-- [x] months_between
-- [x] next_day
-- [x] now
- - Alias of `current_timestamp`; constant-folded to a literal by Spark's `ComputeCurrentTime` rule before Comet sees the plan.
-- [x] quarter
-- [x] second
-- [ ] session_window
-- [ ] time_diff
-- [ ] time_trunc
-- [x] timestamp_micros
-- [x] timestamp_millis
-- [x] timestamp_seconds
-- [x] to_date
- - Rewrites to `Cast` (no format, native) or `Cast(GetTimestamp(...))` (with format, via the codegen dispatcher) before Comet sees the plan.
-- [ ] to_time
-- [x] to_timestamp
- - Rewrites to `Cast` (no format, native) or `GetTimestamp` (with format, via the codegen dispatcher) before Comet sees the plan.
-- [x] to_timestamp_ltz
- - Rewrites to `to_timestamp` with `TimestampType`; same support as `to_timestamp`.
-- [x] to_timestamp_ntz
- - Rewrites to `to_timestamp` with `TimestampNTZType`; same support as `to_timestamp`.
-- [x] to_unix_timestamp
-- [x] to_utc_timestamp
- - Spark 3.4.3 (audited 2026-05-12): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-12): baseline.
- - Spark 4.0.1 (audited 2026-05-12): `inputTypes` widened to `StringTypeWithCollation`; behaviour unchanged for ASCII timezone strings.
- - Known divergence: Comet's native timezone parser does not accept Spark's legacy zone forms (`GMT+1`, `UTC+1`, three-letter abbreviations like `PST`). Such timezones throw a native parse error at execution.
-- [x] trunc
-- [ ] try_make_interval
-- [x] try_make_timestamp
- - Native for valid inputs; returns wrong values for invalid inputs instead of NULL (https://github.com/apache/datafusion-comet/issues/4554).
-- [ ] try_to_date
-- [ ] try_to_time
-- [ ] try_to_timestamp
-- [x] unix_date
-- [x] unix_micros
-- [x] unix_millis
-- [x] unix_seconds
-- [x] unix_timestamp
-- [x] weekday
-- [x] weekofyear
-- [ ] window
-- [ ] window_time
-- [x] year
-
-### generator_funcs
-
-- [x] explode
- - Handled at the operator level as a `GenerateExec` (`CometExplodeExec`), not via the expression serde maps, so it is not auto-detected by the function-registry checkbox logic. Compatible for array inputs; map inputs fall back ([#2837](https://github.com/apache/datafusion-comet/issues/2837)).
-- [x] explode_outer
- - Same `CometExplodeExec` path as `explode`, but the `outer=true` case is `Incompatible` (empty arrays are not preserved as null outputs) and falls back unless `spark.comet.expr.allowIncompatible=true` ([datafusion#19053](https://github.com/apache/datafusion/issues/19053)).
-- [ ] inline
-- [ ] inline_outer
-- [x] posexplode
- - Handled at the operator level as a `GenerateExec` (`CometExplodeExec`), like `explode`. Compatible for array inputs; map inputs fall back ([#2837](https://github.com/apache/datafusion-comet/issues/2837)).
-- [x] posexplode_outer
- - Same `CometExplodeExec` path as `posexplode`, but the `outer=true` case is `Incompatible` and falls back unless `spark.comet.expr.allowIncompatible=true` ([datafusion#19053](https://github.com/apache/datafusion/issues/19053)).
-- [ ] stack
-
-### hash_funcs
-
-- [x] crc32
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `Crc32(child) extends UnaryExpression`; `inputTypes = Seq(BinaryType) -> LongType`. Wired as `CometScalarFunction("crc32")`.
- - Spark 4.0.1 (audited 2026-05-27): semantics unchanged; `NullIntolerant` trait replaced by `nullIntolerant: Boolean` override.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] hash
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `Murmur3Hash(children, seed) extends HashExpression[Int]`; produces a Murmur3 hash with a configurable Int seed and `IntegerType` result. Comet routes via `CometMurmur3Hash` to the native `murmur3_hash` UDF.
- - Spark 4.0.1 (audited 2026-05-27): semantics unchanged; some inner helper refactors only.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
- - Known limitation: `DecimalType` children with precision > 18 fall back because Spark hashes them through Java `BigDecimal`; `TimeType` (Spark 4.0+) is also unsupported. The same limitations apply to `xxhash64`, `sha1`, `sha2` through the shared `HashUtils`.
-- [x] md5
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `Md5(child) extends UnaryExpression`; `inputTypes = Seq(BinaryType) -> StringType`. Wired as `CometScalarFunction("md5")`.
- - Spark 4.0.1 (audited 2026-05-27): semantics unchanged; trait set gains `DefaultStringProducingExpression` and the `nullIntolerant: Boolean` refactor.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] sha
- - Spark 3.4.3 (audited 2026-05-27): registry alias of `Sha1`. Same support as `sha1`.
- - Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.
- - Spark 4.0.1 (audited 2026-05-27): identical to 3.4.3.
- - Spark 4.1.1 (audited 2026-05-27): identical to 3.4.3.
-- [x] sha1
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `Sha1(child) extends UnaryExpression with NullIntolerant`; `inputTypes = Seq(BinaryType) -> StringType`. Comet routes via `CometSha1` to the native `sha1` UDF.
- - Spark 4.0.1 (audited 2026-05-27): trait set gains `DefaultStringProducingExpression` and `NullIntolerant` is replaced by `nullIntolerant: Boolean`. Runtime unchanged.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] sha2
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `Sha2(left, right) extends BinaryExpression`; `inputTypes = Seq(BinaryType, IntegerType) -> StringType`. The `numBits` argument selects SHA-224/256/384/512 (0 is treated as 256); other values return NULL. Comet routes via `CometSha2` to the native `sha2` UDF; non-foldable `numBits` falls back to Spark.
- - Spark 4.0.1 (audited 2026-05-27): trait set gains `DefaultStringProducingExpression` and the `nullIntolerant: Boolean` refactor. Runtime unchanged.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] xxhash64
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `XxHash64(children, seed) extends HashExpression[Long]`; produces an xxHash64 hash with a configurable Long seed and `LongType` result. Comet routes via `CometXxHash64` to the native `xxhash64` UDF.
- - Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-
-### json_funcs
-
-- [x] from_json
- - Partial native support, marked `Incompatible` (requires explicit schema).
-- [x] get_json_object
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `BinaryExpression with ExpectsInputTypes with CodegenFallback`; `inputTypes = Seq(StringType, StringType) -> StringType`. Eval is inline and uses Jackson with `RawStyle` output. Foldable paths are parsed once. Returns NULL for invalid JSON, missing paths, or `JsonProcessingException`.
- - Spark 4.0.1 (audited 2026-05-27): the eval is extracted into a `GetJsonObjectEvaluator` helper (no behaviour change). The trait set now mixes in `DefaultStringProducingExpression`, and `inputTypes` is widened to `StringTypeWithCollation(supportsTrimCollation = true)` for both arguments.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
- - Known incompatibility: Spark accepts single-quoted JSON and unescaped control characters; Comet's native parser (built on `serde_json`) rejects both, so those inputs require `spark.comet.expression.GetJsonObject.allowIncompatible=true` and may still produce different results. Non-default Spark 4.0 string collations are not propagated (https://github.com/apache/datafusion-comet/issues/2190).
-- [ ] json_array_length
-- [ ] json_object_keys
-- [ ] json_tuple
-- [ ] schema_of_json
-- [x] to_json
- - Partial native support; options and map/array inputs fall back.
-
-### lambda_funcs
-
-- [ ] aggregate
-- [ ] array_sort
-- [ ] exists
-- [ ] filter
-- [ ] forall
-- [ ] map_filter
-- [ ] map_zip_with
-- [ ] reduce
-- [ ] transform
-- [ ] transform_keys
-- [ ] transform_values
-- [ ] zip_with
-
-### map_funcs
-
-- [x] element_at
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `ElementAt(left, right, defaultValueOutOfBound, failOnError) extends GetMapValueUtil`; the parser routes `element_at(<array>, ...)` to one overload and `element_at(<map>, ...)` to another. Comet `CometElementAt` only supports `ArrayType` input; `MapType` input falls back.
- - Spark 4.0.1 (audited 2026-05-27): adds `nullIntolerant: Boolean` field; semantics unchanged.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [ ] map
-- [ ] map_concat
-- [x] map_contains_key
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `MapContainsKey(left, right) extends RuntimeReplaceable with InheritAnalysisRules`; the analyzer rewrites to `ArrayContains(MapKeys(left), right)`. Comet routes via `CometMapContainsKey` which emits the equivalent `array_has(map_keys(map), key)`.
- - Spark 4.0.1 (audited 2026-05-27): semantics unchanged; minor trait refactors.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] map_entries
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `MapEntries(child)` returns an array of structs `<key, value>`. Wired to native `map_entries`.
- - Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] map_from_arrays
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `MapFromArrays(left, right) extends BinaryExpression with NullIntolerant`; Spark uses `ArrayBasedMapBuilder` to detect duplicate keys (subject to `spark.sql.mapKeyDedupPolicy`) and rejects null keys with `RuntimeException("Cannot use null as map key")`. Comet `CometMapFromArrays` wraps the inputs in `CaseWhen(IsNotNull(left) AND IsNotNull(right), map(left, right), null)` so NULL-array inputs return NULL rather than triggering the previously [...]
- - Spark 4.0.1 (audited 2026-05-27): semantics unchanged; `NullIntolerant` trait replaced by `nullIntolerant: Boolean`.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] map_from_entries
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `MapFromEntries(child) extends UnaryExpression with NullIntolerant`; expects an array of structs and produces a map. Wired as `CometScalarFunction("map_from_entries")`.
- - Spark 4.0.1 (audited 2026-05-27): semantics unchanged; trait refactor.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
- - Known limitation: input arrays where the struct's key or value type contains `BinaryType` are marked `Incompatible` and fall back unless `spark.comet.expression.MapFromEntries.allowIncompatible=true`.
-- [x] map_keys
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `MapKeys(child)` returns the map's keys as an array. Wired to native `map_keys`.
- - Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] map_values
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `MapValues(child)` returns the map's values as an array. Wired to native `map_values`.
- - Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] str_to_map
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `StringToMap(text, pairDelim, keyValueDelim) extends TernaryExpression`; splits `text` on `pairDelim`, then each pair on `keyValueDelim` (default `","` and `":"`). Uses `ArrayBasedMapBuilder` for duplicate-key handling. Wired as `CometScalarFunction("str_to_map")`.
- - Spark 4.0.1 (audited 2026-05-27): `inputTypes` widened to `StringTypeNonCSAICollation`; uses `CollationAwareUTF8String.splitSQL` with a `collationId`. Runtime unchanged for `UTF8_BINARY`.
- - Spark 4.1.1 (audited 2026-05-27): adds the `legacySplitTruncate` flag (driven by `spark.sql.legacy.truncateForEmptyRegexSplit`) to both `splitSQL` calls (https://github.com/apache/datafusion-comet/issues/4477). The Comet native impl does not honour this flag; behaviour matches the non-legacy default.
-- [x] try_element_at
-
-### math_funcs
-
-- [x] `%`
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `Remainder(left, right, evalMode)` signature identical across versions. Native path uses Rust `spark_modulo` UDF; non-ANSI returns NULL on divide-by-zero, ANSI raises `DIVIDE_BY_ZERO` / `REMAINDER_BY_ZERO`. `CometRemainder` rejects `EvalMode.TRY`, so `try_mod` (Spark 4.0+) falls back to Spark (https://github.com/apache/datafusion-comet/issues/4484).
-- [x] `*`
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `Multiply(left, right, evalMode)` signature identical. Decimal results exceeding `DECIMAL128_MAX_PRECISION` go through `WideDecimalBinaryExpr` (Decimal256 intermediate); smaller decimals and primitives use DataFusion `BinaryExpr`. ANSI integer overflow uses Rust `checked_mul`. Interval multiplication falls back.
-- [x] `+`
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `Add(left, right, evalMode)` with the same Decimal / ANSI plumbing as `*`. `Date + Int8/16/32` dispatches to the Rust `date_add` UDF to work around DataFusion's Date32 + Interval-only kernel.
-- [x] `-`
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `Subtract(left, right, evalMode)` mirrors `+`. `Date - Int8/16/32` uses the Rust `date_sub` UDF.
-- [x] `/`
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `Divide(left, right, evalMode)`. Non-ANSI mode wraps the divisor in `If(EqualTo(right, 0), null, right)` so DataFusion never throws. Decimal output is wrapped in `CheckOverflow(failOnError = ANSI)`; ANSI surfaces `NUMERIC_VALUE_OUT_OF_RANGE`, non-ANSI returns NULL.
-- [x] abs
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `Abs(child, failOnError)` over `NumericType` plus the two interval types. `failOnError` (ANSI) is propagated to the native `abs` UDF, which throws `ARITHMETIC_OVERFLOW` on `Int.MinValue` / `Long.MinValue` / Decimal MIN. `DayTimeIntervalType` and `YearMonthIntervalType` fall back to Spark. Spark 4.0 / 4.1 do the `NullIntolerant` -> `nullIntolerant: Boolean` refactor; behaviour unchanged.
-- [x] acos
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `UnaryMathExpression(math.acos, "ACOS")` unchanged across versions; wired as `CometScalarFunction("acos")` to DataFusion's `acos` UDF. NaN for `|x| > 1`.
-- [x] acosh
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): custom `StrictMath.log(x + sqrt(x*x - 1))` unchanged across versions. NaN for `x < 1`. Routes to DataFusion's `acosh`.
-- [x] asin
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `UnaryMathExpression(math.asin, "ASIN")` unchanged. NaN for `|x| > 1`.
-- [x] asinh
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): special-cases `Double.NegativeInfinity` to avoid `log(NaN)`, otherwise `StrictMath.log(x + sqrt(x*x + 1))`. Identical across versions.
-- [x] atan
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `UnaryMathExpression(math.atan, "ATAN")` unchanged.
-- [x] atan2
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `BinaryMathExpression(math.atan2, "ATAN2")` with both inputs adjusted by `+0.0` to flip `-0.0` to `+0.0`. `CometAtan2` reproduces this by wrapping each child in `Add(child, Literal.default(child.dataType))` before dispatching to DataFusion's `atan2`.
-- [x] atanh
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): custom `0.5 * (log1p(x) - log1p(-x))` (SPARK-28519). NaN for `|x| > 1`, +/-Infinity for `x = +/-1`.
-- [x] bin
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `Bin(child)` over `LongType -> StringType`. Spark 4.x gains `DefaultStringProducingExpression` and the `nullIntolerant: Boolean` refactor; no behaviour change. Routes to datafusion-spark `SparkBin`.
-- [ ] bround
-- [x] cbrt
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): passthrough to DataFusion `cbrt`.
-- [x] ceil
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): one-arg `ceil(expr)` supported (`LongType` / `DoubleType` / `DecimalType` with scale >= 0). Decimal with negative scale falls back at convert time. The two-arg `ceil(expr, scale)` form (`RoundCeil`) is not wired and falls back to Spark.
-- [x] ceiling
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): registry alias for `Ceil`. Same support as `ceil`.
-- [ ] conv
-- [x] cos
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `UnaryMathExpression(math.cos, "COS")` unchanged across versions.
-- [x] cosh
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `UnaryMathExpression(math.cosh, "COSH")` unchanged.
-- [x] cot
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): custom `1 / math.tan(x)`. DataFusion's `cot` is also `1.0 / tan(x)`, so the result matches.
-- [x] csc
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): custom `1 / math.sin(x)`. Routed to datafusion-spark's `SparkCsc` (registered in `jni_api.rs`).
-- [x] degrees
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `UnaryMathExpression(math.toDegrees, "DEGREES")` unchanged across versions.
-- [x] div
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `IntegralDivide(left, right, evalMode)`. Non-decimal operands are cast to `DecimalType(19, 0)`; result is recomputed per `IntegralDivide.resultDecimalType`, wrapped in `CheckOverflow`, then cast to `Long`. ANSI overflow for `Long.MinValue div -1` and decimal-overflow ANSI cases are covered by existing tests.
-- [x] e
- - Foldable; rewritten to a literal by ConstantFolding (like `pi`).
-- [x] exp
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `UnaryMathExpression(StrictMath.exp, "EXP")` unchanged. ULP-level differences vs DataFusion `exp` are possible but unflagged.
-- [x] expm1
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `UnaryMathExpression(StrictMath.expm1, "EXPM1")` unchanged.
-- [x] factorial
- - 3.4.3 (audited 2026-05-15): identical to v3.5.8.
- - 3.5.8 (audited 2026-05-15): canonical reference; `extends UnaryExpression with ImplicitCastInputTypes with NullIntolerant`. Returns NULL for NULL input or values outside `[0, 20]`.
- - 4.0.1 (audited 2026-05-15): `NullIntolerant` trait replaced by `nullIntolerant: Boolean` method override; behavior unchanged.
- - 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] floor
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): mirror of `ceil`. Two-arg `floor(expr, scale)` form (`RoundFloor`) falls back to Spark.
-- [x] greatest
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): NULL-skipping variadic. Wired as `CometScalarFunction("greatest")` to DataFusion's `GreatestFunc`. Comet does not gate input types, so interval inputs and other Spark-only orderings rely on the native UDF accepting them; no explicit fallback path.
-- [x] hex
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): accepts `LongType` / `BinaryType` / `StringType`. Spark 4.x widens `StringType` to `StringTypeWithCollation` and preserves collation in `dataType`; `CometHex` passes `expr.dataType` to native `SparkHex`, which always returns `Utf8` -- collation propagation may diverge on Spark 4.x.
-- [ ] hypot
-- [x] least
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): mirror of `greatest`; same caveats. Spark 4.1.1 adds `contextIndependentFoldable` (no Comet impact).
-- [x] ln
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): registry alias for `Log`. Comet wires through `CometLog` to DataFusion `ln` with a `nullIfNegative` rewrite to match Spark's NULL behaviour for `x <= 0`.
-- [x] log
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): one-arg `log(x)` -> `CometLog` (DataFusion `ln`); two-arg `log(base, x)` -> `CometLogarithm` (custom `spark_log` UDF, returns NULL when `base <= 0` or `x <= 0` to match `Logarithm.nullSafeEval`).
-- [x] log10
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `UnaryLogExpression(StrictMath.log10, "LOG10")`; returns NULL for `x <= 0`. Possible ULP differences from `StrictMath`.
-- [ ] log1p
-- [x] log2
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `UnaryLogExpression(StrictMath.log(x) / StrictMath.log(2), "LOG2")`; returns NULL for `x <= 0`.
-- [x] mod
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): registry alias for `Remainder`. Same support as `%`.
-- [x] negative
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `UnaryMinus(child, failOnError)` -> Rust `NegativeExpr`. ANSI overflow is detected for `Int8/Int16/Int32/Int64` and `IntervalYearMonth/IntervalDayTime`. Float / Double / Decimal cannot overflow on negate. Spark 4.0 `NullIntolerant` -> `nullIntolerant: Boolean` refactor; no impact.
-- [x] pi
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `LeafMathExpression(math.Pi, "PI")`; foldable, so Spark `ConstantFolding` rewrites it to a `Literal` before Comet sees the plan. The `CometScalarFunction("pi")` registration is exercised only when `ConstantFolding` is excluded.
-- [ ] pmod
-- [x] positive
- - Spark 3.4.3, 3.5.8 (audited 2026-05-27): `UnaryPositive(child)` is a regular expression. There is no Comet serde for `UnaryPositive`, so projections containing `+col` silently disable Comet for the projection on 3.4/3.5.
- - Spark 4.0.1, 4.1.1 (audited 2026-05-27): `UnaryPositive` is `RuntimeReplaceable` with `replacement = child`; the optimizer removes it before Comet sees the plan, so the gap is transparent on 4.x.
-- [x] pow
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `Pow(left, right) extends BinaryMathExpression(StrictMath.pow, "POWER")`; routes to DataFusion `pow`. ULP-level differences possible.
-- [x] power
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): registry alias for `Pow`. Same support as `pow`.
-- [x] radians
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `UnaryMathExpression(math.toRadians, "RADIANS")` unchanged across versions.
-- [x] rand
- - See `misc_funcs / rand`.
-- [x] randn
- - See `misc_funcs / randn`.
-- [x] random
-- [ ] randstr
-- [x] rint
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `UnaryMathExpression(math.rint, "ROUND")` with `funcName = "rint"`. Passthrough to DataFusion `rint` (round-half-to-even).
-- [x] round
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): HALF_UP rounding for integer / decimal. Float / Double child types always fall back to Spark because `BigDecimal`-via-`toString` rounding cannot be precisely matched (documented inline in `CometRound`). ANSI `failOnError` is propagated for integer overflow. `BRound` (HALF_EVEN) is not wired.
-- [x] sec
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): custom `1 / math.cos(x)`. Routed to datafusion-spark's `SparkSec`.
-- [x] shiftleft
- - See `bitwise_funcs / <<` (audited in PR #4479). Same support as the operator alias added in 4.0.
-- [x] sign
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): registry alias for `Signum`. Same support as `signum`.
-- [x] signum
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `Signum(child)` over `DoubleType`. Spark also restricts to the two interval types via `inputTypes`; Comet handles only the `Double` case via DataFusion `signum`.
-- [x] sin
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `UnaryMathExpression(math.sin, "SIN")` unchanged.
-- [x] sinh
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `UnaryMathExpression(math.sinh, "SINH")` unchanged.
-- [x] sqrt
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `UnaryMathExpression(math.sqrt, "SQRT")` unchanged.
-- [x] tan
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `UnaryMathExpression(math.tan, "TAN")` unchanged.
-- [x] tanh
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `UnaryMathExpression(math.tanh, "TANH")` unchanged.
-- [x] try_add
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `TryAdd` is `RuntimeReplaceable` and rewrites to `Add(.., EvalMode.TRY)` for numeric inputs (datetime / interval go through `TryEval(Add(.., ANSI))` and fall back). Numeric path uses the Rust `checked_add` UDF, returning NULL on overflow. Decimal goes through `WideDecimalBinaryExpr` with `EvalMode.Try`.
-- [x] try_divide
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `TryDivide` rewrites to `Divide(.., EvalMode.TRY)`. The `nullIfWhenPrimitive` wrapper swaps zero divisors to NULL; integer / float divide uses `checked_div`; decimal uses `decimal_div` + `CheckOverflow(failOnError = false)` returning NULL.
-- [ ] try_mod
-- [x] try_multiply
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): rewrites to `Multiply(.., EvalMode.TRY)`. Integer path uses `checked_mul`; decimal uses `WideDecimalBinaryExpr` with `EvalMode.Try`, returning NULL on overflow.
-- [x] try_subtract
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): rewrites to `Subtract(.., EvalMode.TRY)`. Integer path uses `checked_sub`; decimal uses `WideDecimalBinaryExpr` as needed.
-- [x] unhex
- - Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): `Unhex(child, failOnError)`. Spark 4.x widens input to `StringTypeWithCollation` and wraps the inner call in try/catch; Comet `CometUnhex` forwards `failOnError` to native `spark_unhex` but does not gate on collation.
-- [x] uniform
-- [x] width_bucket
- - Spark 3.5.8 (audited 2026-05-27): introduced; not available in 3.4.3.
- - Spark 4.0.1, 4.1.1 (audited 2026-05-27): same semantics; `NullIntolerant` -> `nullIntolerant: Boolean` refactor.
- - Known limitation: wired via per-version `CometExprShim` rather than a `CometExpressionSerde`, so it bypasses the support-level framework and the auto-generated compatibility doc (https://github.com/apache/datafusion-comet/issues/4485). Native path uses datafusion-spark `SparkWidthBucket`; interval input types are not exercised by Comet tests.
-
-### misc_funcs
-
-- [ ] aes_decrypt
-- [ ] aes_encrypt
-- [ ] approx_top_k_estimate
-- [ ] assert_true
-- [ ] bitmap_and_agg
-- [ ] bitmap_bit_position
-- [ ] bitmap_bucket_number
-- [ ] bitmap_construct_agg
-- [ ] bitmap_count
-- [ ] bitmap_or_agg
-- [x] current_catalog
- - Resolved to a literal by the analyzer (`ReplaceCurrentLike`).
-- [x] current_database
- - Resolved to a literal by the analyzer (`ReplaceCurrentLike`).
-- [x] current_schema
- - Alias of `current_database`; resolved to a literal by the analyzer.
-- [x] current_user
- - Resolved to a literal by the analyzer; same as `user`.
-- [x] equal_null
-- [ ] from_avro
-- [ ] from_protobuf
-- [ ] hll_sketch_estimate
-- [ ] hll_union
-- [ ] input_file_block_length
-- [ ] input_file_block_start
-- [ ] input_file_name
-- [ ] is_variant_null
-- [ ] java_method
-- [x] monotonically_increasing_id
- - Spark 3.4.3 (audited 2026-05-27): byte-for-byte identical to 4.1.1. `MonotonicallyIncreasingID() extends LeafExpression with Stateful`; produces a Long that encodes the partition id in the upper 31 bits and a per-partition row counter in the lower 33 bits. Comet emits an empty `MonotonicallyIncreasingId` proto and the native side produces the same encoding.
- - Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.
- - Spark 4.0.1 (audited 2026-05-27): identical to 3.4.3.
- - Spark 4.1.1 (audited 2026-05-27): identical to 3.4.3.
-- [ ] parse_json
-- [ ] raise_error
-- [x] rand
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `Rand(child, hideSeed) extends RDG` (an `UnaryExpression with ExpectsInputTypes with Nondeterministic with ExpressionWithRandomSeed`); `child` is the seed expression, coerced to `IntegerType` or `LongType` via `ImplicitCastInputTypes`. Uses `XORShiftRandom(seed + partitionIndex)` per partition and returns `nextDouble()` in `[0, 1)`. NULL seed evaluates to `0L` (via `null.asInstanceOf[Long]`).
- - Spark 4.0.1 (audited 2026-05-27): `RDG` is refactored from an `abstract class` into a trait, and `Rand` now extends a new `NondeterministicUnaryRDG` base. `ExpressionWithRandomSeed.expressionToSeed` is hoisted as a shared helper and throws `QueryCompilationErrors.invalidRandomSeedParameter` for non-literal seeds at analysis time. Runtime semantics unchanged.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
- - Comet limitation: the seed argument must be a literal (column-reference seeds are rejected via `getSupportLevel`). Pre-4.0 Spark would otherwise silently fail at runtime; 4.0+ rejects at analysis time before the expression reaches Comet.
-- [x] randn
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): same base as `Rand`; differs only in the eval body (`nextGaussian()` instead of `nextDouble()`), producing values from the standard normal distribution.
- - Spark 4.0.1 (audited 2026-05-27): same refactor as `Rand`; runtime unchanged.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
- - Comet limitation: same as `rand` — the seed argument must be a literal.
-- [ ] reflect
-- [ ] schema_of_avro
-- [ ] schema_of_variant
-- [ ] schema_of_variant_agg
-- [x] session_user
- - Alias of `current_user`; resolved to a literal by the analyzer.
-- [x] spark_partition_id
- - Spark 3.4.3 (audited 2026-05-27): byte-for-byte identical to 4.1.1. `SparkPartitionID() extends LeafExpression with Nondeterministic`; returns the integer index of the partition being processed. Comet emits an empty `SparkPartitionId` proto.
- - Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.
- - Spark 4.0.1 (audited 2026-05-27): identical to 3.4.3.
- - Spark 4.1.1 (audited 2026-05-27): identical to 3.4.3.
-- [ ] st_asbinary
-- [ ] st_geogfromwkb
-- [ ] st_geomfromwkb
-- [ ] st_setsrid
-- [ ] st_srid
-- [ ] theta_difference
-- [ ] theta_intersection
-- [ ] theta_sketch_estimate
-- [ ] theta_union
-- [ ] to_avro
-- [ ] to_protobuf
-- [ ] to_variant_object
-- [ ] try_aes_decrypt
-- [ ] try_parse_json
-- [ ] try_reflect
-- [ ] try_variant_get
-- [x] typeof
- - Foldable; resolved to a literal before Comet sees the plan.
-- [x] user
- - Spark 3.4.3 (audited 2026-05-27): `CurrentUser() extends LeafExpression with Unevaluable`; the analyzer's `ResolveCurrentLike` rule replaces it with a `StringType` literal of the current user name before Comet sees the plan. No Comet serde needed; the literal flows through `CometLiteral`.
- - Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.
- - Spark 4.0.1 (audited 2026-05-27): identical to 3.4.3 except the resulting literal carries the default string collation.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [ ] uuid
-- [ ] variant_get
-- [ ] version
-
-### predicate_funcs
-
-- [x] `!`
- - Spark 3.4.3 (audited 2026-05-27): registry alias of `Not`. Same support as `not`.
- - Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.
- - Spark 4.0.1 (audited 2026-05-27): identical to 3.4.3.
- - Spark 4.1.1 (audited 2026-05-27): identical to 3.4.3.
-- [x] `<`
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `LessThan(left, right) extends BinaryComparison`. Comet routes via `CometLessThan` to the proto's `lt` binary expression.
- - Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] `<=`
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `LessThanOrEqual(left, right) extends BinaryComparison`. Comet routes via `CometLessThanOrEqual` to the proto's `lt_eq` binary expression.
- - Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] `<=>`
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `EqualNullSafe(left, right) extends BinaryComparison`; treats two NULLs as equal and a NULL with a non-NULL as not equal. Comet routes via `CometEqualNullSafe` to the proto's `eq_null_safe` binary expression.
- - Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] `=`
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `EqualTo(left, right) extends BinaryComparison`. Comet routes via `CometEqualTo` to the proto's `eq` binary expression.
- - Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] `==`
- - Spark 3.4.3 (audited 2026-05-27): registry alias of `EqualTo`. Same support as `=`.
- - Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.
- - Spark 4.0.1 (audited 2026-05-27): identical to 3.4.3.
- - Spark 4.1.1 (audited 2026-05-27): identical to 3.4.3.
-- [x] `>`
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `GreaterThan(left, right) extends BinaryComparison`. Comet routes via `CometGreaterThan` to the proto's `gt` binary expression.
- - Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] `>=`
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `GreaterThanOrEqual(left, right) extends BinaryComparison`. Comet routes via `CometGreaterThanOrEqual` to the proto's `gt_eq` binary expression.
- - Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] and
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `And(left, right) extends BinaryOperator with Predicate`; short-circuit left-to-right. Comet routes via `CometAnd` to the proto's `and` binary expression.
- - Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] between
- - Spark 3.4.3 (audited 2026-05-27): the SQL form `expr BETWEEN low AND high` is rewritten at the parser level to `expr >= low AND expr <= high`. Comet sees only the resulting `And(GreaterThanOrEqual, LessThanOrEqual)` and routes via `CometAnd` + `CometGreaterThanOrEqual` + `CometLessThanOrEqual`.
- - Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.
- - Spark 4.0.1 (audited 2026-05-27): identical to 3.4.3.
- - Spark 4.1.1 (audited 2026-05-27): identical to 3.4.3.
-- [x] ilike
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `ILike(left, right, escapeChar) extends RuntimeReplaceable`; the analyzer rewrites to `Like(Lower(left), Lower(right), escapeChar)`. Comet handles via `CometLike` and `CometLower` (case-conversion path, gated by `spark.comet.caseConversion.enabled=false` by default).
- - Spark 4.0.1 (audited 2026-05-27): identical to 3.5.8.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] in
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `In(value, list: Seq[Expression]) extends Predicate`; NULL-sensitive set membership (NULL in any element of the list yields NULL when no match found). Comet routes via `CometIn` to the proto's `In` expression with `negated = false`; `InSet` is rewritten to `In` so the native side can perform its own set-lookup optimization.
- - Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] isnan
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `IsNaN(child) extends UnaryExpression`; returns `BooleanType`; `false` for NULL or non-float types. Comet routes via `CometIsNaN` to the native `isnan` scalar.
- - Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] isnotnull
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `IsNotNull(child) extends UnaryExpression with Predicate`. Comet routes via `CometIsNotNull` to the proto's `is_not_null` unary expression.
- - Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] isnull
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `IsNull(child) extends UnaryExpression with Predicate`. Comet routes via `CometIsNull` to the proto's `is_null` unary expression.
- - Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] like
- - See `string_funcs / like` (audited in PR #4461). `CometLike` only supports the default `\\` escape character.
-- [x] not
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `Not(child) extends UnaryExpression with Predicate`. Comet routes via `CometNot`, which optimizes a few special cases: `Not(EqualTo)` -> proto `neq`, `Not(EqualNullSafe)` -> proto `neq_null_safe`, `Not(In)` -> proto `In(negated = true)`.
- - Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] or
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `Or(left, right) extends BinaryOperator with Predicate`; short-circuit left-to-right. Comet routes via `CometOr` to the proto's `or` binary expression.
- - Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
- - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- [x] regexp
-- [x] regexp_like
-- [x] rlike
- - See `string_funcs / regexp_replace` and the `CometRLike` notes (audited in PR #4461). Uses the Rust `regex` crate, which differs from Java's `Pattern` engine; requires `spark.comet.expression.regexp.allowIncompatible=true`.
-
-### string_funcs
-
-- [x] ascii
-- [ ] base64
-- [x] bit_length
-- [x] btrim
-- [x] char
-- [x] char_length
-- [x] character_length
-- [x] chr
-- [ ] collate
-- [x] collation
-- [x] concat_ws
-- [x] contains
-- [x] decode
-- [ ] elt
-- [ ] encode
-- [x] endswith
-- [ ] find_in_set
-- [ ] format_number
-- [ ] format_string
-- [x] initcap
-- [x] instr
-- [ ] is_valid_utf8
-- [x] lcase
-- [x] left
-- [x] len
-- [x] length
-- [ ] levenshtein
-- [ ] locate
-- [x] lower
-- [x] lpad
-- [x] ltrim
-- [x] luhn_check
-- [ ] make_valid_utf8
-- [ ] mask
-- [x] octet_length
-- [ ] overlay
-- [ ] position
-- [ ] printf
-- [ ] quote
-- [ ] regexp_count
-- [ ] regexp_extract
-- [ ] regexp_extract_all
-- [ ] regexp_instr
-- [x] regexp_replace
-- [ ] regexp_substr
-- [x] repeat
-- [x] replace
-- [x] right
-- [x] rpad
-- [x] rtrim
-- [ ] sentences
-- [ ] soundex
-- [x] space
-- [x] split
-- [ ] split_part
-- [x] startswith
-- [x] substr
-- [x] substring
-- [x] substring_index
-- [x] to_binary
-- [ ] to_char
-- [ ] to_number
-- [ ] to_varchar
-- [x] translate
-- [x] trim
-- [ ] try_to_binary
-- [ ] try_to_number
-- [ ] try_validate_utf8
-- [x] ucase
-- [ ] unbase64
-- [x] upper
-- [ ] validate_utf8
-
-### struct_funcs
-
-- [x] named_struct
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): baseline. `CreateNamedStruct(children)` with `NoThrow`; `children` is `Seq(name1, val1, name2, val2, ...)`. Names must be foldable `StringType` and distinct; non-null. `dataType` is the `StructType` of the resulting fields. Comet routes via `CometCreateNamedStruct`, builds a `CreateNamedStruct` proto, and the native `CreateNamedStruct` expression (in `native/spark-expr/src/struct_funcs/create_named_struct.rs`) emits a `StructArray`.
- - Spark 4.0.1 (audited 2026-05-27): semantics unchanged; an `override def stateful: Boolean = true` flag is added for the optimizer. Parser-level Alias unwrapping (`case (a: Alias, _) => Seq(Literal(a.name), a)`) is transparent to Comet.
- - Spark 4.1.1 (audited 2026-05-27): semantics unchanged; adds `override def contextIndependentFoldable: Boolean = children.forall(_.contextIndependentFoldable)` for the new constant-folding pass; no impact on Comet conversion.
- - Comet limitation: a `CreateNamedStruct` whose names contain duplicates falls back to Spark. (Spark's analyzer also tolerates duplicates at the column-name level, but the proto layer would lose them, so the fallback is the safe choice.)
-- [x] struct
- - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- - Spark 3.5.8 (audited 2026-05-27): the SQL function `struct(a, b, c)` is built by `CreateStruct.create`, which lowers to a `CreateNamedStruct` with synthetic field names `col1`, `col2`, etc. Comet handles both `struct` and `named_struct` via the same `CometCreateNamedStruct` path; the synthetic-name and explicit-name cases behave identically.
- - Spark 4.0.1 (audited 2026-05-27): same lowering; identical to 3.5.8.
- - Spark 4.1.1 (audited 2026-05-27): same lowering; identical to 3.5.8.
-
-### url_funcs
-
-- [x] parse_url
-- [x] try_url_decode
- - 4.0.1, 2026-05-05
-- [x] url_decode
- - 3.4.3, 2026-04-29
- - 3.5.8, 2026-04-29
- - 4.0.1, 2026-04-29
-- [x] url_encode
- - 3.4.3, 2026-04-29
- - 3.5.8, 2026-04-29
- - 4.0.1, 2026-04-29
-
-### window_funcs
-
-- [ ] cume_dist
-- [ ] dense_rank
-- [x] lag
- - Supported via `CometWindowExec` (operator level), not the expression serde.
-- [x] lead
- - Supported via `CometWindowExec` (operator level), not the expression serde.
-- [ ] nth_value
-- [ ] ntile
-- [ ] percent_rank
-- [ ] rank
-- [ ] row_number
-
-### xml_funcs
-
-- [ ] from_xml
-- [ ] schema_of_xml
-- [ ] to_xml
-- [ ] xpath
-- [ ] xpath_boolean
-- [ ] xpath_double
-- [ ] xpath_float
-- [ ] xpath_int
-- [ ] xpath_long
-- [ ] xpath_number
-- [ ] xpath_short
-- [ ] xpath_string
diff --git a/_sources/user-guide/latest/expressions.md.txt b/_sources/user-guide/latest/expressions.md.txt
index 515d6f38dc..217d06936a 100644
--- a/_sources/user-guide/latest/expressions.md.txt
+++ b/_sources/user-guide/latest/expressions.md.txt
@@ -36,12 +36,12 @@ Most expressions can also be disabled with `spark.comet.expression.EXPRNAME.enab
## Status legend
-| Status | Meaning |
-| ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| ✅ Supported | Native or codegen path; compatible with Spark by default. |
-| ⚠️ Supported (caveats) | Works, but may diverge from Spark in some cases: incompatible, flag-gated (`allowIncompatible`), or restricted to certain types. See the [Compatibility Guide](compatibility/index.md). |
-| 🔜 Planned | Intended; tracked by an open issue or pull request. |
-| 💤 Not currently planned | Not on the current roadmap; falls back to Spark and may be reconsidered later. |
+| Status | Meaning |
+| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| ✅ Supported | Comet produces Spark-compatible results by default. Some inputs or forms may fall back to Spark, and any incompatible behavior is opt-in (off by default). |
+| ⚠️ Incorrect by default | Comet runs natively by default but can return results that differ from Spark (a wrong value, or a native error on valid input). See the linked detail on each row. |
+| 🔜 Planned | Intended; tracked by an open issue or pull request. |
+| 💤 Not currently planned | Not on the current roadmap; falls back to Spark and may be reconsidered later. |
## Not currently planned
@@ -70,7 +70,7 @@ The tables below list every Spark built-in expression with its current status.
| `any_value` | ✅ | |
| `approx_count_distinct` | 🔜 | tracking #4098 |
| `array_agg` | 🔜 | Array aggregate (related to `collect_list`, #2524) |
-| `avg` | ⚠️ | Interval types (YearMonth, DayTime) fall back |
+| `avg` | ✅ | Interval types fall back |
| `bit_and` | ✅ | |
| `bit_or` | ✅ | |
| `bit_xor` | ✅ | |
@@ -129,33 +129,33 @@ The tables below list every Spark built-in expression with its current status.
## array_funcs
-| Function | Status | Notes |
-| ----------------- | ------ | ------------------------------------------------------------------------------------------------------------------------- |
-| `array` | ✅ | |
-| `array_append` | ⚠️ | On Spark 4.0+ rewrites to `array_insert`; inherits its incompatibilities |
-| `array_compact` | ✅ | |
-| `array_contains` | ⚠️ | NaN-canonicalization may differ for float/double arrays ([#4481](https://github.com/apache/datafusion-comet/issues/4481)) |
-| `array_distinct` | ⚠️ | NaN/signed-zero canonicalization may differ ([#4481](https://github.com/apache/datafusion-comet/issues/4481)) |
-| `array_except` | ⚠️ | Null handling and ordering may differ; `Incompatible`, flag-gated |
-| `array_insert` | ✅ | |
-| `array_intersect` | ⚠️ | Result element order may differ when right array is longer than left |
-| `array_join` | ⚠️ | Null handling may differ ([#3178](https://github.com/apache/datafusion-comet/issues/3178)); `Incompatible`, flag-gated |
-| `array_max` | ⚠️ | NaN ordering may differ for float/double ([#4482](https://github.com/apache/datafusion-comet/issues/4482)) |
-| `array_min` | ⚠️ | NaN ordering may differ for float/double ([#4482](https://github.com/apache/datafusion-comet/issues/4482)) |
-| `array_position` | ⚠️ | Falls back for binary/struct/map/null element types |
-| `array_prepend` | 🔜 | Sibling of `array_append` |
-| `array_remove` | ✅ | |
-| `array_repeat` | ✅ | |
-| `array_union` | ⚠️ | NaN/signed-zero canonicalization may differ ([#4481](https://github.com/apache/datafusion-comet/issues/4481)) |
-| `arrays_overlap` | ✅ | |
-| `arrays_zip` | ✅ | |
-| `element_at` | ⚠️ | Only `ArrayType` input; `MapType` input falls back |
-| `flatten` | ⚠️ | Falls back for binary/struct/map child element types |
-| `get` | ✅ | |
-| `sequence` | 🔜 | #4538 |
-| `shuffle` | 🔜 | Random array shuffle |
-| `slice` | ✅ | Native (#4149) |
-| `sort_array` | ⚠️ | Incompatible under strict floating-point; falls back for nested struct/null arrays |
+| Function | Status | Notes |
+| ----------------- | ------ | ----------------------------------------------------------------------------------- |
+| `array` | ✅ | |
+| `array_append` | ✅ | |
+| `array_compact` | ✅ | |
+| `array_contains` | ✅ | NaN/signed-zero handling may differ ([details](compatibility/floating-point.md)) |
+| `array_distinct` | ✅ | NaN/signed-zero handling may differ ([details](compatibility/floating-point.md)) |
+| `array_except` | ✅ | Incompatible; falls back by default ([details](compatibility/expressions/array.md)) |
+| `array_insert` | ✅ | |
+| `array_intersect` | ✅ | Incompatible; falls back by default ([details](compatibility/expressions/array.md)) |
+| `array_join` | ✅ | Incompatible; falls back by default ([details](compatibility/expressions/array.md)) |
+| `array_max` | ✅ | NaN ordering may differ ([details](compatibility/floating-point.md)) |
+| `array_min` | ✅ | NaN ordering may differ ([details](compatibility/floating-point.md)) |
+| `array_position` | ✅ | Binary/struct/map/null elements fall back |
+| `array_prepend` | 🔜 | Sibling of `array_append` |
+| `array_remove` | ✅ | |
+| `array_repeat` | ✅ | |
+| `array_union` | ✅ | NaN/signed-zero handling may differ ([details](compatibility/floating-point.md)) |
+| `arrays_overlap` | ✅ | |
+| `arrays_zip` | ✅ | |
+| `element_at` | ✅ | MapType input falls back |
+| `flatten` | ✅ | Binary/struct/map elements fall back |
+| `get` | ✅ | |
+| `sequence` | 🔜 | #4538 |
+| `shuffle` | 🔜 | Random array shuffle |
+| `slice` | ✅ | Native (#4149) |
+| `sort_array` | ✅ | Nested struct/null arrays fall back |
---
@@ -180,13 +180,13 @@ The tables below list every Spark built-in expression with its current status.
## collection_funcs
-| Function | Status | Notes |
-| ------------- | ------ | -------------------------------------------------------------------------------------------------------------------------------- |
-| `array_size` | ⚠️ | Lowers to `size`; accelerated, but returns -1 instead of NULL for NULL input (#4560) |
-| `cardinality` | ⚠️ | Alias for `size`; `MapType` input falls back ([#4472](https://github.com/apache/datafusion-comet/issues/4472)) |
-| `concat` | ⚠️ | Only `StringType` children; `BinaryType`/`ArrayType` fall back ([#4471](https://github.com/apache/datafusion-comet/issues/4471)) |
-| `reverse` | ⚠️ | Array with `BinaryType` elements is `Incompatible`, flag-gated ([#2763](https://github.com/apache/datafusion-comet/issues/2763)) |
-| `size` | ⚠️ | `MapType` input falls back ([#4472](https://github.com/apache/datafusion-comet/issues/4472)) |
+| Function | Status | Notes |
+| ------------- | ------ | ----------------------------------------------------------------------------------------------------------- |
+| `array_size` | ⚠️ | Returns -1 instead of NULL for NULL input ([#4560](https://github.com/apache/datafusion-comet/issues/4560)) |
+| `cardinality` | ✅ | MapType input falls back |
+| `concat` | ✅ | Binary/array children fall back |
+| `reverse` | ✅ | Binary-element arrays fall back (Incompatible) ([details](compatibility/expressions/array.md)) |
+| `size` | ✅ | MapType input falls back |
---
@@ -211,89 +211,89 @@ The tables below list every Spark built-in expression with its current status.
The type-name conversion functions (`bigint`, `binary`, `boolean`, `date`, `decimal`, `double`, `float`, `int`, `smallint`, `string`, `timestamp`, `tinyint`) are SQL aliases for `CAST(... AS <type>)` and share the support and caveats of `cast`.
-| Function | Status | Notes |
-| -------- | ------ | ------------------------------------------------------------------------------------------------------------------ |
-| `cast` | ⚠️ | Many type pairs supported; float-to-decimal rounding may differ; see [Compatibility Guide](compatibility/index.md) |
+| Function | Status | Notes |
+| -------- | ------ | ----------------------------------------------------------------------------------------------- |
+| `cast` | ✅ | Some casts fall back; float-to-decimal is opt-in ([details](compatibility/expressions/cast.md)) |
---
## datetime_funcs
-| Function | Status | Notes |
-| --------------------- | ------ | ------------------------------------------------------------------------------------------------------ |
-| `add_months` | ✅ | |
-| `convert_timezone` | ✅ | |
-| `curdate` | ✅ | Constant-folded to a literal (alias of `current_date`) |
-| `current_date` | ✅ | Constant-folded to a literal before Comet sees the plan |
-| `current_time` | 🔜 | Blocked on Spark 4.1 TIME type support (#4288) |
-| `current_timestamp` | ✅ | Constant-folded to a literal before Comet sees the plan |
-| `current_timezone` | ✅ | |
-| `date_add` | ✅ | |
-| `date_diff` | ✅ | |
-| `date_format` | ✅ | |
-| `date_from_unix_date` | ✅ | |
-| `date_part` | ✅ | |
-| `date_sub` | ✅ | |
-| `date_trunc` | ✅ | |
-| `dateadd` | ✅ | |
-| `datediff` | ✅ | |
-| `datepart` | ✅ | |
-| `day` | ✅ | |
-| `dayname` | 🔜 | #4544 |
-| `dayofmonth` | ✅ | |
-| `dayofweek` | ✅ | |
-| `dayofyear` | ✅ | |
-| `extract` | ✅ | |
-| `from_unixtime` | ✅ | |
-| `from_utc_timestamp` | ⚠️ | Legacy zone forms (`GMT+1`, `PST`) throw a native parse error |
-| `hour` | ✅ | |
-| `last_day` | ✅ | |
-| `localtimestamp` | ✅ | |
-| `make_date` | ✅ | |
-| `make_dt_interval` | 🔜 | #4541 |
-| `make_interval` | 🔜 | Produces legacy CalendarInterval; tracked by #4540 |
-| `make_time` | 🔜 | Spark 4.1 TIME type; tracked by #4288 |
-| `make_timestamp` | ✅ | |
-| `make_timestamp_ltz` | ⚠️ | 6-arg form runs via the codegen dispatcher; 2-arg `(date, time)` form (Spark 4.1 TIME type) falls back |
-| `make_timestamp_ntz` | ⚠️ | 6-arg form runs via the codegen dispatcher; 2-arg `(date, time)` form (Spark 4.1 TIME type) falls back |
-| `make_ym_interval` | 🔜 | #4541 |
-| `minute` | ✅ | |
-| `month` | ✅ | |
-| `monthname` | 🔜 | #4544 |
-| `months_between` | ✅ | |
-| `next_day` | ✅ | |
-| `now` | ✅ | Constant-folded to a literal (alias of `current_timestamp`) |
-| `quarter` | ✅ | |
-| `second` | ✅ | |
-| `session_window` | 🔜 | Time-window grouping; tracked by #4553 |
-| `time_diff` | 🔜 | Spark 4.1 TIME type; tracked by #4288 |
-| `time_trunc` | 🔜 | Spark 4.1 TIME type; tracked by #4288 |
-| `timestamp_micros` | ✅ | |
-| `timestamp_millis` | ✅ | |
-| `timestamp_seconds` | ✅ | |
-| `to_date` | ✅ | Rewrites to `Cast` (or `Cast(GetTimestamp)` with a format) before Comet sees the plan |
-| `to_time` | 🔜 | Spark 4.1 TIME type; tracked by #4288 |
-| `to_timestamp` | ✅ | Rewrites to `Cast` (or `GetTimestamp` with a format) before Comet sees the plan |
-| `to_timestamp_ltz` | ✅ | Rewrites to `to_timestamp` (`TimestampType`) |
-| `to_timestamp_ntz` | ✅ | Rewrites to `to_timestamp` (`TimestampNTZType`) |
-| `to_unix_timestamp` | ✅ | |
-| `to_utc_timestamp` | ⚠️ | Legacy zone forms (`GMT+1`, `PST`) throw a native parse error |
-| `trunc` | ✅ | |
-| `try_make_interval` | 🔜 | Produces legacy CalendarInterval; tracked by #4540 |
-| `try_make_timestamp` | ⚠️ | Runs natively for valid inputs, but returns wrong values for invalid inputs instead of NULL (#4554) |
-| `try_to_date` | 🔜 | Rewrites to `Cast`/`GetTimestamp` but currently falls back; tracked by #4556 |
-| `try_to_time` | 🔜 | Spark 4.1 TIME type; tracked by #4288 |
-| `try_to_timestamp` | 🔜 | Rewrites to `Cast`/`GetTimestamp` but currently falls back; tracked by #4556 |
-| `unix_date` | ✅ | |
-| `unix_micros` | ✅ | |
-| `unix_millis` | ✅ | |
-| `unix_seconds` | ✅ | |
-| `unix_timestamp` | ✅ | |
-| `weekday` | ✅ | |
-| `weekofyear` | ✅ | |
-| `window` | 🔜 | Time-window grouping; tracked by #4553 |
-| `window_time` | 🔜 | Time-window grouping; tracked by #4553 |
-| `year` | ✅ | |
+| Function | Status | Notes |
+| --------------------- | ------ | -------------------------------------------------------------------------------------------------------------------------- |
+| `add_months` | ✅ | |
+| `convert_timezone` | ✅ | |
+| `curdate` | ✅ | Constant-folded to a literal (alias of `current_date`) |
+| `current_date` | ✅ | Constant-folded to a literal before Comet sees the plan |
+| `current_time` | 🔜 | Blocked on Spark 4.1 TIME type support (#4288) |
+| `current_timestamp` | ✅ | Constant-folded to a literal before Comet sees the plan |
+| `current_timezone` | ✅ | |
+| `date_add` | ✅ | |
+| `date_diff` | ✅ | |
+| `date_format` | ✅ | |
+| `date_from_unix_date` | ✅ | |
+| `date_part` | ✅ | |
+| `date_sub` | ✅ | |
+| `date_trunc` | ✅ | |
+| `dateadd` | ✅ | |
+| `datediff` | ✅ | |
+| `datepart` | ✅ | |
+| `day` | ✅ | |
+| `dayname` | 🔜 | #4544 |
+| `dayofmonth` | ✅ | |
+| `dayofweek` | ✅ | |
+| `dayofyear` | ✅ | |
+| `extract` | ✅ | |
+| `from_unixtime` | ✅ | |
+| `from_utc_timestamp` | ✅ | Legacy zone forms fall back (Incompatible) ([details](compatibility/expressions/datetime.md)) |
+| `hour` | ✅ | |
+| `last_day` | ✅ | |
+| `localtimestamp` | ✅ | |
+| `make_date` | ✅ | |
+| `make_dt_interval` | 🔜 | #4541 |
+| `make_interval` | 🔜 | Produces legacy CalendarInterval; tracked by #4540 |
+| `make_time` | 🔜 | Spark 4.1 TIME type; tracked by #4288 |
+| `make_timestamp` | ✅ | |
+| `make_timestamp_ltz` | ✅ | 2-arg TIME form falls back |
+| `make_timestamp_ntz` | ✅ | 2-arg TIME form falls back |
+| `make_ym_interval` | 🔜 | #4541 |
+| `minute` | ✅ | |
+| `month` | ✅ | |
+| `monthname` | 🔜 | #4544 |
+| `months_between` | ✅ | |
+| `next_day` | ✅ | |
+| `now` | ✅ | Constant-folded to a literal (alias of `current_timestamp`) |
+| `quarter` | ✅ | |
+| `second` | ✅ | |
+| `session_window` | 🔜 | Time-window grouping; tracked by #4553 |
+| `time_diff` | 🔜 | Spark 4.1 TIME type; tracked by #4288 |
+| `time_trunc` | 🔜 | Spark 4.1 TIME type; tracked by #4288 |
+| `timestamp_micros` | ✅ | |
+| `timestamp_millis` | ✅ | |
+| `timestamp_seconds` | ✅ | |
+| `to_date` | ✅ | Rewrites to `Cast` (or `Cast(GetTimestamp)` with a format) before Comet sees the plan |
+| `to_time` | 🔜 | Spark 4.1 TIME type; tracked by #4288 |
+| `to_timestamp` | ✅ | Rewrites to `Cast` (or `GetTimestamp` with a format) before Comet sees the plan |
+| `to_timestamp_ltz` | ✅ | Rewrites to `to_timestamp` (`TimestampType`) |
+| `to_timestamp_ntz` | ✅ | Rewrites to `to_timestamp` (`TimestampNTZType`) |
+| `to_unix_timestamp` | ✅ | |
+| `to_utc_timestamp` | ✅ | Legacy zone forms fall back (Incompatible) ([details](compatibility/expressions/datetime.md)) |
+| `trunc` | ✅ | |
+| `try_make_interval` | 🔜 | Produces legacy CalendarInterval; tracked by #4540 |
+| `try_make_timestamp` | ⚠️ | Returns a wrong value instead of NULL for invalid inputs ([#4554](https://github.com/apache/datafusion-comet/issues/4554)) |
+| `try_to_date` | 🔜 | Rewrites to `Cast`/`GetTimestamp` but currently falls back; tracked by #4556 |
+| `try_to_time` | 🔜 | Spark 4.1 TIME type; tracked by #4288 |
+| `try_to_timestamp` | 🔜 | Rewrites to `Cast`/`GetTimestamp` but currently falls back; tracked by #4556 |
+| `unix_date` | ✅ | |
+| `unix_micros` | ✅ | |
+| `unix_millis` | ✅ | |
+| `unix_seconds` | ✅ | |
+| `unix_timestamp` | ✅ | |
+| `weekday` | ✅ | |
+| `weekofyear` | ✅ | |
+| `window` | 🔜 | Time-window grouping; tracked by #4553 |
+| `window_time` | 🔜 | Time-window grouping; tracked by #4553 |
+| `year` | ✅ | |
---
@@ -303,15 +303,15 @@ The type-name conversion functions (`bigint`, `binary`, `boolean`, `date`, `deci
expression-level). The `outer` variants are wired but marked `Incompatible`; they require
`spark.comet.exec.explode.enabled=true` and `allowIncompatible`.
-| Function | Status | Notes |
-| ------------------ | ------ | ---------------------------------------------------- |
-| `explode` | ✅ | via `CometExplodeExec` |
-| `explode_outer` | ⚠️ | `outer=true` incompatible; needs `allowIncompatible` |
-| `inline` | 🔜 | Operator-level generator (like `explode`) |
-| `inline_outer` | 🔜 | Operator-level generator (like `explode`) |
-| `posexplode` | ✅ | via `CometExplodeExec` |
-| `posexplode_outer` | ⚠️ | `outer=true` incompatible; needs `allowIncompatible` |
-| `stack` | 🔜 | Operator-level generator |
+| Function | Status | Notes |
+| ------------------ | ------ | ----------------------------------------------------------------------------------------------------------------------------- |
+| `explode` | ✅ | via `CometExplodeExec` |
+| `explode_outer` | ✅ | outer=true falls back (Incompatible) ([audit](../../contributor-guide/expression-audits/generator_funcs.md#explode_outer)) |
+| `inline` | 🔜 | Operator-level generator (like `explode`) |
+| `inline_outer` | 🔜 | Operator-level generator (like `explode`) |
+| `posexplode` | ✅ | via `CometExplodeExec` |
+| `posexplode_outer` | ✅ | outer=true falls back (Incompatible) ([audit](../../contributor-guide/expression-audits/generator_funcs.md#posexplode_outer)) |
+| `stack` | 🔜 | Operator-level generator |
---
@@ -331,15 +331,15 @@ expression-level). The `outer` variants are wired but marked `Incompatible`; the
## json_funcs
-| Function | Status | Notes |
-| ------------------- | ------ | --------------------------------------------------------------------------------------------------------------------- |
-| `from_json` | ⚠️ | Partial native support (requires explicit schema, marked `Incompatible`); fuller support via codegen dispatch (#4305) |
-| `get_json_object` | ⚠️ | Single-quoted JSON and unescaped control chars require `allowIncompatible` |
-| `json_array_length` | 🔜 | tracking #4098 |
-| `json_object_keys` | 🔜 | [#3161](https://github.com/apache/datafusion-comet/issues/3161) |
-| `json_tuple` | 🔜 | [#3160](https://github.com/apache/datafusion-comet/issues/3160) |
-| `schema_of_json` | 🔜 | [#3163](https://github.com/apache/datafusion-comet/issues/3163) |
-| `to_json` | ⚠️ | Partial native support (options and map/array inputs fall back); fuller support via codegen dispatch (#4305) |
+| Function | Status | Notes |
+| ------------------- | ------ | -------------------------------------------------------------------------------------------------------------------------------- |
+| `from_json` | ✅ | Falls back by default; opt-in via allowIncompatible ([audit](../../contributor-guide/expression-audits/json_funcs.md#from_json)) |
+| `get_json_object` | ✅ | Some inputs need allowIncompatible ([audit](../../contributor-guide/expression-audits/json_funcs.md#get_json_object)) |
+| `json_array_length` | 🔜 | tracking #4098 |
+| `json_object_keys` | 🔜 | [#3161](https://github.com/apache/datafusion-comet/issues/3161) |
+| `json_tuple` | 🔜 | [#3160](https://github.com/apache/datafusion-comet/issues/3160) |
+| `schema_of_json` | 🔜 | [#3163](https://github.com/apache/datafusion-comet/issues/3163) |
+| `to_json` | ✅ | Options and map/array inputs fall back ([audit](../../contributor-guide/expression-audits/json_funcs.md#to_json)) |
---
@@ -366,96 +366,96 @@ All higher-order functions are planned via [#4224](https://github.com/apache/dat
## map_funcs
-| Function | Status | Notes |
-| ------------------ | ------ | ------------------------------------------------------------ |
-| `element_at` | ⚠️ | Only `ArrayType` input; `MapType` input falls back |
-| `map` | 🔜 | Constructs a map |
-| `map_concat` | 🔜 | Concatenates maps |
-| `map_contains_key` | ✅ | |
-| `map_entries` | ✅ | |
-| `map_from_arrays` | ✅ | |
-| `map_from_entries` | ⚠️ | `BinaryType` key/value falls back unless `allowIncompatible` |
-| `map_keys` | ✅ | |
-| `map_values` | ✅ | |
-| `str_to_map` | ✅ | |
-| `try_element_at` | ✅ | Lowers to `element_at`; array input (MapType falls back) |
+| Function | Status | Notes |
+| ------------------ | ------ | -------------------------------------------------------------------------------------------- |
+| `element_at` | ✅ | MapType input falls back |
+| `map` | 🔜 | Constructs a map |
+| `map_concat` | 🔜 | Concatenates maps |
+| `map_contains_key` | ✅ | |
+| `map_entries` | ✅ | |
+| `map_from_arrays` | ✅ | |
+| `map_from_entries` | ✅ | BinaryType key/value falls back (Incompatible) ([details](compatibility/expressions/map.md)) |
+| `map_keys` | ✅ | |
+| `map_values` | ✅ | |
+| `str_to_map` | ✅ | |
+| `try_element_at` | ✅ | Lowers to `element_at`; array input (MapType falls back) |
---
## math_funcs
-| Function | Status | Notes |
-| -------------- | ------ | ------------------------------------------------------------------------------------------------------------------ |
-| `%` | ⚠️ | `try_mod` form (`EvalMode.TRY`) falls back ([#4484](https://github.com/apache/datafusion-comet/issues/4484)) |
-| `*` | ⚠️ | Interval multiplication falls back |
-| `+` | ✅ | |
-| `-` | ✅ | |
-| `/` | ✅ | |
-| `abs` | ⚠️ | Interval types fall back; ANSI overflow for integer min value |
-| `acos` | ✅ | |
-| `acosh` | ✅ | |
-| `asin` | ✅ | |
-| `asinh` | ✅ | |
-| `atan` | ✅ | |
-| `atan2` | ✅ | |
-| `atanh` | ✅ | |
-| `bin` | ✅ | |
-| `bround` | 🔜 | #4538 |
-| `cbrt` | ✅ | |
-| `ceil` | ⚠️ | Two-arg `ceil(expr, scale)` form falls back |
-| `ceiling` | ✅ | |
-| `conv` | 🔜 | #4538 |
-| `cos` | ✅ | |
-| `cosh` | ✅ | |
-| `cot` | ✅ | |
-| `csc` | ✅ | |
-| `degrees` | ✅ | |
-| `div` | ✅ | |
-| `e` | ✅ | Folds to a literal (like `pi`) |
-| `exp` | ✅ | |
-| `expm1` | ✅ | |
-| `factorial` | ✅ | |
-| `floor` | ⚠️ | Two-arg `floor(expr, scale)` form falls back |
-| `greatest` | ✅ | |
-| `hex` | ✅ | |
-| `hypot` | 🔜 | #4538 |
-| `least` | ✅ | |
-| `ln` | ✅ | |
-| `log` | ✅ | |
-| `log10` | ✅ | |
-| `log1p` | 🔜 | #4538 |
-| `log2` | ✅ | |
-| `mod` | ✅ | |
-| `negative` | ✅ | |
-| `pi` | ✅ | |
-| `pmod` | 🔜 | #4538 |
-| `positive` | ✅ | |
-| `pow` | ✅ | |
-| `power` | ✅ | |
-| `radians` | ✅ | |
-| `rand` | ✅ | |
-| `randn` | ✅ | |
-| `random` | ✅ | Alias for `rand` (Spark 4.0+); seed must be a literal |
-| `randstr` | 🔜 | Random string (Spark 4.0+) |
-| `rint` | ✅ | |
-| `round` | ⚠️ | Float/Double inputs always fall back; integer/decimal HALF_UP supported |
-| `sec` | ✅ | |
-| `shiftleft` | ✅ | |
-| `sign` | ✅ | |
-| `signum` | ✅ | |
-| `sin` | ✅ | |
-| `sinh` | ✅ | |
-| `sqrt` | ✅ | |
-| `tan` | ✅ | |
-| `tanh` | ✅ | |
-| `try_add` | ⚠️ | Datetime/interval form falls back; numeric form supported |
-| `try_divide` | ✅ | |
-| `try_mod` | 🔜 | Lowers to `Remainder` with TRY eval mode, which falls back (#4484) |
-| `try_multiply` | ✅ | |
-| `try_subtract` | ✅ | |
-| `unhex` | ✅ | |
-| `uniform` | ✅ | Constant-folded; literal arguments only (Spark 4.0+) |
-| `width_bucket` | ⚠️ | Wired via shim, bypasses support-level framework ([#4485](https://github.com/apache/datafusion-comet/issues/4485)) |
+| Function | Status | Notes |
+| -------------- | ------ | ------------------------------------------------------------------ |
+| `%` | ✅ | try_mod (TRY mode) falls back |
+| `*` | ✅ | Interval multiplication falls back |
+| `+` | ✅ | |
+| `-` | ✅ | |
+| `/` | ✅ | |
+| `abs` | ✅ | Interval types fall back |
+| `acos` | ✅ | |
+| `acosh` | ✅ | |
+| `asin` | ✅ | |
+| `asinh` | ✅ | |
+| `atan` | ✅ | |
+| `atan2` | ✅ | |
+| `atanh` | ✅ | |
+| `bin` | ✅ | |
+| `bround` | 🔜 | #4538 |
+| `cbrt` | ✅ | |
+| `ceil` | ✅ | Two-arg form falls back |
+| `ceiling` | ✅ | |
+| `conv` | 🔜 | #4538 |
+| `cos` | ✅ | |
+| `cosh` | ✅ | |
+| `cot` | ✅ | |
+| `csc` | ✅ | |
+| `degrees` | ✅ | |
+| `div` | ✅ | |
+| `e` | ✅ | Folds to a literal (like `pi`) |
+| `exp` | ✅ | |
+| `expm1` | ✅ | |
+| `factorial` | ✅ | |
+| `floor` | ✅ | Two-arg form falls back |
+| `greatest` | ✅ | |
+| `hex` | ✅ | |
+| `hypot` | 🔜 | #4538 |
+| `least` | ✅ | |
+| `ln` | ✅ | |
+| `log` | ✅ | |
+| `log10` | ✅ | |
+| `log1p` | 🔜 | #4538 |
+| `log2` | ✅ | |
+| `mod` | ✅ | |
+| `negative` | ✅ | |
+| `pi` | ✅ | |
+| `pmod` | 🔜 | #4538 |
+| `positive` | ✅ | |
+| `pow` | ✅ | |
+| `power` | ✅ | |
+| `radians` | ✅ | |
+| `rand` | ✅ | |
+| `randn` | ✅ | |
+| `random` | ✅ | Alias for `rand` (Spark 4.0+); seed must be a literal |
+| `randstr` | 🔜 | Random string (Spark 4.0+) |
+| `rint` | ✅ | |
+| `round` | ✅ | Float/double inputs fall back |
+| `sec` | ✅ | |
+| `shiftleft` | ✅ | |
+| `sign` | ✅ | |
+| `signum` | ✅ | |
+| `sin` | ✅ | |
+| `sinh` | ✅ | |
+| `sqrt` | ✅ | |
+| `tan` | ✅ | |
+| `tanh` | ✅ | |
+| `try_add` | ✅ | Datetime/interval form falls back |
+| `try_divide` | ✅ | |
+| `try_mod` | 🔜 | Lowers to `Remainder` with TRY eval mode, which falls back (#4484) |
+| `try_multiply` | ✅ | |
+| `try_subtract` | ✅ | |
+| `unhex` | ✅ | |
+| `uniform` | ✅ | Constant-folded; literal arguments only (Spark 4.0+) |
+| `width_bucket` | ✅ | |
---
@@ -494,29 +494,29 @@ All higher-order functions are planned via [#4224](https://github.com/apache/dat
## predicate_funcs
-| Function | Status | Notes |
-| ------------- | ------ | --------------------------------------------------------------------------------------------- |
-| `!` | ✅ | |
-| `<` | ✅ | |
-| `<=` | ✅ | |
-| `<=>` | ✅ | |
-| `=` | ✅ | |
-| `==` | ✅ | |
-| `>` | ✅ | |
-| `>=` | ✅ | |
-| `and` | ✅ | |
-| `between` | ✅ | |
-| `ilike` | ✅ | |
-| `in` | ✅ | |
-| `isnan` | ✅ | |
-| `isnotnull` | ✅ | |
-| `isnull` | ✅ | |
-| `like` | ✅ | |
-| `not` | ✅ | |
-| `or` | ✅ | |
-| `regexp` | ⚠️ | Alias for `rlike`; uses Rust `regex` crate, requires `allowIncompatible` |
-| `regexp_like` | ⚠️ | Alias for `rlike`; uses Rust `regex` crate, requires `allowIncompatible` |
-| `rlike` | ⚠️ | Uses Rust `regex` crate; requires `allowIncompatible`; results may differ from Java `Pattern` |
+| Function | Status | Notes |
+| ------------- | ------ | --------------------------------------------------------------------------------------- |
+| `!` | ✅ | |
+| `<` | ✅ | |
+| `<=` | ✅ | |
+| `<=>` | ✅ | |
+| `=` | ✅ | |
+| `==` | ✅ | |
+| `>` | ✅ | |
+| `>=` | ✅ | |
+| `and` | ✅ | |
+| `between` | ✅ | |
+| `ilike` | ✅ | |
+| `in` | ✅ | |
+| `isnan` | ✅ | |
+| `isnotnull` | ✅ | |
+| `isnull` | ✅ | |
+| `like` | ✅ | |
+| `not` | ✅ | |
+| `or` | ✅ | |
+| `regexp` | ✅ | Falls back by default; opt-in via allowIncompatible ([details](compatibility/regex.md)) |
+| `regexp_like` | ✅ | Falls back by default; opt-in via allowIncompatible ([details](compatibility/regex.md)) |
+| `rlike` | ✅ | Falls back by default; opt-in via allowIncompatible ([details](compatibility/regex.md)) |
---
@@ -579,7 +579,7 @@ All higher-order functions are planned via [#4224](https://github.com/apache/dat
| `substr` | ✅ | |
| `substring` | ✅ | |
| `substring_index` | ✅ | |
-| `to_binary` | ⚠️ | Only the hex format is accelerated (lowers to `unhex`); UTF-8/base64 fall back |
+| `to_binary` | ✅ | Hex form accelerated; other formats fall back |
| `to_char` | 🔜 | #4538 |
| `to_number` | 🔜 | #4538 |
| `to_varchar` | 🔜 | #4538 |
@@ -595,10 +595,10 @@ All higher-order functions are planned via [#4224](https://github.com/apache/dat
## struct_funcs
-| Function | Status | Notes |
-| -------------- | ------ | ---------------------------------------- |
-| `named_struct` | ⚠️ | Duplicate field names fall back to Spark |
-| `struct` | ✅ | |
+| Function | Status | Notes |
+| -------------- | ------ | ------------------------------- |
+| `named_struct` | ✅ | Duplicate field names fall back |
+| `struct` | ✅ | |
---
@@ -651,4 +651,4 @@ This list is illustrative, not exhaustive: the per-function tables are not the c
## See also
- [Comet Compatibility Guide](compatibility/index.md) - known incompatibilities and edge cases for ⚠️ expressions.
-- [Spark Expressions Support (contributor guide)](../../contributor-guide/spark_expressions_support.md) - per-version (Spark 3.4 / 3.5 / 4.0 / 4.1) audit notes for each expression.
+- [Expression Audits (contributor guide)](../../contributor-guide/expression-audits/index.md) - per-version (Spark 3.4 / 3.5 / 4.0 / 4.1) audit notes for audited expressions.
diff --git a/contributor-guide/adding_a_new_expression.html b/contributor-guide/adding_a_new_expression.html
index 1102654fe3..463b13a5cb 100644
--- a/contributor-guide/adding_a_new_expression.html
+++ b/contributor-guide/adding_a_new_expression.html
@@ -433,7 +433,26 @@ under the License.
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Reference</span></p>
<ul class="nav bd-sidenav">
-<li class="toctree-l1"><a class="reference internal" href="spark_expressions_support.html">Supported Spark Expressions</a></li>
+<li class="toctree-l1 has-children"><a class="reference internal" href="expression-audits/index.html">Expression Audits</a><details><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/agg_funcs.html">agg_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/array_funcs.html">array_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/bitwise_funcs.html">bitwise_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/collection_funcs.html">collection_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/conditional_funcs.html">conditional_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/conversion_funcs.html">conversion_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/datetime_funcs.html">datetime_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/generator_funcs.html">generator_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/hash_funcs.html">hash_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/json_funcs.html">json_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/map_funcs.html">map_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/math_funcs.html">math_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/misc_funcs.html">misc_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/predicate_funcs.html">predicate_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/struct_funcs.html">struct_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/url_funcs.html">url_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/window_funcs.html">window_funcs</a></li>
+</ul>
+</details></li>
<li class="toctree-l1"><a class="reference internal" href="spark_configs_support.html">Supported Spark Configurations</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Project Mechanics</span></p>
@@ -528,9 +547,8 @@ under the License.
<p>Before you start, have a look through <a class="reference external" href="https://docs.google.com/presentation/d/1H0fF2MOkkBK8fPBlnqK6LejUeLcVD917JhVWfp3mb8A/edit#slide=id.p">these slides</a> as they provide a conceptual overview. And a video of a presentation on those slides is available <a class="reference external" href="https://drive.google.com/file/d/1POU4lFAZfYwZR8zV1X2eoLiAmc1GDtSP/view?usp=sharing">here</a>.</p>
<section id="finding-an-expression-to-add">
<h2>Finding an Expression to Add<a class="headerlink" href="#finding-an-expression-to-add" title="Link to this heading">#</a></h2>
-<p>You may have a specific expression in mind that you’d like to add, but if not, you can review the <a class="reference internal" href="spark_expressions_support.html"><span class="std std-doc">expression coverage document</span></a> to see which expressions are not yet supported.</p>
-<p>When you add or change an expression, update <strong>both</strong> the coverage checklist
-(<code class="docutils literal notranslate"><span class="pre">spark_expressions_support.md</span></code>) and the user-facing status in
+<p>You may have a specific expression in mind that you’d like to add, but if not, you can review <a class="reference internal" href="../user-guide/latest/expressions.html"><span class="std std-doc">Supported Spark Expressions</span></a> in the user guide to see which expressions are not yet supported. For deep-dive audit notes on expressions that are already supported, see the <a class="reference internal" href="expression-audits/index.html"><span class="std std-doc">Expression Audits</s [...]
+<p>When you add or change an expression, update its status in
<a class="reference internal" href="../user-guide/latest/expressions.html"><span class="std std-doc">Supported Spark Expressions</span></a>.</p>
</section>
<section id="implementing-the-expression">
diff --git a/contributor-guide/adding_a_new_operator.html b/contributor-guide/adding_a_new_operator.html
index 3dc7d15ed3..72238f2490 100644
--- a/contributor-guide/adding_a_new_operator.html
+++ b/contributor-guide/adding_a_new_operator.html
@@ -433,7 +433,26 @@ under the License.
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Reference</span></p>
<ul class="nav bd-sidenav">
-<li class="toctree-l1"><a class="reference internal" href="spark_expressions_support.html">Supported Spark Expressions</a></li>
+<li class="toctree-l1 has-children"><a class="reference internal" href="expression-audits/index.html">Expression Audits</a><details><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/agg_funcs.html">agg_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/array_funcs.html">array_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/bitwise_funcs.html">bitwise_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/collection_funcs.html">collection_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/conditional_funcs.html">conditional_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/conversion_funcs.html">conversion_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/datetime_funcs.html">datetime_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/generator_funcs.html">generator_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/hash_funcs.html">hash_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/json_funcs.html">json_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/map_funcs.html">map_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/math_funcs.html">math_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/misc_funcs.html">misc_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/predicate_funcs.html">predicate_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/struct_funcs.html">struct_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/url_funcs.html">url_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/window_funcs.html">window_funcs</a></li>
+</ul>
+</details></li>
<li class="toctree-l1"><a class="reference internal" href="spark_configs_support.html">Supported Spark Configurations</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Project Mechanics</span></p>
diff --git a/contributor-guide/adding_a_new_spark_version.html b/contributor-guide/adding_a_new_spark_version.html
index afab82c958..34f0d57c3c 100644
--- a/contributor-guide/adding_a_new_spark_version.html
+++ b/contributor-guide/adding_a_new_spark_version.html
@@ -433,7 +433,26 @@ under the License.
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Reference</span></p>
<ul class="nav bd-sidenav">
-<li class="toctree-l1"><a class="reference internal" href="spark_expressions_support.html">Supported Spark Expressions</a></li>
+<li class="toctree-l1 has-children"><a class="reference internal" href="expression-audits/index.html">Expression Audits</a><details><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/agg_funcs.html">agg_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/array_funcs.html">array_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/bitwise_funcs.html">bitwise_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/collection_funcs.html">collection_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/conditional_funcs.html">conditional_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/conversion_funcs.html">conversion_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/datetime_funcs.html">datetime_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/generator_funcs.html">generator_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/hash_funcs.html">hash_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/json_funcs.html">json_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/map_funcs.html">map_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/math_funcs.html">math_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/misc_funcs.html">misc_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/predicate_funcs.html">predicate_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/struct_funcs.html">struct_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/url_funcs.html">url_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/window_funcs.html">window_funcs</a></li>
+</ul>
+</details></li>
<li class="toctree-l1"><a class="reference internal" href="spark_configs_support.html">Supported Spark Configurations</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Project Mechanics</span></p>
diff --git a/contributor-guide/benchmark-results/tpc-ds.html b/contributor-guide/benchmark-results/tpc-ds.html
index 4d5bf9b1c4..5f9e8b80b0 100644
--- a/contributor-guide/benchmark-results/tpc-ds.html
+++ b/contributor-guide/benchmark-results/tpc-ds.html
@@ -433,7 +433,26 @@ under the License.
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Reference</span></p>
<ul class="nav bd-sidenav">
-<li class="toctree-l1"><a class="reference internal" href="../spark_expressions_support.html">Supported Spark Expressions</a></li>
+<li class="toctree-l1 has-children"><a class="reference internal" href="../expression-audits/index.html">Expression Audits</a><details><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul>
+<li class="toctree-l2"><a class="reference internal" href="../expression-audits/agg_funcs.html">agg_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../expression-audits/array_funcs.html">array_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../expression-audits/bitwise_funcs.html">bitwise_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../expression-audits/collection_funcs.html">collection_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../expression-audits/conditional_funcs.html">conditional_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../expression-audits/conversion_funcs.html">conversion_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../expression-audits/datetime_funcs.html">datetime_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../expression-audits/generator_funcs.html">generator_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../expression-audits/hash_funcs.html">hash_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../expression-audits/json_funcs.html">json_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../expression-audits/map_funcs.html">map_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../expression-audits/math_funcs.html">math_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../expression-audits/misc_funcs.html">misc_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../expression-audits/predicate_funcs.html">predicate_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../expression-audits/struct_funcs.html">struct_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../expression-audits/url_funcs.html">url_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../expression-audits/window_funcs.html">window_funcs</a></li>
+</ul>
+</details></li>
<li class="toctree-l1"><a class="reference internal" href="../spark_configs_support.html">Supported Spark Configurations</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Project Mechanics</span></p>
diff --git a/contributor-guide/benchmark-results/tpc-h.html b/contributor-guide/benchmark-results/tpc-h.html
index b75aa4ff2a..00108cd1e0 100644
--- a/contributor-guide/benchmark-results/tpc-h.html
+++ b/contributor-guide/benchmark-results/tpc-h.html
@@ -433,7 +433,26 @@ under the License.
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Reference</span></p>
<ul class="nav bd-sidenav">
-<li class="toctree-l1"><a class="reference internal" href="../spark_expressions_support.html">Supported Spark Expressions</a></li>
+<li class="toctree-l1 has-children"><a class="reference internal" href="../expression-audits/index.html">Expression Audits</a><details><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul>
+<li class="toctree-l2"><a class="reference internal" href="../expression-audits/agg_funcs.html">agg_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../expression-audits/array_funcs.html">array_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../expression-audits/bitwise_funcs.html">bitwise_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../expression-audits/collection_funcs.html">collection_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../expression-audits/conditional_funcs.html">conditional_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../expression-audits/conversion_funcs.html">conversion_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../expression-audits/datetime_funcs.html">datetime_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../expression-audits/generator_funcs.html">generator_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../expression-audits/hash_funcs.html">hash_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../expression-audits/json_funcs.html">json_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../expression-audits/map_funcs.html">map_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../expression-audits/math_funcs.html">math_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../expression-audits/misc_funcs.html">misc_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../expression-audits/predicate_funcs.html">predicate_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../expression-audits/struct_funcs.html">struct_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../expression-audits/url_funcs.html">url_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../expression-audits/window_funcs.html">window_funcs</a></li>
+</ul>
+</details></li>
<li class="toctree-l1"><a class="reference internal" href="../spark_configs_support.html">Supported Spark Configurations</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Project Mechanics</span></p>
diff --git a/contributor-guide/benchmarking.html b/contributor-guide/benchmarking.html
index 12b6e298cf..3fe0cb15b3 100644
--- a/contributor-guide/benchmarking.html
+++ b/contributor-guide/benchmarking.html
@@ -433,7 +433,26 @@ under the License.
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Reference</span></p>
<ul class="nav bd-sidenav">
-<li class="toctree-l1"><a class="reference internal" href="spark_expressions_support.html">Supported Spark Expressions</a></li>
+<li class="toctree-l1 has-children"><a class="reference internal" href="expression-audits/index.html">Expression Audits</a><details><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/agg_funcs.html">agg_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/array_funcs.html">array_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/bitwise_funcs.html">bitwise_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/collection_funcs.html">collection_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/conditional_funcs.html">conditional_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/conversion_funcs.html">conversion_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/datetime_funcs.html">datetime_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/generator_funcs.html">generator_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/hash_funcs.html">hash_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/json_funcs.html">json_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/map_funcs.html">map_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/math_funcs.html">math_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/misc_funcs.html">misc_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/predicate_funcs.html">predicate_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/struct_funcs.html">struct_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/url_funcs.html">url_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/window_funcs.html">window_funcs</a></li>
+</ul>
+</details></li>
<li class="toctree-l1"><a class="reference internal" href="spark_configs_support.html">Supported Spark Configurations</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Project Mechanics</span></p>
diff --git a/contributor-guide/benchmarking_aws_ec2.html b/contributor-guide/benchmarking_aws_ec2.html
index 5009d150cb..d6f30736b0 100644
--- a/contributor-guide/benchmarking_aws_ec2.html
+++ b/contributor-guide/benchmarking_aws_ec2.html
@@ -433,7 +433,26 @@ under the License.
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Reference</span></p>
<ul class="nav bd-sidenav">
-<li class="toctree-l1"><a class="reference internal" href="spark_expressions_support.html">Supported Spark Expressions</a></li>
+<li class="toctree-l1 has-children"><a class="reference internal" href="expression-audits/index.html">Expression Audits</a><details><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/agg_funcs.html">agg_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/array_funcs.html">array_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/bitwise_funcs.html">bitwise_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/collection_funcs.html">collection_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/conditional_funcs.html">conditional_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/conversion_funcs.html">conversion_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/datetime_funcs.html">datetime_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/generator_funcs.html">generator_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/hash_funcs.html">hash_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/json_funcs.html">json_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/map_funcs.html">map_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/math_funcs.html">math_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/misc_funcs.html">misc_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/predicate_funcs.html">predicate_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/struct_funcs.html">struct_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/url_funcs.html">url_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/window_funcs.html">window_funcs</a></li>
+</ul>
+</details></li>
<li class="toctree-l1"><a class="reference internal" href="spark_configs_support.html">Supported Spark Configurations</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Project Mechanics</span></p>
diff --git a/contributor-guide/benchmarking_macos.html b/contributor-guide/benchmarking_macos.html
index 627a554691..e3ceb45ecd 100644
--- a/contributor-guide/benchmarking_macos.html
+++ b/contributor-guide/benchmarking_macos.html
@@ -433,7 +433,26 @@ under the License.
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Reference</span></p>
<ul class="nav bd-sidenav">
-<li class="toctree-l1"><a class="reference internal" href="spark_expressions_support.html">Supported Spark Expressions</a></li>
+<li class="toctree-l1 has-children"><a class="reference internal" href="expression-audits/index.html">Expression Audits</a><details><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/agg_funcs.html">agg_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/array_funcs.html">array_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/bitwise_funcs.html">bitwise_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/collection_funcs.html">collection_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/conditional_funcs.html">conditional_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/conversion_funcs.html">conversion_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/datetime_funcs.html">datetime_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/generator_funcs.html">generator_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/hash_funcs.html">hash_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/json_funcs.html">json_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/map_funcs.html">map_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/math_funcs.html">math_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/misc_funcs.html">misc_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/predicate_funcs.html">predicate_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/struct_funcs.html">struct_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/url_funcs.html">url_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/window_funcs.html">window_funcs</a></li>
+</ul>
+</details></li>
<li class="toctree-l1"><a class="reference internal" href="spark_configs_support.html">Supported Spark Configurations</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Project Mechanics</span></p>
diff --git a/contributor-guide/benchmarking_spark_sql_perf.html b/contributor-guide/benchmarking_spark_sql_perf.html
index 989f18aef8..8622410fc2 100644
--- a/contributor-guide/benchmarking_spark_sql_perf.html
+++ b/contributor-guide/benchmarking_spark_sql_perf.html
@@ -433,7 +433,26 @@ under the License.
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Reference</span></p>
<ul class="nav bd-sidenav">
-<li class="toctree-l1"><a class="reference internal" href="spark_expressions_support.html">Supported Spark Expressions</a></li>
+<li class="toctree-l1 has-children"><a class="reference internal" href="expression-audits/index.html">Expression Audits</a><details><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/agg_funcs.html">agg_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/array_funcs.html">array_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/bitwise_funcs.html">bitwise_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/collection_funcs.html">collection_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/conditional_funcs.html">conditional_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/conversion_funcs.html">conversion_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/datetime_funcs.html">datetime_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/generator_funcs.html">generator_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/hash_funcs.html">hash_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/json_funcs.html">json_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/map_funcs.html">map_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/math_funcs.html">math_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/misc_funcs.html">misc_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/predicate_funcs.html">predicate_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/struct_funcs.html">struct_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/url_funcs.html">url_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/window_funcs.html">window_funcs</a></li>
+</ul>
+</details></li>
<li class="toctree-l1"><a class="reference internal" href="spark_configs_support.html">Supported Spark Configurations</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Project Mechanics</span></p>
diff --git a/contributor-guide/bug_triage.html b/contributor-guide/bug_triage.html
index 4288959183..3357da1c88 100644
--- a/contributor-guide/bug_triage.html
+++ b/contributor-guide/bug_triage.html
@@ -433,7 +433,26 @@ under the License.
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Reference</span></p>
<ul class="nav bd-sidenav">
-<li class="toctree-l1"><a class="reference internal" href="spark_expressions_support.html">Supported Spark Expressions</a></li>
+<li class="toctree-l1 has-children"><a class="reference internal" href="expression-audits/index.html">Expression Audits</a><details><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/agg_funcs.html">agg_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/array_funcs.html">array_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/bitwise_funcs.html">bitwise_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/collection_funcs.html">collection_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/conditional_funcs.html">conditional_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/conversion_funcs.html">conversion_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/datetime_funcs.html">datetime_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/generator_funcs.html">generator_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/hash_funcs.html">hash_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/json_funcs.html">json_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/map_funcs.html">map_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/math_funcs.html">math_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/misc_funcs.html">misc_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/predicate_funcs.html">predicate_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/struct_funcs.html">struct_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/url_funcs.html">url_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/window_funcs.html">window_funcs</a></li>
+</ul>
+</details></li>
<li class="toctree-l1"><a class="reference internal" href="spark_configs_support.html">Supported Spark Configurations</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Project Mechanics</span></p>
diff --git a/contributor-guide/contributing.html b/contributor-guide/contributing.html
index eb95b9e38c..41b147a49d 100644
--- a/contributor-guide/contributing.html
+++ b/contributor-guide/contributing.html
@@ -433,7 +433,26 @@ under the License.
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Reference</span></p>
<ul class="nav bd-sidenav">
-<li class="toctree-l1"><a class="reference internal" href="spark_expressions_support.html">Supported Spark Expressions</a></li>
+<li class="toctree-l1 has-children"><a class="reference internal" href="expression-audits/index.html">Expression Audits</a><details><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/agg_funcs.html">agg_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/array_funcs.html">array_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/bitwise_funcs.html">bitwise_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/collection_funcs.html">collection_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/conditional_funcs.html">conditional_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/conversion_funcs.html">conversion_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/datetime_funcs.html">datetime_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/generator_funcs.html">generator_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/hash_funcs.html">hash_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/json_funcs.html">json_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/map_funcs.html">map_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/math_funcs.html">math_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/misc_funcs.html">misc_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/predicate_funcs.html">predicate_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/struct_funcs.html">struct_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/url_funcs.html">url_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/window_funcs.html">window_funcs</a></li>
+</ul>
+</details></li>
<li class="toctree-l1"><a class="reference internal" href="spark_configs_support.html">Supported Spark Configurations</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Project Mechanics</span></p>
diff --git a/contributor-guide/debugging.html b/contributor-guide/debugging.html
index c14f17daec..1d2c7824c8 100644
--- a/contributor-guide/debugging.html
+++ b/contributor-guide/debugging.html
@@ -433,7 +433,26 @@ under the License.
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Reference</span></p>
<ul class="nav bd-sidenav">
-<li class="toctree-l1"><a class="reference internal" href="spark_expressions_support.html">Supported Spark Expressions</a></li>
+<li class="toctree-l1 has-children"><a class="reference internal" href="expression-audits/index.html">Expression Audits</a><details><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/agg_funcs.html">agg_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/array_funcs.html">array_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/bitwise_funcs.html">bitwise_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/collection_funcs.html">collection_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/conditional_funcs.html">conditional_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/conversion_funcs.html">conversion_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/datetime_funcs.html">datetime_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/generator_funcs.html">generator_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/hash_funcs.html">hash_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/json_funcs.html">json_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/map_funcs.html">map_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/math_funcs.html">math_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/misc_funcs.html">misc_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/predicate_funcs.html">predicate_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/struct_funcs.html">struct_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/url_funcs.html">url_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/window_funcs.html">window_funcs</a></li>
+</ul>
+</details></li>
<li class="toctree-l1"><a class="reference internal" href="spark_configs_support.html">Supported Spark Configurations</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Project Mechanics</span></p>
diff --git a/contributor-guide/development.html b/contributor-guide/development.html
index 7eb947cd87..e79049c2dc 100644
--- a/contributor-guide/development.html
+++ b/contributor-guide/development.html
@@ -433,7 +433,26 @@ under the License.
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Reference</span></p>
<ul class="nav bd-sidenav">
-<li class="toctree-l1"><a class="reference internal" href="spark_expressions_support.html">Supported Spark Expressions</a></li>
+<li class="toctree-l1 has-children"><a class="reference internal" href="expression-audits/index.html">Expression Audits</a><details><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/agg_funcs.html">agg_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/array_funcs.html">array_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/bitwise_funcs.html">bitwise_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/collection_funcs.html">collection_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/conditional_funcs.html">conditional_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/conversion_funcs.html">conversion_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/datetime_funcs.html">datetime_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/generator_funcs.html">generator_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/hash_funcs.html">hash_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/json_funcs.html">json_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/map_funcs.html">map_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/math_funcs.html">math_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/misc_funcs.html">misc_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/predicate_funcs.html">predicate_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/struct_funcs.html">struct_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/url_funcs.html">url_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="expression-audits/window_funcs.html">window_funcs</a></li>
+</ul>
+</details></li>
<li class="toctree-l1"><a class="reference internal" href="spark_configs_support.html">Supported Spark Configurations</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Project Mechanics</span></p>
diff --git a/contributor-guide/benchmark-results/tpc-h.html b/contributor-guide/expression-audits/agg_funcs.html
similarity index 74%
copy from contributor-guide/benchmark-results/tpc-h.html
copy to contributor-guide/expression-audits/agg_funcs.html
index b75aa4ff2a..941a14e9ef 100644
--- a/contributor-guide/benchmark-results/tpc-h.html
+++ b/contributor-guide/expression-audits/agg_funcs.html
@@ -27,7 +27,7 @@ under the License.
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
- <title>Apache DataFusion Comet: Benchmarks Derived From TPC-H — Apache DataFusion Comet documentation</title>
+ <title>agg_funcs Expression Audits — Apache DataFusion Comet documentation</title>
@@ -61,12 +61,12 @@ under the License.
<script src="../../_static/documentation_options.js?v=5929fcd5"></script>
<script src="../../_static/doctools.js?v=9a2dae69"></script>
<script src="../../_static/sphinx_highlight.js?v=dc90522c"></script>
- <script>DOCUMENTATION_OPTIONS.pagename = 'contributor-guide/benchmark-results/tpc-h';</script>
+ <script>DOCUMENTATION_OPTIONS.pagename = 'contributor-guide/expression-audits/agg_funcs';</script>
<script async="true" defer="true" src="https://buttons.github.io/buttons.js"></script>
<link rel="index" title="Index" href="../../genindex.html" />
<link rel="search" title="Search" href="../../search.html" />
- <link rel="next" title="Apache DataFusion Comet: Benchmarks Derived From TPC-DS" href="tpc-ds.html" />
- <link rel="prev" title="Comet Benchmarking Guide" href="../benchmarking.html" />
+ <link rel="next" title="array_funcs Expression Audits" href="array_funcs.html" />
+ <link rel="prev" title="Expression Audits" href="index.html" />
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<meta name="docsearch:language" content="en"/>
<meta name="docsearch:version" content="" />
@@ -418,11 +418,11 @@ under the License.
<li class="toctree-l1"><a class="reference internal" href="../iceberg-spark-tests.html">Iceberg Spark Tests</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Debugging and Performance</span></p>
-<ul class="current nav bd-sidenav">
+<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../debugging.html">Debugging Guide</a></li>
-<li class="toctree-l1 current active has-children"><a class="reference internal" href="../benchmarking.html">Benchmarking Guide</a><details open="open"><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul class="current">
-<li class="toctree-l2 current active"><a class="current reference internal" href="#">Apache DataFusion Comet: Benchmarks Derived From TPC-H</a></li>
-<li class="toctree-l2"><a class="reference internal" href="tpc-ds.html">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</a></li>
+<li class="toctree-l1 has-children"><a class="reference internal" href="../benchmarking.html">Benchmarking Guide</a><details><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul>
+<li class="toctree-l2"><a class="reference internal" href="../benchmark-results/tpc-h.html">Apache DataFusion Comet: Benchmarks Derived From TPC-H</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../benchmark-results/tpc-ds.html">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_macos.html">Comet Benchmarking on macOS</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_aws_ec2.html">Comet Benchmarking in EC2</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_spark_sql_perf.html">TPC-DS Benchmarking with spark-sql-perf</a></li>
@@ -432,8 +432,27 @@ under the License.
<li class="toctree-l1"><a class="reference internal" href="../tracing.html">Tracing</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Reference</span></p>
-<ul class="nav bd-sidenav">
-<li class="toctree-l1"><a class="reference internal" href="../spark_expressions_support.html">Supported Spark Expressions</a></li>
+<ul class="current nav bd-sidenav">
+<li class="toctree-l1 current active has-children"><a class="reference internal" href="index.html">Expression Audits</a><details open="open"><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul class="current">
+<li class="toctree-l2 current active"><a class="current reference internal" href="#">agg_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="array_funcs.html">array_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="bitwise_funcs.html">bitwise_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="collection_funcs.html">collection_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="conditional_funcs.html">conditional_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="conversion_funcs.html">conversion_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="datetime_funcs.html">datetime_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="generator_funcs.html">generator_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="hash_funcs.html">hash_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="json_funcs.html">json_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="map_funcs.html">map_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="math_funcs.html">math_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="misc_funcs.html">misc_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="predicate_funcs.html">predicate_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="struct_funcs.html">struct_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="url_funcs.html">url_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="window_funcs.html">window_funcs</a></li>
+</ul>
+</details></li>
<li class="toctree-l1"><a class="reference internal" href="../spark_configs_support.html">Supported Spark Configurations</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Project Mechanics</span></p>
@@ -488,9 +507,9 @@ under the License.
<li class="breadcrumb-item"><a href="../index.html" class="nav-link">Comet Contributor Guide</a></li>
- <li class="breadcrumb-item"><a href="../benchmarking.html" class="nav-link">Comet Benchmarking Guide</a></li>
+ <li class="breadcrumb-item"><a href="index.html" class="nav-link">Expression Audits</a></li>
- <li class="breadcrumb-item active" aria-current="page"><span class="ellipsis">Apache DataFusion Comet: Benchmarks Derived From TPC-H</span></li>
+ <li class="breadcrumb-item active" aria-current="page"><span class="ellipsis">agg_funcs Expression Audits</span></li>
</ul>
</nav>
</div>
@@ -507,70 +526,52 @@ under the License.
<div id="searchbox"></div>
<article class="bd-article">
- <!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
+ <!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
-->
-<section id="apache-datafusion-comet-benchmarks-derived-from-tpc-h">
-<h1>Apache DataFusion Comet: Benchmarks Derived From TPC-H<a class="headerlink" href="#apache-datafusion-comet-benchmarks-derived-from-tpc-h" title="Link to this heading">#</a></h1>
-<p>The following benchmarks were performed on an EKS cluster (<code class="docutils literal notranslate"><span class="pre">r6i.24xlarge</span></code> instances with EBS storage) with data stored in S3.</p>
-<section id="benchmark-results">
-<h2>Benchmark Results<a class="headerlink" href="#benchmark-results" title="Link to this heading">#</a></h2>
-<p>Total time to run all queries (lower is better).</p>
-<p><img alt="" src="../../_images/tpch_allqueries_with_tuned.png" /></p>
-<p>The following charts are based on the tuned run using hash join.</p>
-<p>Per-query breakdown showing the relative performance of Spark and Comet.</p>
-<p><img alt="" src="../../_images/tpch_queries_compare.png" /></p>
-<p>How much Comet accelerates each query in relative terms.</p>
-<p><img alt="" src="../../_images/tpch_queries_speedup_rel.png" /></p>
-<p>How much Comet accelerates each query in absolute terms.</p>
-<p><img alt="" src="../../_images/tpch_queries_speedup_abs.png" /></p>
+<section id="agg-funcs-expression-audits">
+<h1>agg_funcs Expression Audits<a class="headerlink" href="#agg-funcs-expression-audits" title="Link to this heading">#</a></h1>
+<blockquote>
+<div><p>Audit notes for expressions in this category that have been audited. Absence of an entry means the expression has not been audited yet, not that it is unsupported. See the user guide <a class="reference internal" href="../../user-guide/latest/expressions.html"><span class="std std-doc">Spark Expression Support</span></a> for current support status.</p>
+</div></blockquote>
+<section id="any">
+<h2>any<a class="headerlink" href="#any" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-26): registered as a SQL alias of <code class="docutils literal notranslate"><span class="pre">BoolOr</span></code>, which extends <code class="docutils literal notranslate"><span class="pre">RuntimeReplaceableAggregate</span></code> with <code class="docutils literal notranslate"><span class="pre">replacement</span> <span class="pre">=</span> <span class="pre">Max(child)</span></code>. Catalyst rewrites <code class="docutils literal notranslate"><span [...]
+<li><p>Spark 3.5.8 (audited 2026-05-26): identical to 3.4.3.</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-26): identical to 3.4.3.</p></li>
+</ul>
</section>
-<section id="configuration">
-<h2>Configuration<a class="headerlink" href="#configuration" title="Link to this heading">#</a></h2>
-<p>Common:</p>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.executor.instances</span><span class="o">=</span><span class="s">32</span>
-<span class="na">spark.executor.cores</span><span class="o">=</span><span class="s">16</span>
-<span class="na">spark.memory.fraction</span><span class="o">=</span><span class="s">0.6</span>
-<span class="na">spark.memory.storageFraction</span><span class="o">=</span><span class="s">0.2</span>
-<span class="c1"># Kubernetes CPU constraints</span>
-<span class="na">spark.kubernetes.executor.request.cores</span><span class="o">=</span><span class="s">8</span>
-<span class="na">spark.kubernetes.executor.limit.cores</span><span class="o">=</span><span class="s">8</span>
-</pre></div>
-</div>
-<p>Spark:</p>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.executor.memory</span><span class="o">=</span><span class="s">64G</span>
-<span class="na">spark.executor.memoryOverhead</span><span class="o">=</span><span class="s">10G</span>
-</pre></div>
-</div>
-<p>Comet:</p>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.executor.memory</span><span class="o">=</span><span class="s">32G</span>
-<span class="na">spark.executor.memoryOverhead</span><span class="o">=</span><span class="s">10G</span>
-<span class="na">spark.memory.offHeap.enabled</span><span class="o">=</span><span class="s">true</span>
-<span class="na">spark.memory.offHeap.size</span><span class="o">=</span><span class="s">32G</span>
-</pre></div>
-</div>
-<section id="comet-tuned">
-<h3>Comet (Tuned)<a class="headerlink" href="#comet-tuned" title="Link to this heading">#</a></h3>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.comet.exec.replaceSortMergeJoin</span><span class="o">=</span><span class="s">true</span>
-<span class="na">spark.comet.memoryPool.fraction</span><span class="o">=</span><span class="s">0.8</span>
-</pre></div>
-</div>
+<section id="avg">
+<h2>avg<a class="headerlink" href="#avg" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (2026-05-26)</p></li>
+<li><p>Spark 3.5.8 (2026-05-26): aggregate logic identical to 3.4.3</p></li>
+<li><p>Spark 4.0.1 (2026-05-26): aggregate logic identical to 3.5.8; only <code class="docutils literal notranslate"><span class="pre">QueryContext</span></code> import path differs. <code class="docutils literal notranslate"><span class="pre">YearMonthIntervalType</span></code> and <code class="docutils literal notranslate"><span class="pre">DayTimeIntervalType</span></code> inputs (supported by Spark) fall back to Spark in Comet.</p></li>
+</ul>
</section>
+<section id="bit-and">
+<h2>bit_and<a class="headerlink" href="#bit-and" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (2026-05-26)</p></li>
+<li><p>Spark 3.5.8 (2026-05-26)</p></li>
+<li><p>Spark 4.0.1 (2026-05-26)</p></li>
+</ul>
</section>
</section>
@@ -585,20 +586,20 @@ under the License.
<div class="prev-next-area">
<a class="left-prev"
- href="../benchmarking.html"
+ href="index.html"
title="previous page">
<i class="fa-solid fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">Comet Benchmarking Guide</p>
+ <p class="prev-next-title">Expression Audits</p>
</div>
</a>
<a class="right-next"
- href="tpc-ds.html"
+ href="array_funcs.html"
title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
- <p class="prev-next-title">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</p>
+ <p class="prev-next-title">array_funcs Expression Audits</p>
</div>
<i class="fa-solid fa-angle-right"></i>
</a>
diff --git a/contributor-guide/expression-audits/array_funcs.html b/contributor-guide/expression-audits/array_funcs.html
new file mode 100644
index 0000000000..cf588b4dbd
--- /dev/null
+++ b/contributor-guide/expression-audits/array_funcs.html
@@ -0,0 +1,811 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+
+<!DOCTYPE html>
+
+
+<html lang="en" data-content_root="../../" data-theme="light">
+
+ <head>
+ <meta charset="utf-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
+
+ <title>array_funcs Expression Audits — Apache DataFusion Comet documentation</title>
+
+
+
+ <script data-cfasync="false">
+ document.documentElement.dataset.mode = localStorage.getItem("mode") || "light";
+ document.documentElement.dataset.theme = localStorage.getItem("theme") || "light";
+ </script>
+ <!--
+ this give us a css class that will be invisible only if js is disabled
+ -->
+ <noscript>
+ <style>
+ .pst-js-only { display: none !important; }
+
+ </style>
+ </noscript>
+
+ <!-- Loaded before other Sphinx assets -->
+ <link href="../../_static/styles/theme.css?digest=8878045cc6db502f8baf" rel="stylesheet" />
+<link href="../../_static/styles/pydata-sphinx-theme.css?digest=8878045cc6db502f8baf" rel="stylesheet" />
+
+ <link rel="stylesheet" type="text/css" href="../../_static/pygments.css?v=8f2a1f02" />
+ <link rel="stylesheet" type="text/css" href="../../_static/theme_overrides.css?v=cd442bcd" />
+
+ <!-- So that users can add custom icons -->
+ <script src="../../_static/scripts/fontawesome.js?digest=8878045cc6db502f8baf"></script>
+ <!-- Pre-loaded scripts that we'll load fully later -->
+ <link rel="preload" as="script" href="../../_static/scripts/bootstrap.js?digest=8878045cc6db502f8baf" />
+<link rel="preload" as="script" href="../../_static/scripts/pydata-sphinx-theme.js?digest=8878045cc6db502f8baf" />
+
+ <script src="../../_static/documentation_options.js?v=5929fcd5"></script>
+ <script src="../../_static/doctools.js?v=9a2dae69"></script>
+ <script src="../../_static/sphinx_highlight.js?v=dc90522c"></script>
+ <script>DOCUMENTATION_OPTIONS.pagename = 'contributor-guide/expression-audits/array_funcs';</script>
+ <script async="true" defer="true" src="https://buttons.github.io/buttons.js"></script>
+ <link rel="index" title="Index" href="../../genindex.html" />
+ <link rel="search" title="Search" href="../../search.html" />
+ <link rel="next" title="bitwise_funcs Expression Audits" href="bitwise_funcs.html" />
+ <link rel="prev" title="agg_funcs Expression Audits" href="agg_funcs.html" />
+ <meta name="viewport" content="width=device-width, initial-scale=1"/>
+ <meta name="docsearch:language" content="en"/>
+ <meta name="docsearch:version" content="" />
+ </head>
+
+
+ <body data-bs-spy="scroll" data-bs-target=".bd-toc-nav" data-offset="180" data-bs-root-margin="0px 0px -60%" data-default-mode="light">
+
+
+
+ <div id="pst-skip-link" class="skip-link d-print-none"><a href="#main-content">Skip to main content</a></div>
+
+ <div id="pst-scroll-pixel-helper"></div>
+
+ <button type="button" class="btn rounded-pill" id="pst-back-to-top">
+ <i class="fa-solid fa-arrow-up"></i>Back to top</button>
+
+
+ <dialog id="pst-search-dialog">
+
+<form class="bd-search d-flex align-items-center"
+ action="../../search.html"
+ method="get">
+ <i class="fa-solid fa-magnifying-glass"></i>
+ <input type="search"
+ class="form-control"
+ name="q"
+ placeholder="Search the docs ..."
+ aria-label="Search the docs ..."
+ autocomplete="off"
+ autocorrect="off"
+ autocapitalize="off"
+ spellcheck="false"/>
+ <span class="search-button__kbd-shortcut"><kbd class="kbd-shortcut__modifier">Ctrl</kbd>+<kbd>K</kbd></span>
+</form>
+ </dialog>
+
+ <div class="pst-async-banner-revealer d-none">
+ <aside id="bd-header-version-warning" class="d-none d-print-none" aria-label="Version warning"></aside>
+</div>
+
+
+ <header class="bd-header navbar navbar-expand-lg bd-navbar d-print-none">
+<div class="bd-header__inner bd-page-width">
+ <button class="pst-navbar-icon sidebar-toggle primary-toggle" aria-label="Site navigation">
+ <span class="fa-solid fa-bars"></span>
+ </button>
+
+
+ <div class="col-lg-3 navbar-header-items__start">
+
+ <div class="navbar-item">
+
+
+
+
+
+<a class="navbar-brand logo" href="../../index.html">
+
+
+
+
+
+
+
+
+ <img src="../../_static/DataFusionComet-Logo-Light.png" class="logo__image only-light" alt="Apache DataFusion Comet documentation - Home"/>
+ <img src="../../_static/DataFusionComet-Logo-Dark.png" class="logo__image only-dark pst-js-only" alt="Apache DataFusion Comet documentation - Home"/>
+
+
+</a></div>
+
+ </div>
+
+ <div class="col-lg-9 navbar-header-items">
+
+ <div class="me-auto navbar-header-items__center">
+
+ <div class="navbar-item">
+<nav>
+ <ul class="bd-navbar-elements navbar-nav">
+
+<li class="nav-item ">
+ <a class="nav-link nav-internal" href="../../user-guide/index.html">
+ User Guide
+ </a>
+</li>
+
+
+<li class="nav-item current active">
+ <a class="nav-link nav-internal" href="../index.html">
+ Contributor Guide
+ </a>
+</li>
+
+
+<li class="nav-item ">
+ <a class="nav-link nav-internal" href="../../changelog/index.html">
+ Changelog
+ </a>
+</li>
+
+
+<li class="nav-item ">
+ <a class="nav-link nav-internal" href="../../about/gluten_comparison.html">
+ Comparison with Gluten
+ </a>
+</li>
+
+
+<li class="nav-item ">
+ <a class="nav-link nav-internal" href="../../about/versioning_policy.html">
+ Versioning Policy
+ </a>
+</li>
+
+ <li class="nav-item dropdown">
+ <button class="btn dropdown-toggle nav-item" type="button"
+ data-bs-toggle="dropdown" aria-expanded="false"
+ aria-controls="pst-nav-more-links">
+ More
+ </button>
+ <ul id="pst-nav-more-links" class="dropdown-menu">
+
+<li class=" ">
+ <a class="nav-link dropdown-item nav-internal" href="../../asf/index.html">
+ ASF Links
+ </a>
+</li>
+
+ </ul>
+ </li>
+
+ </ul>
+</nav></div>
+
+ </div>
+
+
+ <div class="navbar-header-items__end">
+
+ <div class="navbar-item navbar-persistent--container">
+
+
+<button class="btn search-button-field search-button__button pst-js-only" title="Search" aria-label="Search" data-bs-placement="bottom" data-bs-toggle="tooltip">
+ <i class="fa-solid fa-magnifying-glass"></i>
+ <span class="search-button__default-text">Search</span>
+ <span class="search-button__kbd-shortcut"><kbd class="kbd-shortcut__modifier">Ctrl</kbd>+<kbd class="kbd-shortcut__modifier">K</kbd></span>
+</button>
+ </div>
+
+
+ <div class="navbar-item"><ul class="navbar-icon-links"
+ aria-label="Icon Links">
+ <li class="nav-item">
+
+
+
+
+
+
+
+
+ <a href="https://github.com/apache/datafusion-comet" title="GitHub" class="nav-link pst-navbar-icon" rel="noopener" target="_blank" data-bs-toggle="tooltip" data-bs-placement="bottom"><i class="fa-brands fa-github fa-lg" aria-hidden="true"></i>
+ <span class="sr-only">GitHub</span></a>
+ </li>
+</ul></div>
+
+ <div class="navbar-item">
+
+<button class="btn btn-sm nav-link pst-navbar-icon theme-switch-button pst-js-only" aria-label="Color mode" data-bs-title="Color mode" data-bs-placement="bottom" data-bs-toggle="tooltip">
+ <i class="theme-switch fa-solid fa-sun fa-lg" data-mode="light" title="Light"></i>
+ <i class="theme-switch fa-solid fa-moon fa-lg" data-mode="dark" title="Dark"></i>
+ <i class="theme-switch fa-solid fa-circle-half-stroke fa-lg" data-mode="auto" title="System Settings"></i>
+</button></div>
+
+ </div>
+
+ </div>
+
+
+ <div class="navbar-persistent--mobile">
+
+<button class="btn search-button-field search-button__button pst-js-only" title="Search" aria-label="Search" data-bs-placement="bottom" data-bs-toggle="tooltip">
+ <i class="fa-solid fa-magnifying-glass"></i>
+ <span class="search-button__default-text">Search</span>
+ <span class="search-button__kbd-shortcut"><kbd class="kbd-shortcut__modifier">Ctrl</kbd>+<kbd class="kbd-shortcut__modifier">K</kbd></span>
+</button>
+ </div>
+
+
+
+</div>
+
+ </header>
+
+
+ <div class="bd-container">
+ <div class="bd-container__inner bd-page-width">
+
+
+
+ <dialog id="pst-primary-sidebar-modal"></dialog>
+ <div id="pst-primary-sidebar" class="bd-sidebar-primary bd-sidebar">
+
+
+
+ <div class="sidebar-header-items sidebar-primary__section">
+
+
+ <div class="sidebar-header-items__center">
+
+
+
+ <div class="navbar-item">
+<nav>
+ <ul class="bd-navbar-elements navbar-nav">
+
+<li class="nav-item ">
+ <a class="nav-link nav-internal" href="../../user-guide/index.html">
+ User Guide
+ </a>
+</li>
+
+
+<li class="nav-item current active">
+ <a class="nav-link nav-internal" href="../index.html">
+ Contributor Guide
+ </a>
+</li>
+
+
+<li class="nav-item ">
+ <a class="nav-link nav-internal" href="../../changelog/index.html">
+ Changelog
+ </a>
+</li>
+
+
+<li class="nav-item ">
+ <a class="nav-link nav-internal" href="../../about/gluten_comparison.html">
+ Comparison with Gluten
+ </a>
+</li>
+
+
+<li class="nav-item ">
+ <a class="nav-link nav-internal" href="../../about/versioning_policy.html">
+ Versioning Policy
+ </a>
+</li>
+
+
+<li class="nav-item ">
+ <a class="nav-link nav-internal" href="../../asf/index.html">
+ ASF Links
+ </a>
+</li>
+
+ </ul>
+</nav></div>
+
+
+ </div>
+
+
+
+ <div class="sidebar-header-items__end">
+
+ <div class="navbar-item"><ul class="navbar-icon-links"
+ aria-label="Icon Links">
+ <li class="nav-item">
+
+
+
+
+
+
+
+
+ <a href="https://github.com/apache/datafusion-comet" title="GitHub" class="nav-link pst-navbar-icon" rel="noopener" target="_blank" data-bs-toggle="tooltip" data-bs-placement="bottom"><i class="fa-brands fa-github fa-lg" aria-hidden="true"></i>
+ <span class="sr-only">GitHub</span></a>
+ </li>
+</ul></div>
+
+ <div class="navbar-item">
+
+<button class="btn btn-sm nav-link pst-navbar-icon theme-switch-button pst-js-only" aria-label="Color mode" data-bs-title="Color mode" data-bs-placement="bottom" data-bs-toggle="tooltip">
+ <i class="theme-switch fa-solid fa-sun fa-lg" data-mode="light" title="Light"></i>
+ <i class="theme-switch fa-solid fa-moon fa-lg" data-mode="dark" title="Dark"></i>
+ <i class="theme-switch fa-solid fa-circle-half-stroke fa-lg" data-mode="auto" title="System Settings"></i>
+</button></div>
+
+ </div>
+
+ </div>
+
+ <div class="sidebar-primary-items__start sidebar-primary__section">
+ <div class="sidebar-primary-item"><!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+
+
+
+
+<nav class="bd-links" id="bd-docs-nav" aria-label="Main navigation">
+ <div class="bd-toc-item active">
+ <p aria-level="2" class="caption" role="heading"><span class="caption-text">Getting Started</span></p>
+<ul class="nav bd-sidenav">
+<li class="toctree-l1"><a class="reference internal" href="../contributing.html">Getting Started</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../development.html">Development Guide</a></li>
+</ul>
+<p aria-level="2" class="caption" role="heading"><span class="caption-text">Project Architecture</span></p>
+<ul class="nav bd-sidenav">
+<li class="toctree-l1"><a class="reference internal" href="../plugin_overview.html">Comet Plugin Architecture</a></li>
+
+<li class="toctree-l1"><a class="reference internal" href="../ffi.html">Arrow FFI</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../jvm_shuffle.html">JVM Shuffle</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../native_shuffle.html">Native Shuffle</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../sql_error_propagation.html">ANSI Error Propagation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../s3-credential-provider-design.html">S3 Credential Provider Design</a></li>
+</ul>
+<p aria-level="2" class="caption" role="heading"><span class="caption-text">Adding Functionality</span></p>
+<ul class="nav bd-sidenav">
+<li class="toctree-l1"><a class="reference internal" href="../adding_a_new_operator.html">Adding a New Operator</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../adding_a_new_expression.html">Adding a New Expression</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../adding_a_new_spark_version.html">Adding a New Spark Version</a></li>
+</ul>
+<p aria-level="2" class="caption" role="heading"><span class="caption-text">Testing</span></p>
+<ul class="nav bd-sidenav">
+<li class="toctree-l1"><a class="reference internal" href="../sql-file-tests.html">Comet SQL Tests</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../spark-sql-tests.html">Spark SQL Tests</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../iceberg-spark-tests.html">Iceberg Spark Tests</a></li>
+</ul>
+<p aria-level="2" class="caption" role="heading"><span class="caption-text">Debugging and Performance</span></p>
+<ul class="nav bd-sidenav">
+<li class="toctree-l1"><a class="reference internal" href="../debugging.html">Debugging Guide</a></li>
+<li class="toctree-l1 has-children"><a class="reference internal" href="../benchmarking.html">Benchmarking Guide</a><details><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul>
+<li class="toctree-l2"><a class="reference internal" href="../benchmark-results/tpc-h.html">Apache DataFusion Comet: Benchmarks Derived From TPC-H</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../benchmark-results/tpc-ds.html">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../benchmarking_macos.html">Comet Benchmarking on macOS</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../benchmarking_aws_ec2.html">Comet Benchmarking in EC2</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../benchmarking_spark_sql_perf.html">TPC-DS Benchmarking with spark-sql-perf</a></li>
+</ul>
+</details></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tracing.html">Tracing</a></li>
+</ul>
+<p aria-level="2" class="caption" role="heading"><span class="caption-text">Reference</span></p>
+<ul class="current nav bd-sidenav">
+<li class="toctree-l1 current active has-children"><a class="reference internal" href="index.html">Expression Audits</a><details open="open"><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul class="current">
+<li class="toctree-l2"><a class="reference internal" href="agg_funcs.html">agg_funcs</a></li>
+<li class="toctree-l2 current active"><a class="current reference internal" href="#">array_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="bitwise_funcs.html">bitwise_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="collection_funcs.html">collection_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="conditional_funcs.html">conditional_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="conversion_funcs.html">conversion_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="datetime_funcs.html">datetime_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="generator_funcs.html">generator_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="hash_funcs.html">hash_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="json_funcs.html">json_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="map_funcs.html">map_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="math_funcs.html">math_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="misc_funcs.html">misc_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="predicate_funcs.html">predicate_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="struct_funcs.html">struct_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="url_funcs.html">url_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="window_funcs.html">window_funcs</a></li>
+</ul>
+</details></li>
+<li class="toctree-l1"><a class="reference internal" href="../spark_configs_support.html">Supported Spark Configurations</a></li>
+</ul>
+<p aria-level="2" class="caption" role="heading"><span class="caption-text">Project Mechanics</span></p>
+<ul class="nav bd-sidenav">
+<li class="toctree-l1"><a class="reference internal" href="../bug_triage.html">Bug Triage</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../release_process.html">Release Process</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../roadmap.html">Roadmap</a></li>
+<li class="toctree-l1"><a class="reference external" href="https://github.com/apache/datafusion-comet">Github and Issue Tracker</a></li>
+</ul>
+
+ </div>
+</nav>
+</div>
+ </div>
+
+
+ <div class="sidebar-primary-items__end sidebar-primary__section">
+ <div class="sidebar-primary-item">
+<div id="ethical-ad-placement"
+ class="flat"
+ data-ea-publisher="readthedocs"
+ data-ea-type="readthedocs-sidebar"
+ data-ea-manual="true">
+</div></div>
+ </div>
+
+
+ </div>
+
+ <main id="main-content" class="bd-main" role="main">
+
+
+ <div class="bd-content">
+ <div class="bd-article-container">
+
+ <div class="bd-header-article d-print-none">
+<div class="header-article-items header-article__inner">
+
+ <div class="header-article-items__start">
+
+ <div class="header-article-item">
+
+<nav aria-label="Breadcrumb" class="d-print-none">
+ <ul class="bd-breadcrumbs">
+
+ <li class="breadcrumb-item breadcrumb-home">
+ <a href="../../index.html" class="nav-link" aria-label="Home">
+ <i class="fa-solid fa-home"></i>
+ </a>
+ </li>
+
+ <li class="breadcrumb-item"><a href="../index.html" class="nav-link">Comet Contributor Guide</a></li>
+
+
+ <li class="breadcrumb-item"><a href="index.html" class="nav-link">Expression Audits</a></li>
+
+ <li class="breadcrumb-item active" aria-current="page"><span class="ellipsis">array_funcs Expression Audits</span></li>
+ </ul>
+</nav>
+</div>
+
+ </div>
+
+
+</div>
+</div>
+
+
+
+
+<div id="searchbox"></div>
+ <article class="bd-article">
+
+ <!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+<section id="array-funcs-expression-audits">
+<h1>array_funcs Expression Audits<a class="headerlink" href="#array-funcs-expression-audits" title="Link to this heading">#</a></h1>
+<blockquote>
+<div><p>Audit notes for expressions in this category that have been audited. Absence of an entry means the expression has not been audited yet, not that it is unsupported. See the user guide <a class="reference internal" href="../../user-guide/latest/expressions.html"><span class="std std-doc">Spark Expression Support</span></a> for current support status.</p>
+</div></blockquote>
+<section id="array">
+<h2>array<a class="headerlink" href="#array" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">CreateArray(children,</span> <span class="pre">useStringTypeWhenEmpty)</span></code>; element type is the common type of children. Comet routes via <code class="docutils literal notranslate"><span class="pre">CometCreateArray</span></code> (native <code class="docutils literal notranslate"><span class="pre">make_array</span></code>) and special-cases the empty-array case to dod [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): semantics unchanged.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): adds <code class="docutils literal notranslate"><span class="pre">contextIndependentFoldable</span></code> override; runtime semantics unchanged.</p></li>
+</ul>
+</section>
+<section id="array-append">
+<h2>array_append<a class="headerlink" href="#array-append" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): standalone <code class="docutils literal notranslate"><span class="pre">BinaryExpression</span></code>, evaluated directly. Comet routes via <code class="docutils literal notranslate"><span class="pre">CometArrayAppend</span></code>.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): now <code class="docutils literal notranslate"><span class="pre">RuntimeReplaceable</span></code> and rewritten to <code class="docutils literal notranslate"><span class="pre">ArrayInsert(arr,</span> <span class="pre">Literal(-1),</span> <span class="pre">elem)</span></code>. <code class="docutils literal notranslate"><span class="pre">CometArrayAppend</span></code> is therefore unreachable; dispatch goes through <code class="docutils literal notr [...]
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</section>
+<section id="array-compact">
+<h2>array_compact<a class="headerlink" href="#array-compact" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">RuntimeReplaceable</span></code> -> <code class="docutils literal notranslate"><span class="pre">ArrayFilter(arr,</span> <span class="pre">IsNotNull(lambda))</span></code>. Comet receives the rewritten form, dispatches through <code class="docutils literal notranslate"><span class="pre">CometArrayFilter</span></code>, which delegates back to <code class="docutils literal notranslate"> [...]
+<li><p>Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): the replacement is wrapped in <code class="docutils literal notranslate"><span class="pre">KnownNotContainsNull(...)</span></code> (analysis-only hint, no semantic change).</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</section>
+<section id="array-contains">
+<h2>array_contains<a class="headerlink" href="#array-contains" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">ArrayContains(left,</span> <span class="pre">right)</span> <span class="pre">extends</span> <span class="pre">BinaryExpression</span> <span class="pre">with</span> <span class="pre">NullIntolerant</span> <span class="pre">with</span> <span class="pre">Predicate</span></code>; <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code> uses <code class= [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">NullIntolerant</span></code> trait replaced by <code class="docutils literal notranslate"><span class="pre">nullIntolerant:</span> <span class="pre">Boolean</span></code>; <code class="docutils literal notranslate"><span class="pre">checkInputDataTypes</span></code> adopts <code class="docutils literal notranslate"><span class="pre">DataTypeUtils.sameType</span></code> (collation-aware i [...]
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+<li><p>Known limitation: no NaN-canonicalization guard in <code class="docutils literal notranslate"><span class="pre">getSupportLevel</span></code>. For <code class="docutils literal notranslate"><span class="pre">Float</span></code>/<code class="docutils literal notranslate"><span class="pre">Double</span></code> arrays containing NaN, Spark’s <code class="docutils literal notranslate"><span class="pre">SQLOrderingUtil</span></code> may produce different results than DataFusion’s IEEE [...]
+</ul>
+</section>
+<section id="array-distinct">
+<h2>array_distinct<a class="headerlink" href="#array-distinct" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">ArrayDistinct(child)</span></code> over <code class="docutils literal notranslate"><span class="pre">ArraySetLike</span></code>; uses <code class="docutils literal notranslate"><span class="pre">SQLOpenHashSet</span></code> so NaN and <code class="docutils literal notranslate"><span class="pre">+0.0</span></code>/<code class="docutils literal notranslate"><span class="pre">-0.0 [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">NullIntolerant</span></code> -> <code class="docutils literal notranslate"><span class="pre">nullIntolerant</span></code> field refactor.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+<li><p>Known divergence: DataFusion <code class="docutils literal notranslate"><span class="pre">array_distinct</span></code> uses hash-based equality without NaN/signed-zero canonicalization, so float/double arrays may produce different results (https://github.com/apache/datafusion-comet/issues/4481).</p></li>
+</ul>
+</section>
+<section id="array-except">
+<h2>array_except<a class="headerlink" href="#array-except" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">ArrayExcept(left,</span> <span class="pre">right)</span> <span class="pre">extends</span> <span class="pre">ArrayBinaryLike</span> <span class="pre">with</span> <span class="pre">ComplexTypeMergingExpression</span></code>; result preserves left-side first occurrences not present in right. Comet routes via <code class="docutils literal notranslate"><span class="pre">CometArrayEx [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">nullIntolerant</span> <span class="pre">=</span> <span class="pre">true</span></code> moves into <code class="docutils literal notranslate"><span class="pre">ArrayBinaryLike</span></code>; the overflow path uses <code class="docutils literal notranslate"><span class="pre">arrayFunctionWithElementsExceedLimitError</span></code>.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+<li><p>Known divergence: same NaN/signed-zero canonicalization gap as <code class="docutils literal notranslate"><span class="pre">array_distinct</span></code> for float/double arrays (https://github.com/apache/datafusion-comet/issues/4481).</p></li>
+</ul>
+</section>
+<section id="array-insert">
+<h2>array_insert<a class="headerlink" href="#array-insert" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 audited 2026-04-02</p></li>
+<li><p>Spark 3.5.8 audited 2026-04-02</p></li>
+<li><p>Spark 4.0.1 audited 2026-04-02 (pos=0 error message differs from Spark)</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</section>
+<section id="array-intersect">
+<h2>array_intersect<a class="headerlink" href="#array-intersect" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 audited 2026-04-24 (result element order may differ from Spark when the right array is longer than the left; DataFusion probes the longer side)</p></li>
+<li><p>Spark 3.5.8 audited 2026-04-24 (same ordering incompatibility as 3.4.3)</p></li>
+<li><p>Spark 4.0.1 audited 2026-04-24 (ordering incompatibility as above; collated strings now fall back to Spark)</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</section>
+<section id="array-join">
+<h2>array_join<a class="headerlink" href="#array-join" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">ArrayJoin(array,</span> <span class="pre">delimiter,</span> <span class="pre">nullReplacement)</span></code>. Comet routes via <code class="docutils literal notranslate"><span class="pre">CometArrayJoin</span></code> to DataFusion’s <code class="docutils literal notranslate"><span class="pre">array_to_string</span></code> and is unconditionally flagged <code class="docutils lit [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code> widened to <code class="docutils literal notranslate"><span class="pre">AbstractArrayType(StringTypeWithCollation(supportsTrimCollation</span> <span class="pre">=</span> <span class="pre">true))</span></code>; non-binary collations not propagated (https://github.com/apache/datafusion-comet/issues/2190).</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): adds <code class="docutils literal notranslate"><span class="pre">contextIndependentFoldable</span></code> override; runtime unchanged.</p></li>
+</ul>
+</section>
+<section id="array-max">
+<h2>array_max<a class="headerlink" href="#array-max" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">ArrayMax(child)</span> <span class="pre">extends</span> <span class="pre">UnaryExpression</span> <span class="pre">with</span> <span class="pre">ImplicitCastInputTypes</span></code>; skips NULL elements; for float/double Spark’s <code class="docutils literal notranslate"><span class="pre">SQLOrderingUtil</span></code> treats NaN as greater than any non-NaN. Wired as <code class [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">NullIntolerant</span></code> -> <code class="docutils literal notranslate"><span class="pre">nullIntolerant</span></code> field refactor.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+<li><p>Known divergence: DataFusion’s <code class="docutils literal notranslate"><span class="pre">array_max</span></code> uses Arrow <code class="docutils literal notranslate"><span class="pre">partial_cmp</span></code>-based ordering, so float/double arrays containing NaN may produce different results (https://github.com/apache/datafusion-comet/issues/4482).</p></li>
+</ul>
+</section>
+<section id="array-min">
+<h2>array_min<a class="headerlink" href="#array-min" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): mirror of <code class="docutils literal notranslate"><span class="pre">ArrayMax</span></code> with <code class="docutils literal notranslate"><span class="pre">evalInternal</span></code> returning the minimum. Same NULL-skip and NaN-ordering semantics. Wired as <code class="docutils literal notranslate"><span class="pre">CometScalarFunction("array_min")</span></code>.</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): same trait refactor as <code class="docutils literal notranslate"><span class="pre">array_max</span></code>.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+<li><p>Known divergence: same NaN-handling gap as <code class="docutils literal notranslate"><span class="pre">array_max</span></code> (https://github.com/apache/datafusion-comet/issues/4482).</p></li>
+</ul>
+</section>
+<section id="array-position">
+<h2>array_position<a class="headerlink" href="#array-position" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">ArrayPosition(left,</span> <span class="pre">right)</span></code>; returns 1-based <code class="docutils literal notranslate"><span class="pre">LongType</span></code> position, 0 if not found, NULL if either input is NULL. <code class="docutils literal notranslate"><span class="pre">CometArrayPosition</span></code> falls back for all-foldable args (constant folding handles thos [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">NullIntolerant</span></code> -> <code class="docutils literal notranslate"><span class="pre">nullIntolerant</span></code> field refactor.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</section>
+<section id="array-remove">
+<h2>array_remove<a class="headerlink" href="#array-remove" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">ArrayRemove(left,</span> <span class="pre">right)</span></code>; removes all occurrences equal to <code class="docutils literal notranslate"><span class="pre">right</span></code>. Wired as <code class="docutils literal notranslate"><span class="pre">CometScalarFunction("array_remove")</span></code>. Falls back via <code class="docutils literal notranslate"><span class [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">NullIntolerant</span></code> -> <code class="docutils literal notranslate"><span class="pre">nullIntolerant</span></code> field refactor.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</section>
+<section id="array-repeat">
+<h2>array_repeat<a class="headerlink" href="#array-repeat" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">ArrayRepeat(left,</span> <span class="pre">right)</span> <span class="pre">extends</span> <span class="pre">BinaryExpression</span> <span class="pre">with</span> <span class="pre">ExpectsInputTypes</span></code>; <code class="docutils literal notranslate"><span class="pre">inputTypes</span> <span class="pre">=</span> <span class="pre">Seq(AnyDataType,</span> <span class="pre">I [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): error message uses <code class="docutils literal notranslate"><span class="pre">createArrayWithElementsExceedLimitError(prettyName,</span> <span class="pre">count)</span></code>; semantics unchanged.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</section>
+<section id="array-union">
+<h2>array_union<a class="headerlink" href="#array-union" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">ArrayUnion(left,</span> <span class="pre">right)</span> <span class="pre">extends</span> <span class="pre">ArrayBinaryLike</span> <span class="pre">with</span> <span class="pre">ComplexTypeMergingExpression</span></code>; result is left-side distinct elements followed by new right-side elements. Wired as <code class="docutils literal notranslate"><span class="pre">CometScalarFu [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">nullIntolerant</span> <span class="pre">=</span> <span class="pre">true</span></code> moves into <code class="docutils literal notranslate"><span class="pre">ArrayBinaryLike</span></code>; overflow path uses <code class="docutils literal notranslate"><span class="pre">arrayFunctionWithElementsExceedLimitError</span></code>.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+<li><p>Known divergence: same NaN/signed-zero canonicalization gap as <code class="docutils literal notranslate"><span class="pre">array_distinct</span></code> (https://github.com/apache/datafusion-comet/issues/4481). Result ordering versus DataFusion is also unverified; compare the <code class="docutils literal notranslate"><span class="pre">array_intersect</span></code> ordering caveat.</p></li>
+</ul>
+</section>
+<section id="arrays-overlap">
+<h2>arrays_overlap<a class="headerlink" href="#arrays-overlap" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">ArraysOverlap(left,</span> <span class="pre">right)</span></code>; three-valued logic (TRUE if any common non-null element, NULL if a null is present and no overlap is found in non-nulls, FALSE otherwise). Comet routes via <code class="docutils literal notranslate"><span class="pre">CometArraysOverlap</span></code> to the native <code class="docutils literal notranslate"><span [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">NullIntolerant</span></code> -> <code class="docutils literal notranslate"><span class="pre">nullIntolerant</span></code> field refactor.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</section>
+<section id="arrays-zip">
+<h2>arrays_zip<a class="headerlink" href="#arrays-zip" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">ArraysZip(children,</span> <span class="pre">names)</span></code>; returns an array of structs, padding shorter inputs with NULL. Comet routes via <code class="docutils literal notranslate"><span class="pre">CometArraysZip</span></code> and rejects unsupported child element types (anything outside primitives, decimals, dates/timestamps, strings, binary, and nested arrays/struct [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): the length-mismatch error switches from <code class="docutils literal notranslate"><span class="pre">IllegalArgumentException</span></code> to <code class="docutils literal notranslate"><span class="pre">SparkIllegalArgumentException("_LEGACY_ERROR_TEMP_3235")</span></code>; runtime unchanged.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</section>
+<section id="element-at">
+<h2>element_at<a class="headerlink" href="#element-at" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">ElementAt(left,</span> <span class="pre">right,</span> <span class="pre">defaultValueOutOfBound,</span> <span class="pre">failOnError)</span></code>; group label <code class="docutils literal notranslate"><span class="pre">map_funcs</span></code>. Comet supports only <code class="docutils literal notranslate"><span class="pre">ArrayType</span></code> input; <code class="docutil [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">NullIntolerant</span></code> -> <code class="docutils literal notranslate"><span class="pre">nullIntolerant</span></code> field refactor; group label changes to <code class="docutils literal notranslate"><span class="pre">collection_funcs</span></code>; ANSI default flips to <code class="docutils literal notranslate"><span class="pre">true</span></code> so out-of-bound throws by defau [...]
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</section>
+<section id="flatten">
+<h2>flatten<a class="headerlink" href="#flatten" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">Flatten(child)</span> <span class="pre">extends</span> <span class="pre">UnaryExpression</span></code>; returns NULL if any inner sub-array is NULL. Comet routes via <code class="docutils literal notranslate"><span class="pre">CometFlatten</span></code> and falls back for child types containing <code class="docutils literal notranslate"><span class="pre">BinaryType</span></code [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">NullIntolerant</span></code> -> <code class="docutils literal notranslate"><span class="pre">nullIntolerant</span></code> field refactor.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</section>
+<section id="get">
+<h2>get<a class="headerlink" href="#get" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">GetArrayItem(child,</span> <span class="pre">ordinal,</span> <span class="pre">failOnError)</span></code>; <code class="docutils literal notranslate"><span class="pre">inputTypes</span> <span class="pre">=</span> <span class="pre">Seq(AnyDataType,</span> <span class="pre">IntegralType)</span></code>. Comet routes via <code class="docutils literal notranslate"><span class="pre"> [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): semantics unchanged; ANSI default flips to <code class="docutils literal notranslate"><span class="pre">true</span></code>.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code> tightened to <code class="docutils literal notranslate"><span class="pre">Seq(ArrayType,</span> <span class="pre">IntegralType)</span></code> (analysis-time only); runtime unchanged.</p></li>
+</ul>
+</section>
+<section id="sort-array">
+<h2>sort_array<a class="headerlink" href="#sort-array" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">SortArray(base,</span> <span class="pre">ascendingOrder)</span> <span class="pre">extends</span> <span class="pre">BinaryExpression</span> <span class="pre">with</span> <span class="pre">ArraySortLike</span></code>; the second arg must be a <code class="docutils literal notranslate"><span class="pre">Literal(_:</span> <span class="pre">Boolean,</span> <span class="pre">BooleanT [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): trait set changes substantively: <code class="docutils literal notranslate"><span class="pre">ArraySortLike</span></code> and <code class="docutils literal notranslate"><span class="pre">NullIntolerant</span></code> are removed, <code class="docutils literal notranslate"><span class="pre">nullIntolerant</span> <span class="pre">=</span> <span class="pre">true</span></code> becomes an override, and <code class="docutils literal notranslate"><span c [...]
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</section>
+</section>
+
+
+ </article>
+
+
+
+
+
+ <footer class="prev-next-footer d-print-none">
+
+<div class="prev-next-area">
+ <a class="left-prev"
+ href="agg_funcs.html"
+ title="previous page">
+ <i class="fa-solid fa-angle-left"></i>
+ <div class="prev-next-info">
+ <p class="prev-next-subtitle">previous</p>
+ <p class="prev-next-title">agg_funcs Expression Audits</p>
+ </div>
+ </a>
+ <a class="right-next"
+ href="bitwise_funcs.html"
+ title="next page">
+ <div class="prev-next-info">
+ <p class="prev-next-subtitle">next</p>
+ <p class="prev-next-title">bitwise_funcs Expression Audits</p>
+ </div>
+ <i class="fa-solid fa-angle-right"></i>
+ </a>
+</div>
+ </footer>
+
+ </div>
+
+
+
+
+ </div>
+ <footer class="bd-footer-content">
+
+ </footer>
+
+ </main>
+ </div>
+ </div>
+
+ <!-- Scripts loaded after <body> so the DOM is not blocked -->
+ <script defer src="../../_static/scripts/bootstrap.js?digest=8878045cc6db502f8baf"></script>
+<script defer src="../../_static/scripts/pydata-sphinx-theme.js?digest=8878045cc6db502f8baf"></script>
+
+<!-- Based on pydata_sphinx_theme/footer.html -->
+<footer class="footer mt-5 mt-md-0">
+ <div class="container">
+
+ <div class="footer-item">
+ <p>Apache DataFusion, Apache DataFusion Comet, Apache, the Apache feather logo, and the Apache DataFusion project logo</p>
+ <p>are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.</p>
+ </div>
+ </div>
+</footer>
+
+
+ </body>
+</html>
\ No newline at end of file
diff --git a/contributor-guide/benchmark-results/tpc-h.html b/contributor-guide/expression-audits/bitwise_funcs.html
similarity index 53%
copy from contributor-guide/benchmark-results/tpc-h.html
copy to contributor-guide/expression-audits/bitwise_funcs.html
index b75aa4ff2a..70c9dd06d8 100644
--- a/contributor-guide/benchmark-results/tpc-h.html
+++ b/contributor-guide/expression-audits/bitwise_funcs.html
@@ -27,7 +27,7 @@ under the License.
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
- <title>Apache DataFusion Comet: Benchmarks Derived From TPC-H — Apache DataFusion Comet documentation</title>
+ <title>bitwise_funcs Expression Audits — Apache DataFusion Comet documentation</title>
@@ -61,12 +61,12 @@ under the License.
<script src="../../_static/documentation_options.js?v=5929fcd5"></script>
<script src="../../_static/doctools.js?v=9a2dae69"></script>
<script src="../../_static/sphinx_highlight.js?v=dc90522c"></script>
- <script>DOCUMENTATION_OPTIONS.pagename = 'contributor-guide/benchmark-results/tpc-h';</script>
+ <script>DOCUMENTATION_OPTIONS.pagename = 'contributor-guide/expression-audits/bitwise_funcs';</script>
<script async="true" defer="true" src="https://buttons.github.io/buttons.js"></script>
<link rel="index" title="Index" href="../../genindex.html" />
<link rel="search" title="Search" href="../../search.html" />
- <link rel="next" title="Apache DataFusion Comet: Benchmarks Derived From TPC-DS" href="tpc-ds.html" />
- <link rel="prev" title="Comet Benchmarking Guide" href="../benchmarking.html" />
+ <link rel="next" title="collection_funcs Expression Audits" href="collection_funcs.html" />
+ <link rel="prev" title="array_funcs Expression Audits" href="array_funcs.html" />
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<meta name="docsearch:language" content="en"/>
<meta name="docsearch:version" content="" />
@@ -418,11 +418,11 @@ under the License.
<li class="toctree-l1"><a class="reference internal" href="../iceberg-spark-tests.html">Iceberg Spark Tests</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Debugging and Performance</span></p>
-<ul class="current nav bd-sidenav">
+<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../debugging.html">Debugging Guide</a></li>
-<li class="toctree-l1 current active has-children"><a class="reference internal" href="../benchmarking.html">Benchmarking Guide</a><details open="open"><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul class="current">
-<li class="toctree-l2 current active"><a class="current reference internal" href="#">Apache DataFusion Comet: Benchmarks Derived From TPC-H</a></li>
-<li class="toctree-l2"><a class="reference internal" href="tpc-ds.html">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</a></li>
+<li class="toctree-l1 has-children"><a class="reference internal" href="../benchmarking.html">Benchmarking Guide</a><details><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul>
+<li class="toctree-l2"><a class="reference internal" href="../benchmark-results/tpc-h.html">Apache DataFusion Comet: Benchmarks Derived From TPC-H</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../benchmark-results/tpc-ds.html">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_macos.html">Comet Benchmarking on macOS</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_aws_ec2.html">Comet Benchmarking in EC2</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_spark_sql_perf.html">TPC-DS Benchmarking with spark-sql-perf</a></li>
@@ -432,8 +432,27 @@ under the License.
<li class="toctree-l1"><a class="reference internal" href="../tracing.html">Tracing</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Reference</span></p>
-<ul class="nav bd-sidenav">
-<li class="toctree-l1"><a class="reference internal" href="../spark_expressions_support.html">Supported Spark Expressions</a></li>
+<ul class="current nav bd-sidenav">
+<li class="toctree-l1 current active has-children"><a class="reference internal" href="index.html">Expression Audits</a><details open="open"><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul class="current">
+<li class="toctree-l2"><a class="reference internal" href="agg_funcs.html">agg_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="array_funcs.html">array_funcs</a></li>
+<li class="toctree-l2 current active"><a class="current reference internal" href="#">bitwise_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="collection_funcs.html">collection_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="conditional_funcs.html">conditional_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="conversion_funcs.html">conversion_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="datetime_funcs.html">datetime_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="generator_funcs.html">generator_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="hash_funcs.html">hash_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="json_funcs.html">json_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="map_funcs.html">map_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="math_funcs.html">math_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="misc_funcs.html">misc_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="predicate_funcs.html">predicate_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="struct_funcs.html">struct_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="url_funcs.html">url_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="window_funcs.html">window_funcs</a></li>
+</ul>
+</details></li>
<li class="toctree-l1"><a class="reference internal" href="../spark_configs_support.html">Supported Spark Configurations</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Project Mechanics</span></p>
@@ -488,9 +507,9 @@ under the License.
<li class="breadcrumb-item"><a href="../index.html" class="nav-link">Comet Contributor Guide</a></li>
- <li class="breadcrumb-item"><a href="../benchmarking.html" class="nav-link">Comet Benchmarking Guide</a></li>
+ <li class="breadcrumb-item"><a href="index.html" class="nav-link">Expression Audits</a></li>
- <li class="breadcrumb-item active" aria-current="page"><span class="ellipsis">Apache DataFusion Comet: Benchmarks Derived From TPC-H</span></li>
+ <li class="breadcrumb-item active" aria-current="page"><span class="ellipsis">bitwise_funcs Expression Audits</span></li>
</ul>
</nav>
</div>
@@ -507,70 +526,127 @@ under the License.
<div id="searchbox"></div>
<article class="bd-article">
- <!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
+ <!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
-->
-<section id="apache-datafusion-comet-benchmarks-derived-from-tpc-h">
-<h1>Apache DataFusion Comet: Benchmarks Derived From TPC-H<a class="headerlink" href="#apache-datafusion-comet-benchmarks-derived-from-tpc-h" title="Link to this heading">#</a></h1>
-<p>The following benchmarks were performed on an EKS cluster (<code class="docutils literal notranslate"><span class="pre">r6i.24xlarge</span></code> instances with EBS storage) with data stored in S3.</p>
-<section id="benchmark-results">
-<h2>Benchmark Results<a class="headerlink" href="#benchmark-results" title="Link to this heading">#</a></h2>
-<p>Total time to run all queries (lower is better).</p>
-<p><img alt="" src="../../_images/tpch_allqueries_with_tuned.png" /></p>
-<p>The following charts are based on the tuned run using hash join.</p>
-<p>Per-query breakdown showing the relative performance of Spark and Comet.</p>
-<p><img alt="" src="../../_images/tpch_queries_compare.png" /></p>
-<p>How much Comet accelerates each query in relative terms.</p>
-<p><img alt="" src="../../_images/tpch_queries_speedup_rel.png" /></p>
-<p>How much Comet accelerates each query in absolute terms.</p>
-<p><img alt="" src="../../_images/tpch_queries_speedup_abs.png" /></p>
+<section id="bitwise-funcs-expression-audits">
+<h1>bitwise_funcs Expression Audits<a class="headerlink" href="#bitwise-funcs-expression-audits" title="Link to this heading">#</a></h1>
+<blockquote>
+<div><p>Audit notes for expressions in this category that have been audited. Absence of an entry means the expression has not been audited yet, not that it is unsupported. See the user guide <a class="reference internal" href="../../user-guide/latest/expressions.html"><span class="std std-doc">Spark Expression Support</span></a> for current support status.</p>
+</div></blockquote>
+<section id="id1">
+<h2><code class="docutils literal notranslate"><span class="pre">&</span></code><a class="headerlink" href="#id1" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">BitwiseAnd(left,</span> <span class="pre">right)</span> <span class="pre">extends</span> <span class="pre">BinaryArithmetic</span></code> over <code class="docutils literal notranslate"><span class="pre">IntegralType</span></code>. Comet routes via <code class="docutils literal notranslate"><span class="pre">CometBitwiseAnd</span></code> to the proto’s <code class="docutils lit [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): semantics unchanged; <code class="docutils literal notranslate"><span class="pre">NullIntolerant</span></code> trait replaced by <code class="docutils literal notranslate"><span class="pre">nullIntolerant:</span> <span class="pre">Boolean</span></code>.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
</section>
-<section id="configuration">
-<h2>Configuration<a class="headerlink" href="#configuration" title="Link to this heading">#</a></h2>
-<p>Common:</p>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.executor.instances</span><span class="o">=</span><span class="s">32</span>
-<span class="na">spark.executor.cores</span><span class="o">=</span><span class="s">16</span>
-<span class="na">spark.memory.fraction</span><span class="o">=</span><span class="s">0.6</span>
-<span class="na">spark.memory.storageFraction</span><span class="o">=</span><span class="s">0.2</span>
-<span class="c1"># Kubernetes CPU constraints</span>
-<span class="na">spark.kubernetes.executor.request.cores</span><span class="o">=</span><span class="s">8</span>
-<span class="na">spark.kubernetes.executor.limit.cores</span><span class="o">=</span><span class="s">8</span>
-</pre></div>
-</div>
-<p>Spark:</p>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.executor.memory</span><span class="o">=</span><span class="s">64G</span>
-<span class="na">spark.executor.memoryOverhead</span><span class="o">=</span><span class="s">10G</span>
-</pre></div>
-</div>
-<p>Comet:</p>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.executor.memory</span><span class="o">=</span><span class="s">32G</span>
-<span class="na">spark.executor.memoryOverhead</span><span class="o">=</span><span class="s">10G</span>
-<span class="na">spark.memory.offHeap.enabled</span><span class="o">=</span><span class="s">true</span>
-<span class="na">spark.memory.offHeap.size</span><span class="o">=</span><span class="s">32G</span>
-</pre></div>
-</div>
-<section id="comet-tuned">
-<h3>Comet (Tuned)<a class="headerlink" href="#comet-tuned" title="Link to this heading">#</a></h3>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.comet.exec.replaceSortMergeJoin</span><span class="o">=</span><span class="s">true</span>
-<span class="na">spark.comet.memoryPool.fraction</span><span class="o">=</span><span class="s">0.8</span>
-</pre></div>
-</div>
+<section id="id2">
+<h2><code class="docutils literal notranslate"><span class="pre"><<</span></code><a class="headerlink" href="#id2" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): only the function form <code class="docutils literal notranslate"><span class="pre">shiftleft(...)</span></code> exists; the <code class="docutils literal notranslate"><span class="pre"><<</span></code> operator alias is added in Spark 4.0.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre"><<</span></code> parses to <code class="docutils literal notranslate"><span class="pre">ShiftLeft</span></code> (the same <code class="docutils literal notranslate"><span class="pre">case</span> <span class="pre">class</span></code> as <code class="docutils literal notranslate"><span class="pre">shiftleft</span></code>) via a parser alias. <code class="docutils literal notranslate" [...]
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</section>
+<section id="id3">
+<h2><code class="docutils literal notranslate"><span class="pre">>></span></code><a class="headerlink" href="#id3" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): same as <code class="docutils literal notranslate"><span class="pre"><<</span></code> history — only the function form exists pre-4.0.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">>></span></code> parses to <code class="docutils literal notranslate"><span class="pre">ShiftRight</span></code>. Comet <code class="docutils literal notranslate"><span class="pre">CometShiftRight</span></code> mirrors the same operand-cast logic as <code class="docutils literal notranslate"><span class="pre">CometShiftLeft</span></code>.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</section>
+<section id="id4">
+<h2><code class="docutils literal notranslate"><span class="pre">^</span></code><a class="headerlink" href="#id4" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">BitwiseXor(left,</span> <span class="pre">right)</span> <span class="pre">extends</span> <span class="pre">BinaryArithmetic</span></code> over <code class="docutils literal notranslate"><span class="pre">IntegralType</span></code>. Comet routes via <code class="docutils literal notranslate"><span class="pre">CometBitwiseXor</span></code> to the proto’s <code class="docutils lit [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): semantics unchanged; <code class="docutils literal notranslate"><span class="pre">nullIntolerant</span></code> field refactor.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</section>
+<section id="bit-count">
+<h2>bit_count<a class="headerlink" href="#bit-count" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">BitwiseCount(child)</span> <span class="pre">extends</span> <span class="pre">UnaryExpression</span></code>; accepts <code class="docutils literal notranslate"><span class="pre">IntegralType</span></code> or <code class="docutils literal notranslate"><span class="pre">BooleanType</span></code> and returns the population count as <code class="docutils literal notranslate"><span [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): semantics unchanged.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
</section>
+<section id="bit-get">
+<h2>bit_get<a class="headerlink" href="#bit-get" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">BitwiseGet(left,</span> <span class="pre">right)</span> <span class="pre">extends</span> <span class="pre">BinaryExpression</span></code>; <code class="docutils literal notranslate"><span class="pre">inputTypes</span> <span class="pre">=</span> <span class="pre">(IntegralType,</span> <span class="pre">IntegerType)</span> <span class="pre">-></span> <span class="pre">ByteType [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): semantics unchanged.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</section>
+<section id="getbit">
+<h2>getbit<a class="headerlink" href="#getbit" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): registry alias of <code class="docutils literal notranslate"><span class="pre">BitwiseGet</span></code>. Same support as <code class="docutils literal notranslate"><span class="pre">bit_get</span></code>.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): identical to 3.4.3.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 3.4.3.</p></li>
+</ul>
+</section>
+<section id="shiftright">
+<h2>shiftright<a class="headerlink" href="#shiftright" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. Function name for <code class="docutils literal notranslate"><span class="pre">ShiftRight</span></code>. Same support as the <code class="docutils literal notranslate"><span class="pre">>></span></code> operator alias added in 4.0.</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): identical to 3.5.8 (the <code class="docutils literal notranslate"><span class="pre">>></span></code> operator is the only addition).</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</section>
+<section id="shiftrightunsigned">
+<h2>shiftrightunsigned<a class="headerlink" href="#shiftrightunsigned" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">ShiftRightUnsigned(left,</span> <span class="pre">right)</span> <span class="pre">extends</span> <span class="pre">BinaryExpression</span></code>; uses Java’s <code class="docutils literal notranslate"><span class="pre">>>></span></code> (zero-fill) semantics. Comet wires as <code class="docutils literal notranslate"><span class="pre">CometScalarFunction("shiftrig [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): now extends the shared <code class="docutils literal notranslate"><span class="pre">BitShiftOperation</span></code> trait; eval semantics unchanged. The <code class="docutils literal notranslate"><span class="pre">>>></span></code> operator alias is added at the parser level.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</section>
+<section id="id5">
+<h2><code class="docutils literal notranslate"><span class="pre">|</span></code><a class="headerlink" href="#id5" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">BitwiseOr(left,</span> <span class="pre">right)</span> <span class="pre">extends</span> <span class="pre">BinaryArithmetic</span></code> over <code class="docutils literal notranslate"><span class="pre">IntegralType</span></code>. Comet routes via <code class="docutils literal notranslate"><span class="pre">CometBitwiseOr</span></code> to the proto’s <code class="docutils liter [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): semantics unchanged; <code class="docutils literal notranslate"><span class="pre">nullIntolerant</span></code> field refactor.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</section>
+<section id="id6">
+<h2><code class="docutils literal notranslate"><span class="pre">~</span></code><a class="headerlink" href="#id6" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">BitwiseNot(child)</span> <span class="pre">extends</span> <span class="pre">UnaryExpression</span></code>; accepts <code class="docutils literal notranslate"><span class="pre">IntegralType</span></code> and returns the bitwise complement of the same type. Comet routes via <code class="docutils literal notranslate"><span class="pre">CometBitwiseNot</span></code> to the native <c [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): semantics unchanged; <code class="docutils literal notranslate"><span class="pre">nullIntolerant</span></code> field refactor.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
</section>
</section>
@@ -585,20 +661,20 @@ under the License.
<div class="prev-next-area">
<a class="left-prev"
- href="../benchmarking.html"
+ href="array_funcs.html"
title="previous page">
<i class="fa-solid fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">Comet Benchmarking Guide</p>
+ <p class="prev-next-title">array_funcs Expression Audits</p>
</div>
</a>
<a class="right-next"
- href="tpc-ds.html"
+ href="collection_funcs.html"
title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
- <p class="prev-next-title">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</p>
+ <p class="prev-next-title">collection_funcs Expression Audits</p>
</div>
<i class="fa-solid fa-angle-right"></i>
</a>
diff --git a/contributor-guide/benchmark-results/tpc-h.html b/contributor-guide/expression-audits/collection_funcs.html
similarity index 62%
copy from contributor-guide/benchmark-results/tpc-h.html
copy to contributor-guide/expression-audits/collection_funcs.html
index b75aa4ff2a..74a179c23a 100644
--- a/contributor-guide/benchmark-results/tpc-h.html
+++ b/contributor-guide/expression-audits/collection_funcs.html
@@ -27,7 +27,7 @@ under the License.
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
- <title>Apache DataFusion Comet: Benchmarks Derived From TPC-H — Apache DataFusion Comet documentation</title>
+ <title>collection_funcs Expression Audits — Apache DataFusion Comet documentation</title>
@@ -61,12 +61,12 @@ under the License.
<script src="../../_static/documentation_options.js?v=5929fcd5"></script>
<script src="../../_static/doctools.js?v=9a2dae69"></script>
<script src="../../_static/sphinx_highlight.js?v=dc90522c"></script>
- <script>DOCUMENTATION_OPTIONS.pagename = 'contributor-guide/benchmark-results/tpc-h';</script>
+ <script>DOCUMENTATION_OPTIONS.pagename = 'contributor-guide/expression-audits/collection_funcs';</script>
<script async="true" defer="true" src="https://buttons.github.io/buttons.js"></script>
<link rel="index" title="Index" href="../../genindex.html" />
<link rel="search" title="Search" href="../../search.html" />
- <link rel="next" title="Apache DataFusion Comet: Benchmarks Derived From TPC-DS" href="tpc-ds.html" />
- <link rel="prev" title="Comet Benchmarking Guide" href="../benchmarking.html" />
+ <link rel="next" title="conditional_funcs Expression Audits" href="conditional_funcs.html" />
+ <link rel="prev" title="bitwise_funcs Expression Audits" href="bitwise_funcs.html" />
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<meta name="docsearch:language" content="en"/>
<meta name="docsearch:version" content="" />
@@ -418,11 +418,11 @@ under the License.
<li class="toctree-l1"><a class="reference internal" href="../iceberg-spark-tests.html">Iceberg Spark Tests</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Debugging and Performance</span></p>
-<ul class="current nav bd-sidenav">
+<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../debugging.html">Debugging Guide</a></li>
-<li class="toctree-l1 current active has-children"><a class="reference internal" href="../benchmarking.html">Benchmarking Guide</a><details open="open"><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul class="current">
-<li class="toctree-l2 current active"><a class="current reference internal" href="#">Apache DataFusion Comet: Benchmarks Derived From TPC-H</a></li>
-<li class="toctree-l2"><a class="reference internal" href="tpc-ds.html">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</a></li>
+<li class="toctree-l1 has-children"><a class="reference internal" href="../benchmarking.html">Benchmarking Guide</a><details><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul>
+<li class="toctree-l2"><a class="reference internal" href="../benchmark-results/tpc-h.html">Apache DataFusion Comet: Benchmarks Derived From TPC-H</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../benchmark-results/tpc-ds.html">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_macos.html">Comet Benchmarking on macOS</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_aws_ec2.html">Comet Benchmarking in EC2</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_spark_sql_perf.html">TPC-DS Benchmarking with spark-sql-perf</a></li>
@@ -432,8 +432,27 @@ under the License.
<li class="toctree-l1"><a class="reference internal" href="../tracing.html">Tracing</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Reference</span></p>
-<ul class="nav bd-sidenav">
-<li class="toctree-l1"><a class="reference internal" href="../spark_expressions_support.html">Supported Spark Expressions</a></li>
+<ul class="current nav bd-sidenav">
+<li class="toctree-l1 current active has-children"><a class="reference internal" href="index.html">Expression Audits</a><details open="open"><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul class="current">
+<li class="toctree-l2"><a class="reference internal" href="agg_funcs.html">agg_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="array_funcs.html">array_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="bitwise_funcs.html">bitwise_funcs</a></li>
+<li class="toctree-l2 current active"><a class="current reference internal" href="#">collection_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="conditional_funcs.html">conditional_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="conversion_funcs.html">conversion_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="datetime_funcs.html">datetime_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="generator_funcs.html">generator_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="hash_funcs.html">hash_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="json_funcs.html">json_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="map_funcs.html">map_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="math_funcs.html">math_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="misc_funcs.html">misc_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="predicate_funcs.html">predicate_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="struct_funcs.html">struct_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="url_funcs.html">url_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="window_funcs.html">window_funcs</a></li>
+</ul>
+</details></li>
<li class="toctree-l1"><a class="reference internal" href="../spark_configs_support.html">Supported Spark Configurations</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Project Mechanics</span></p>
@@ -488,9 +507,9 @@ under the License.
<li class="breadcrumb-item"><a href="../index.html" class="nav-link">Comet Contributor Guide</a></li>
- <li class="breadcrumb-item"><a href="../benchmarking.html" class="nav-link">Comet Benchmarking Guide</a></li>
+ <li class="breadcrumb-item"><a href="index.html" class="nav-link">Expression Audits</a></li>
- <li class="breadcrumb-item active" aria-current="page"><span class="ellipsis">Apache DataFusion Comet: Benchmarks Derived From TPC-H</span></li>
+ <li class="breadcrumb-item active" aria-current="page"><span class="ellipsis">collection_funcs Expression Audits</span></li>
</ul>
</nav>
</div>
@@ -507,70 +526,64 @@ under the License.
<div id="searchbox"></div>
<article class="bd-article">
- <!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
+ <!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
-->
-<section id="apache-datafusion-comet-benchmarks-derived-from-tpc-h">
-<h1>Apache DataFusion Comet: Benchmarks Derived From TPC-H<a class="headerlink" href="#apache-datafusion-comet-benchmarks-derived-from-tpc-h" title="Link to this heading">#</a></h1>
-<p>The following benchmarks were performed on an EKS cluster (<code class="docutils literal notranslate"><span class="pre">r6i.24xlarge</span></code> instances with EBS storage) with data stored in S3.</p>
-<section id="benchmark-results">
-<h2>Benchmark Results<a class="headerlink" href="#benchmark-results" title="Link to this heading">#</a></h2>
-<p>Total time to run all queries (lower is better).</p>
-<p><img alt="" src="../../_images/tpch_allqueries_with_tuned.png" /></p>
-<p>The following charts are based on the tuned run using hash join.</p>
-<p>Per-query breakdown showing the relative performance of Spark and Comet.</p>
-<p><img alt="" src="../../_images/tpch_queries_compare.png" /></p>
-<p>How much Comet accelerates each query in relative terms.</p>
-<p><img alt="" src="../../_images/tpch_queries_speedup_rel.png" /></p>
-<p>How much Comet accelerates each query in absolute terms.</p>
-<p><img alt="" src="../../_images/tpch_queries_speedup_abs.png" /></p>
+<section id="collection-funcs-expression-audits">
+<h1>collection_funcs Expression Audits<a class="headerlink" href="#collection-funcs-expression-audits" title="Link to this heading">#</a></h1>
+<blockquote>
+<div><p>Audit notes for expressions in this category that have been audited. Absence of an entry means the expression has not been audited yet, not that it is unsupported. See the user guide <a class="reference internal" href="../../user-guide/latest/expressions.html"><span class="std std-doc">Spark Expression Support</span></a> for current support status.</p>
+</div></blockquote>
+<section id="array-size">
+<h2>array_size<a class="headerlink" href="#array-size" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Native via <code class="docutils literal notranslate"><span class="pre">size</span></code>; returns -1 instead of NULL for NULL input (https://github.com/apache/datafusion-comet/issues/4560).</p></li>
+</ul>
</section>
-<section id="configuration">
-<h2>Configuration<a class="headerlink" href="#configuration" title="Link to this heading">#</a></h2>
-<p>Common:</p>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.executor.instances</span><span class="o">=</span><span class="s">32</span>
-<span class="na">spark.executor.cores</span><span class="o">=</span><span class="s">16</span>
-<span class="na">spark.memory.fraction</span><span class="o">=</span><span class="s">0.6</span>
-<span class="na">spark.memory.storageFraction</span><span class="o">=</span><span class="s">0.2</span>
-<span class="c1"># Kubernetes CPU constraints</span>
-<span class="na">spark.kubernetes.executor.request.cores</span><span class="o">=</span><span class="s">8</span>
-<span class="na">spark.kubernetes.executor.limit.cores</span><span class="o">=</span><span class="s">8</span>
-</pre></div>
-</div>
-<p>Spark:</p>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.executor.memory</span><span class="o">=</span><span class="s">64G</span>
-<span class="na">spark.executor.memoryOverhead</span><span class="o">=</span><span class="s">10G</span>
-</pre></div>
-</div>
-<p>Comet:</p>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.executor.memory</span><span class="o">=</span><span class="s">32G</span>
-<span class="na">spark.executor.memoryOverhead</span><span class="o">=</span><span class="s">10G</span>
-<span class="na">spark.memory.offHeap.enabled</span><span class="o">=</span><span class="s">true</span>
-<span class="na">spark.memory.offHeap.size</span><span class="o">=</span><span class="s">32G</span>
-</pre></div>
-</div>
-<section id="comet-tuned">
-<h3>Comet (Tuned)<a class="headerlink" href="#comet-tuned" title="Link to this heading">#</a></h3>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.comet.exec.replaceSortMergeJoin</span><span class="o">=</span><span class="s">true</span>
-<span class="na">spark.comet.memoryPool.fraction</span><span class="o">=</span><span class="s">0.8</span>
-</pre></div>
-</div>
+<section id="concat">
+<h2>concat<a class="headerlink" href="#concat" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">Concat(children)</span> <span class="pre">extends</span> <span class="pre">ComplexTypeMergingExpression</span> <span class="pre">with</span> <span class="pre">QueryErrorsBase</span></code>; <code class="docutils literal notranslate"><span class="pre">allowedTypes</span> <span class="pre">=</span> <span class="pre">Seq(StringType,</span> <span class="pre">BinaryType,</span> <spa [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">allowedTypes</span></code> widens <code class="docutils literal notranslate"><span class="pre">StringType</span></code> to <code class="docutils literal notranslate"><span class="pre">StringTypeWithCollation(supportsTrimCollation</span> <span class="pre">=</span> <span class="pre">true)</span></code>. Error-formatting helper changes from <code class="docutils literal notranslate"><span c [...]
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+<li><p>Known limitation: Comet only supports <code class="docutils literal notranslate"><span class="pre">StringType</span></code> children natively; <code class="docutils literal notranslate"><span class="pre">BinaryType</span></code> and <code class="docutils literal notranslate"><span class="pre">ArrayType</span></code> inputs fall back to Spark (https://github.com/apache/datafusion-comet/issues/4471). Non-default Spark 4.0 string collations are not propagated (https://github.com/apac [...]
+</ul>
+</section>
+<section id="reverse">
+<h2>reverse<a class="headerlink" href="#reverse" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">Reverse(child)</span> <span class="pre">extends</span> <span class="pre">UnaryExpression</span> <span class="pre">with</span> <span class="pre">ImplicitCastInputTypes</span> <span class="pre">with</span> <span class="pre">NullIntolerant</span></code>; <code class="docutils literal notranslate"><span class="pre">inputTypes</span> <span class="pre">=</span> <span class="pre">Seq( [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">NullIntolerant</span></code> trait replaced by <code class="docutils literal notranslate"><span class="pre">override</span> <span class="pre">def</span> <span class="pre">nullIntolerant:</span> <span class="pre">Boolean</span> <span class="pre">=</span> <span class="pre">true</span></code>; <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code> widened to < [...]
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+<li><p>Known limitation: <code class="docutils literal notranslate"><span class="pre">Reverse</span></code> on an array containing <code class="docutils literal notranslate"><span class="pre">BinaryType</span></code> elements is reported as <code class="docutils literal notranslate"><span class="pre">Incompatible</span></code> and falls back unless explicitly enabled (https://github.com/apache/datafusion-comet/issues/2763).</p></li>
+</ul>
</section>
+<section id="size">
+<h2>size<a class="headerlink" href="#size" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">Size(child,</span> <span class="pre">legacySizeOfNull)</span> <span class="pre">extends</span> <span class="pre">UnaryExpression</span> <span class="pre">with</span> <span class="pre">ExpectsInputTypes</span></code>; <code class="docutils literal notranslate"><span class="pre">inputTypes</span> <span class="pre">=</span> <span class="pre">Seq(TypeCollection(ArrayType,</span> <s [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): byte-for-byte identical to 3.5.8.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): byte-for-byte identical to 3.5.8.</p></li>
+<li><p>Known limitation: <code class="docutils literal notranslate"><span class="pre">Size</span></code> over <code class="docutils literal notranslate"><span class="pre">MapType</span></code> falls back to Spark (https://github.com/apache/datafusion-comet/issues/4472).</p></li>
+</ul>
</section>
</section>
@@ -585,20 +598,20 @@ under the License.
<div class="prev-next-area">
<a class="left-prev"
- href="../benchmarking.html"
+ href="bitwise_funcs.html"
title="previous page">
<i class="fa-solid fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">Comet Benchmarking Guide</p>
+ <p class="prev-next-title">bitwise_funcs Expression Audits</p>
</div>
</a>
<a class="right-next"
- href="tpc-ds.html"
+ href="conditional_funcs.html"
title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
- <p class="prev-next-title">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</p>
+ <p class="prev-next-title">conditional_funcs Expression Audits</p>
</div>
<i class="fa-solid fa-angle-right"></i>
</a>
diff --git a/contributor-guide/benchmark-results/tpc-h.html b/contributor-guide/expression-audits/conditional_funcs.html
similarity index 60%
copy from contributor-guide/benchmark-results/tpc-h.html
copy to contributor-guide/expression-audits/conditional_funcs.html
index b75aa4ff2a..9e6e7fd1c3 100644
--- a/contributor-guide/benchmark-results/tpc-h.html
+++ b/contributor-guide/expression-audits/conditional_funcs.html
@@ -27,7 +27,7 @@ under the License.
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
- <title>Apache DataFusion Comet: Benchmarks Derived From TPC-H — Apache DataFusion Comet documentation</title>
+ <title>conditional_funcs Expression Audits — Apache DataFusion Comet documentation</title>
@@ -61,12 +61,12 @@ under the License.
<script src="../../_static/documentation_options.js?v=5929fcd5"></script>
<script src="../../_static/doctools.js?v=9a2dae69"></script>
<script src="../../_static/sphinx_highlight.js?v=dc90522c"></script>
- <script>DOCUMENTATION_OPTIONS.pagename = 'contributor-guide/benchmark-results/tpc-h';</script>
+ <script>DOCUMENTATION_OPTIONS.pagename = 'contributor-guide/expression-audits/conditional_funcs';</script>
<script async="true" defer="true" src="https://buttons.github.io/buttons.js"></script>
<link rel="index" title="Index" href="../../genindex.html" />
<link rel="search" title="Search" href="../../search.html" />
- <link rel="next" title="Apache DataFusion Comet: Benchmarks Derived From TPC-DS" href="tpc-ds.html" />
- <link rel="prev" title="Comet Benchmarking Guide" href="../benchmarking.html" />
+ <link rel="next" title="conversion_funcs Expression Audits" href="conversion_funcs.html" />
+ <link rel="prev" title="collection_funcs Expression Audits" href="collection_funcs.html" />
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<meta name="docsearch:language" content="en"/>
<meta name="docsearch:version" content="" />
@@ -418,11 +418,11 @@ under the License.
<li class="toctree-l1"><a class="reference internal" href="../iceberg-spark-tests.html">Iceberg Spark Tests</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Debugging and Performance</span></p>
-<ul class="current nav bd-sidenav">
+<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../debugging.html">Debugging Guide</a></li>
-<li class="toctree-l1 current active has-children"><a class="reference internal" href="../benchmarking.html">Benchmarking Guide</a><details open="open"><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul class="current">
-<li class="toctree-l2 current active"><a class="current reference internal" href="#">Apache DataFusion Comet: Benchmarks Derived From TPC-H</a></li>
-<li class="toctree-l2"><a class="reference internal" href="tpc-ds.html">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</a></li>
+<li class="toctree-l1 has-children"><a class="reference internal" href="../benchmarking.html">Benchmarking Guide</a><details><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul>
+<li class="toctree-l2"><a class="reference internal" href="../benchmark-results/tpc-h.html">Apache DataFusion Comet: Benchmarks Derived From TPC-H</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../benchmark-results/tpc-ds.html">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_macos.html">Comet Benchmarking on macOS</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_aws_ec2.html">Comet Benchmarking in EC2</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_spark_sql_perf.html">TPC-DS Benchmarking with spark-sql-perf</a></li>
@@ -432,8 +432,27 @@ under the License.
<li class="toctree-l1"><a class="reference internal" href="../tracing.html">Tracing</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Reference</span></p>
-<ul class="nav bd-sidenav">
-<li class="toctree-l1"><a class="reference internal" href="../spark_expressions_support.html">Supported Spark Expressions</a></li>
+<ul class="current nav bd-sidenav">
+<li class="toctree-l1 current active has-children"><a class="reference internal" href="index.html">Expression Audits</a><details open="open"><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul class="current">
+<li class="toctree-l2"><a class="reference internal" href="agg_funcs.html">agg_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="array_funcs.html">array_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="bitwise_funcs.html">bitwise_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="collection_funcs.html">collection_funcs</a></li>
+<li class="toctree-l2 current active"><a class="current reference internal" href="#">conditional_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="conversion_funcs.html">conversion_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="datetime_funcs.html">datetime_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="generator_funcs.html">generator_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="hash_funcs.html">hash_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="json_funcs.html">json_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="map_funcs.html">map_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="math_funcs.html">math_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="misc_funcs.html">misc_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="predicate_funcs.html">predicate_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="struct_funcs.html">struct_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="url_funcs.html">url_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="window_funcs.html">window_funcs</a></li>
+</ul>
+</details></li>
<li class="toctree-l1"><a class="reference internal" href="../spark_configs_support.html">Supported Spark Configurations</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Project Mechanics</span></p>
@@ -488,9 +507,9 @@ under the License.
<li class="breadcrumb-item"><a href="../index.html" class="nav-link">Comet Contributor Guide</a></li>
- <li class="breadcrumb-item"><a href="../benchmarking.html" class="nav-link">Comet Benchmarking Guide</a></li>
+ <li class="breadcrumb-item"><a href="index.html" class="nav-link">Expression Audits</a></li>
- <li class="breadcrumb-item active" aria-current="page"><span class="ellipsis">Apache DataFusion Comet: Benchmarks Derived From TPC-H</span></li>
+ <li class="breadcrumb-item active" aria-current="page"><span class="ellipsis">conditional_funcs Expression Audits</span></li>
</ul>
</nav>
</div>
@@ -507,70 +526,91 @@ under the License.
<div id="searchbox"></div>
<article class="bd-article">
- <!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
+ <!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
-->
-<section id="apache-datafusion-comet-benchmarks-derived-from-tpc-h">
-<h1>Apache DataFusion Comet: Benchmarks Derived From TPC-H<a class="headerlink" href="#apache-datafusion-comet-benchmarks-derived-from-tpc-h" title="Link to this heading">#</a></h1>
-<p>The following benchmarks were performed on an EKS cluster (<code class="docutils literal notranslate"><span class="pre">r6i.24xlarge</span></code> instances with EBS storage) with data stored in S3.</p>
-<section id="benchmark-results">
-<h2>Benchmark Results<a class="headerlink" href="#benchmark-results" title="Link to this heading">#</a></h2>
-<p>Total time to run all queries (lower is better).</p>
-<p><img alt="" src="../../_images/tpch_allqueries_with_tuned.png" /></p>
-<p>The following charts are based on the tuned run using hash join.</p>
-<p>Per-query breakdown showing the relative performance of Spark and Comet.</p>
-<p><img alt="" src="../../_images/tpch_queries_compare.png" /></p>
-<p>How much Comet accelerates each query in relative terms.</p>
-<p><img alt="" src="../../_images/tpch_queries_speedup_rel.png" /></p>
-<p>How much Comet accelerates each query in absolute terms.</p>
-<p><img alt="" src="../../_images/tpch_queries_speedup_abs.png" /></p>
+<section id="conditional-funcs-expression-audits">
+<h1>conditional_funcs Expression Audits<a class="headerlink" href="#conditional-funcs-expression-audits" title="Link to this heading">#</a></h1>
+<blockquote>
+<div><p>Audit notes for expressions in this category that have been audited. Absence of an entry means the expression has not been audited yet, not that it is unsupported. See the user guide <a class="reference internal" href="../../user-guide/latest/expressions.html"><span class="std std-doc">Spark Expression Support</span></a> for current support status.</p>
+</div></blockquote>
+<section id="coalesce">
+<h2>coalesce<a class="headerlink" href="#coalesce" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">Coalesce(children)</span> <span class="pre">extends</span> <span class="pre">Expression</span> <span class="pre">with</span> <span class="pre">ComplexTypeMergingExpression</span></code>; returns the first non-null child, evaluated left-to-right with short-circuit. Result type is the merged child type. Comet routes via <code class="docutils literal notranslate"><span class="pre" [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): byte-for-byte identical to 3.5.8.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): byte-for-byte identical to 3.5.8.</p></li>
+</ul>
</section>
-<section id="configuration">
-<h2>Configuration<a class="headerlink" href="#configuration" title="Link to this heading">#</a></h2>
-<p>Common:</p>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.executor.instances</span><span class="o">=</span><span class="s">32</span>
-<span class="na">spark.executor.cores</span><span class="o">=</span><span class="s">16</span>
-<span class="na">spark.memory.fraction</span><span class="o">=</span><span class="s">0.6</span>
-<span class="na">spark.memory.storageFraction</span><span class="o">=</span><span class="s">0.2</span>
-<span class="c1"># Kubernetes CPU constraints</span>
-<span class="na">spark.kubernetes.executor.request.cores</span><span class="o">=</span><span class="s">8</span>
-<span class="na">spark.kubernetes.executor.limit.cores</span><span class="o">=</span><span class="s">8</span>
-</pre></div>
-</div>
-<p>Spark:</p>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.executor.memory</span><span class="o">=</span><span class="s">64G</span>
-<span class="na">spark.executor.memoryOverhead</span><span class="o">=</span><span class="s">10G</span>
-</pre></div>
-</div>
-<p>Comet:</p>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.executor.memory</span><span class="o">=</span><span class="s">32G</span>
-<span class="na">spark.executor.memoryOverhead</span><span class="o">=</span><span class="s">10G</span>
-<span class="na">spark.memory.offHeap.enabled</span><span class="o">=</span><span class="s">true</span>
-<span class="na">spark.memory.offHeap.size</span><span class="o">=</span><span class="s">32G</span>
-</pre></div>
-</div>
-<section id="comet-tuned">
-<h3>Comet (Tuned)<a class="headerlink" href="#comet-tuned" title="Link to this heading">#</a></h3>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.comet.exec.replaceSortMergeJoin</span><span class="o">=</span><span class="s">true</span>
-<span class="na">spark.comet.memoryPool.fraction</span><span class="o">=</span><span class="s">0.8</span>
-</pre></div>
-</div>
+<section id="if">
+<h2>if<a class="headerlink" href="#if" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">If(predicate,</span> <span class="pre">trueValue,</span> <span class="pre">falseValue)</span> <span class="pre">extends</span> <span class="pre">ComplexTypeMergingExpression</span></code>; standard ternary semantics with short-circuit, predicate must be <code class="docutils literal notranslate"><span class="pre">BooleanType</span></code>. Comet routes via <code class="docutils [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): adds an <code class="docutils literal notranslate"><span class="pre">override</span> <span class="pre">def</span> <span class="pre">withNewAlwaysEvaluatedInputs(...)</span></code> hook for the new optimizer phase; semantics unchanged. Error messages reformatted (<code class="docutils literal notranslate"><span class="pre">paramIndex</span></code> -> <code class="docutils literal notranslate"><span class="pre">ordinalNumber</span></code>).</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
</section>
+<section id="ifnull">
+<h2>ifnull<a class="headerlink" href="#ifnull" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): registry alias of <code class="docutils literal notranslate"><span class="pre">Nvl</span></code> (<code class="docutils literal notranslate"><span class="pre">expression[Nvl]("ifnull",</span> <span class="pre">setAlias</span> <span class="pre">=</span> <span class="pre">true)</span></code>). Same <code class="docutils literal notranslate"><span class="pre">RuntimeReplaceable</span></code> lowering as <code class="docutils literal notrans [...]
+<li><p>Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): identical to 3.4.3.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 3.4.3.</p></li>
+</ul>
+</section>
+<section id="nullif">
+<h2>nullif<a class="headerlink" href="#nullif" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">NullIf(left,</span> <span class="pre">right,</span> <span class="pre">replacement)</span> <span class="pre">extends</span> <span class="pre">RuntimeReplaceable</span> <span class="pre">with</span> <span class="pre">InheritAnalysisRules</span></code>; the analyzer rewrites to <code class="docutils literal notranslate"><span class="pre">If(EqualTo(left,</span> <span class="pre">right),</sp [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 3.5.8.</p></li>
+</ul>
+</section>
+<section id="nvl">
+<h2>nvl<a class="headerlink" href="#nvl" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">Nvl(left,</span> <span class="pre">right,</span> <span class="pre">replacement)</span> <span class="pre">extends</span> <span class="pre">RuntimeReplaceable</span></code>; analyzer rewrites to <code class="docutils literal notranslate"><span class="pre">Coalesce(Seq(left,</span> <span class="pre">right))</span></code>. Comet handles via <code class="docutils literal notranslate"><span cl [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 3.5.8.</p></li>
+</ul>
+</section>
+<section id="nvl2">
+<h2>nvl2<a class="headerlink" href="#nvl2" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">Nvl2(expr1,</span> <span class="pre">expr2,</span> <span class="pre">expr3,</span> <span class="pre">replacement)</span> <span class="pre">extends</span> <span class="pre">RuntimeReplaceable</span></code>; analyzer rewrites to <code class="docutils literal notranslate"><span class="pre">If(IsNotNull(expr1),</span> <span class="pre">expr2,</span> <span class="pre">expr3)</span></code>. Co [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 3.5.8.</p></li>
+</ul>
+</section>
+<section id="when">
+<h2>when<a class="headerlink" href="#when" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): the <code class="docutils literal notranslate"><span class="pre">CASE</span> <span class="pre">WHEN</span> <span class="pre">...</span> <span class="pre">THEN</span> <span class="pre">...</span></code> SQL form lowers to <code class="docutils literal notranslate"><span class="pre">CaseWhen(branches:</span> <span class="pre">Seq[(Expression,</span> <span class="pre">Expression)],</span> <span class="pre">elseValue:</span> <span class="pre">Option[E [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): adds the <code class="docutils literal notranslate"><span class="pre">withNewAlwaysEvaluatedInputs</span></code> optimizer hook; semantics unchanged.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
</section>
</section>
@@ -585,20 +625,20 @@ under the License.
<div class="prev-next-area">
<a class="left-prev"
- href="../benchmarking.html"
+ href="collection_funcs.html"
title="previous page">
<i class="fa-solid fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">Comet Benchmarking Guide</p>
+ <p class="prev-next-title">collection_funcs Expression Audits</p>
</div>
</a>
<a class="right-next"
- href="tpc-ds.html"
+ href="conversion_funcs.html"
title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
- <p class="prev-next-title">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</p>
+ <p class="prev-next-title">conversion_funcs Expression Audits</p>
</div>
<i class="fa-solid fa-angle-right"></i>
</a>
diff --git a/contributor-guide/benchmark-results/tpc-ds.html b/contributor-guide/expression-audits/conversion_funcs.html
similarity index 58%
copy from contributor-guide/benchmark-results/tpc-ds.html
copy to contributor-guide/expression-audits/conversion_funcs.html
index 4d5bf9b1c4..508a1a5cb6 100644
--- a/contributor-guide/benchmark-results/tpc-ds.html
+++ b/contributor-guide/expression-audits/conversion_funcs.html
@@ -27,7 +27,7 @@ under the License.
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
- <title>Apache DataFusion Comet: Benchmarks Derived From TPC-DS — Apache DataFusion Comet documentation</title>
+ <title>conversion_funcs Expression Audits — Apache DataFusion Comet documentation</title>
@@ -61,12 +61,12 @@ under the License.
<script src="../../_static/documentation_options.js?v=5929fcd5"></script>
<script src="../../_static/doctools.js?v=9a2dae69"></script>
<script src="../../_static/sphinx_highlight.js?v=dc90522c"></script>
- <script>DOCUMENTATION_OPTIONS.pagename = 'contributor-guide/benchmark-results/tpc-ds';</script>
+ <script>DOCUMENTATION_OPTIONS.pagename = 'contributor-guide/expression-audits/conversion_funcs';</script>
<script async="true" defer="true" src="https://buttons.github.io/buttons.js"></script>
<link rel="index" title="Index" href="../../genindex.html" />
<link rel="search" title="Search" href="../../search.html" />
- <link rel="next" title="Comet Benchmarking on macOS" href="../benchmarking_macos.html" />
- <link rel="prev" title="Apache DataFusion Comet: Benchmarks Derived From TPC-H" href="tpc-h.html" />
+ <link rel="next" title="datetime_funcs Expression Audits" href="datetime_funcs.html" />
+ <link rel="prev" title="conditional_funcs Expression Audits" href="conditional_funcs.html" />
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<meta name="docsearch:language" content="en"/>
<meta name="docsearch:version" content="" />
@@ -418,11 +418,11 @@ under the License.
<li class="toctree-l1"><a class="reference internal" href="../iceberg-spark-tests.html">Iceberg Spark Tests</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Debugging and Performance</span></p>
-<ul class="current nav bd-sidenav">
+<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../debugging.html">Debugging Guide</a></li>
-<li class="toctree-l1 current active has-children"><a class="reference internal" href="../benchmarking.html">Benchmarking Guide</a><details open="open"><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul class="current">
-<li class="toctree-l2"><a class="reference internal" href="tpc-h.html">Apache DataFusion Comet: Benchmarks Derived From TPC-H</a></li>
-<li class="toctree-l2 current active"><a class="current reference internal" href="#">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</a></li>
+<li class="toctree-l1 has-children"><a class="reference internal" href="../benchmarking.html">Benchmarking Guide</a><details><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul>
+<li class="toctree-l2"><a class="reference internal" href="../benchmark-results/tpc-h.html">Apache DataFusion Comet: Benchmarks Derived From TPC-H</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../benchmark-results/tpc-ds.html">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_macos.html">Comet Benchmarking on macOS</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_aws_ec2.html">Comet Benchmarking in EC2</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_spark_sql_perf.html">TPC-DS Benchmarking with spark-sql-perf</a></li>
@@ -432,8 +432,27 @@ under the License.
<li class="toctree-l1"><a class="reference internal" href="../tracing.html">Tracing</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Reference</span></p>
-<ul class="nav bd-sidenav">
-<li class="toctree-l1"><a class="reference internal" href="../spark_expressions_support.html">Supported Spark Expressions</a></li>
+<ul class="current nav bd-sidenav">
+<li class="toctree-l1 current active has-children"><a class="reference internal" href="index.html">Expression Audits</a><details open="open"><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul class="current">
+<li class="toctree-l2"><a class="reference internal" href="agg_funcs.html">agg_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="array_funcs.html">array_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="bitwise_funcs.html">bitwise_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="collection_funcs.html">collection_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="conditional_funcs.html">conditional_funcs</a></li>
+<li class="toctree-l2 current active"><a class="current reference internal" href="#">conversion_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="datetime_funcs.html">datetime_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="generator_funcs.html">generator_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="hash_funcs.html">hash_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="json_funcs.html">json_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="map_funcs.html">map_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="math_funcs.html">math_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="misc_funcs.html">misc_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="predicate_funcs.html">predicate_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="struct_funcs.html">struct_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="url_funcs.html">url_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="window_funcs.html">window_funcs</a></li>
+</ul>
+</details></li>
<li class="toctree-l1"><a class="reference internal" href="../spark_configs_support.html">Supported Spark Configurations</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Project Mechanics</span></p>
@@ -488,9 +507,9 @@ under the License.
<li class="breadcrumb-item"><a href="../index.html" class="nav-link">Comet Contributor Guide</a></li>
- <li class="breadcrumb-item"><a href="../benchmarking.html" class="nav-link">Comet Benchmarking Guide</a></li>
+ <li class="breadcrumb-item"><a href="index.html" class="nav-link">Expression Audits</a></li>
- <li class="breadcrumb-item active" aria-current="page"><span class="ellipsis">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</span></li>
+ <li class="breadcrumb-item active" aria-current="page"><span class="ellipsis">conversion_funcs Expression Audits</span></li>
</ul>
</nav>
</div>
@@ -507,63 +526,48 @@ under the License.
<div id="searchbox"></div>
<article class="bd-article">
- <!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
+ <!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
-->
-<section id="apache-datafusion-comet-benchmarks-derived-from-tpc-ds">
-<h1>Apache DataFusion Comet: Benchmarks Derived From TPC-DS<a class="headerlink" href="#apache-datafusion-comet-benchmarks-derived-from-tpc-ds" title="Link to this heading">#</a></h1>
-<p>The following benchmarks were performed on an EKS cluster (<code class="docutils literal notranslate"><span class="pre">r6i.24xlarge</span></code> instances with EBS storage) with data stored in S3.</p>
-<section id="benchmark-results">
-<h2>Benchmark Results<a class="headerlink" href="#benchmark-results" title="Link to this heading">#</a></h2>
-<p>Total time to run all queries (lower is better).</p>
-<p><img alt="" src="../../_images/tpcds_allqueries.png" /></p>
-<p>Per-query breakdown showing the relative performance of Spark and Comet.</p>
-<p><img alt="" src="../../_images/tpcds_queries_compare.png" /></p>
-<p>How much Comet accelerates each query in relative terms.</p>
-<p><img alt="" src="../../_images/tpcds_queries_speedup_rel.png" /></p>
-<p>How much Comet accelerates each query in absolute terms.</p>
-<p><img alt="" src="../../_images/tpcds_queries_speedup_abs.png" /></p>
-</section>
-<section id="configuration">
-<h2>Configuration<a class="headerlink" href="#configuration" title="Link to this heading">#</a></h2>
-<p>Common:</p>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.executor.instances</span><span class="o">=</span><span class="s">32</span>
-<span class="na">spark.executor.cores</span><span class="o">=</span><span class="s">16</span>
-<span class="na">spark.memory.fraction</span><span class="o">=</span><span class="s">0.6</span>
-<span class="na">spark.memory.storageFraction</span><span class="o">=</span><span class="s">0.2</span>
-<span class="c1"># Kubernetes CPU constraints</span>
-<span class="na">spark.kubernetes.executor.request.cores</span><span class="o">=</span><span class="s">8</span>
-<span class="na">spark.kubernetes.executor.limit.cores</span><span class="o">=</span><span class="s">8</span>
-</pre></div>
-</div>
-<p>Spark:</p>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.executor.memory</span><span class="o">=</span><span class="s">64G</span>
-<span class="na">spark.executor.memoryOverhead</span><span class="o">=</span><span class="s">10G</span>
-</pre></div>
-</div>
-<p>Comet:</p>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.executor.memory</span><span class="o">=</span><span class="s">64G</span>
-<span class="na">spark.executor.memoryOverhead</span><span class="o">=</span><span class="s">10G</span>
-<span class="na">spark.memory.offHeap.enabled</span><span class="o">=</span><span class="s">true</span>
-<span class="na">spark.memory.offHeap.size</span><span class="o">=</span><span class="s">32G</span>
-<span class="na">spark.comet.memoryPool.fraction</span><span class="o">=</span><span class="s">0.8</span>
-</pre></div>
-</div>
+<section id="conversion-funcs-expression-audits">
+<h1>conversion_funcs Expression Audits<a class="headerlink" href="#conversion-funcs-expression-audits" title="Link to this heading">#</a></h1>
+<blockquote>
+<div><p>Audit notes for expressions in this category that have been audited. Absence of an entry means the expression has not been audited yet, not that it is unsupported. See the user guide <a class="reference internal" href="../../user-guide/latest/expressions.html"><span class="std std-doc">Spark Expression Support</span></a> for current support status.</p>
+</div></blockquote>
+<section id="cast">
+<h2>cast<a class="headerlink" href="#cast" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8 modulo <code class="docutils literal notranslate"><span class="pre">Cast.canUpCast</span></code> refactored to delegate to <code class="docutils literal notranslate"><span class="pre">UpCastRule.canUpCast</span></code>.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">Cast(child,</span> <span class="pre">dataType,</span> <span class="pre">timeZoneId,</span> <span class="pre">evalMode)</span></code>; eval modes are <code class="docutils literal notranslate"><span class="pre">LEGACY</span></code>, <code class="docutils literal notranslate"><span class="pre">ANSI</span></code>, <code class="docutils literal notranslate"><span class="pre">TRY</s [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">VariantType</span></code> added; <code class="docutils literal notranslate"><span class="pre">StringType</span></code> literals replaced with <code class="docutils literal notranslate"><span class="pre">_:</span> <span class="pre">StringType</span></code> to accommodate collated strings. <code class="docutils literal notranslate"><span class="pre">(TimestampType,</span> <span class="pre" [...]
+<li><p>Spark 4.1.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">TimeType</span></code> added; many <code class="docutils literal notranslate"><span class="pre">TimeType</span></code> arms in <code class="docutils literal notranslate"><span class="pre">canCast</span></code>/<code class="docutils literal notranslate"><span class="pre">canAnsiCast</span></code>. Geospatial <code class="docutils literal notranslate"><span class="pre">GeographyType</span> [...]
+<li><p>Known divergences and gaps:</p>
+<ul>
+<li><p><code class="docutils literal notranslate"><span class="pre">CAST(<binary></span> <span class="pre">AS</span> <span class="pre">STRING)</span></code> uses <code class="docutils literal notranslate"><span class="pre">unsafe</span> <span class="pre">{</span> <span class="pre">String::from_utf8_unchecked</span> <span class="pre">}</span></code> in native code, which is undefined behaviour for non-UTF8 inputs (https://github.com/apache/datafusion-comet/issues/4488).</p></li>
+<li><p>Spark 4.0 collated <code class="docutils literal notranslate"><span class="pre">StringType</span></code> is not explicitly guarded; pattern equality is expected to keep collated-string casts falling back, but there is no test (https://github.com/apache/datafusion-comet/issues/4489; umbrella #2190).</p></li>
+<li><p>Spark 4.1 <code class="docutils literal notranslate"><span class="pre">TimeType</span></code> casts have no explicit <code class="docutils literal notranslate"><span class="pre">Unsupported</span></code> arm; they fall back implicitly but do not appear in the auto-generated compatibility doc (https://github.com/apache/datafusion-comet/issues/4490).</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">CAST(<map></span> <span class="pre">AS</span> <span class="pre"><map>)</span></code> falls back to Spark even though native <code class="docutils literal notranslate"><span class="pre">cast_map_to_map</span></code> exists (https://github.com/apache/datafusion-comet/issues/4491).</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">spark.sql.legacy.castComplexTypesToString.enabled=true</span></code> is not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4492).</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">CAST(<float|double></span> <span class="pre">AS</span> <span class="pre">DECIMAL)</span></code> rounding may differ from Spark (<code class="docutils literal notranslate"><span class="pre">Incompatible</span></code>, gated by <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.Cast.allowIncompatible</span></code>, tracked at https://github.com/apache/datafusion-comet/issues/1371).</p></li>
+</ul>
+</li>
+<li><p>Spark registers the type-name conversion functions (<code class="docutils literal notranslate"><span class="pre">bigint</span></code>, <code class="docutils literal notranslate"><span class="pre">binary</span></code>, <code class="docutils literal notranslate"><span class="pre">boolean</span></code>, <code class="docutils literal notranslate"><span class="pre">date</span></code>, <code class="docutils literal notranslate"><span class="pre">decimal</span></code>, <code class="docut [...]
+</ul>
</section>
</section>
@@ -578,20 +582,20 @@ under the License.
<div class="prev-next-area">
<a class="left-prev"
- href="tpc-h.html"
+ href="conditional_funcs.html"
title="previous page">
<i class="fa-solid fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">Apache DataFusion Comet: Benchmarks Derived From TPC-H</p>
+ <p class="prev-next-title">conditional_funcs Expression Audits</p>
</div>
</a>
<a class="right-next"
- href="../benchmarking_macos.html"
+ href="datetime_funcs.html"
title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
- <p class="prev-next-title">Comet Benchmarking on macOS</p>
+ <p class="prev-next-title">datetime_funcs Expression Audits</p>
</div>
<i class="fa-solid fa-angle-right"></i>
</a>
diff --git a/contributor-guide/benchmark-results/tpc-h.html b/contributor-guide/expression-audits/datetime_funcs.html
similarity index 60%
copy from contributor-guide/benchmark-results/tpc-h.html
copy to contributor-guide/expression-audits/datetime_funcs.html
index b75aa4ff2a..ef13c46209 100644
--- a/contributor-guide/benchmark-results/tpc-h.html
+++ b/contributor-guide/expression-audits/datetime_funcs.html
@@ -27,7 +27,7 @@ under the License.
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
- <title>Apache DataFusion Comet: Benchmarks Derived From TPC-H — Apache DataFusion Comet documentation</title>
+ <title>datetime_funcs Expression Audits — Apache DataFusion Comet documentation</title>
@@ -61,12 +61,12 @@ under the License.
<script src="../../_static/documentation_options.js?v=5929fcd5"></script>
<script src="../../_static/doctools.js?v=9a2dae69"></script>
<script src="../../_static/sphinx_highlight.js?v=dc90522c"></script>
- <script>DOCUMENTATION_OPTIONS.pagename = 'contributor-guide/benchmark-results/tpc-h';</script>
+ <script>DOCUMENTATION_OPTIONS.pagename = 'contributor-guide/expression-audits/datetime_funcs';</script>
<script async="true" defer="true" src="https://buttons.github.io/buttons.js"></script>
<link rel="index" title="Index" href="../../genindex.html" />
<link rel="search" title="Search" href="../../search.html" />
- <link rel="next" title="Apache DataFusion Comet: Benchmarks Derived From TPC-DS" href="tpc-ds.html" />
- <link rel="prev" title="Comet Benchmarking Guide" href="../benchmarking.html" />
+ <link rel="next" title="generator_funcs Expression Audits" href="generator_funcs.html" />
+ <link rel="prev" title="conversion_funcs Expression Audits" href="conversion_funcs.html" />
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<meta name="docsearch:language" content="en"/>
<meta name="docsearch:version" content="" />
@@ -418,11 +418,11 @@ under the License.
<li class="toctree-l1"><a class="reference internal" href="../iceberg-spark-tests.html">Iceberg Spark Tests</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Debugging and Performance</span></p>
-<ul class="current nav bd-sidenav">
+<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../debugging.html">Debugging Guide</a></li>
-<li class="toctree-l1 current active has-children"><a class="reference internal" href="../benchmarking.html">Benchmarking Guide</a><details open="open"><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul class="current">
-<li class="toctree-l2 current active"><a class="current reference internal" href="#">Apache DataFusion Comet: Benchmarks Derived From TPC-H</a></li>
-<li class="toctree-l2"><a class="reference internal" href="tpc-ds.html">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</a></li>
+<li class="toctree-l1 has-children"><a class="reference internal" href="../benchmarking.html">Benchmarking Guide</a><details><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul>
+<li class="toctree-l2"><a class="reference internal" href="../benchmark-results/tpc-h.html">Apache DataFusion Comet: Benchmarks Derived From TPC-H</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../benchmark-results/tpc-ds.html">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_macos.html">Comet Benchmarking on macOS</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_aws_ec2.html">Comet Benchmarking in EC2</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_spark_sql_perf.html">TPC-DS Benchmarking with spark-sql-perf</a></li>
@@ -432,8 +432,27 @@ under the License.
<li class="toctree-l1"><a class="reference internal" href="../tracing.html">Tracing</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Reference</span></p>
-<ul class="nav bd-sidenav">
-<li class="toctree-l1"><a class="reference internal" href="../spark_expressions_support.html">Supported Spark Expressions</a></li>
+<ul class="current nav bd-sidenav">
+<li class="toctree-l1 current active has-children"><a class="reference internal" href="index.html">Expression Audits</a><details open="open"><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul class="current">
+<li class="toctree-l2"><a class="reference internal" href="agg_funcs.html">agg_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="array_funcs.html">array_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="bitwise_funcs.html">bitwise_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="collection_funcs.html">collection_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="conditional_funcs.html">conditional_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="conversion_funcs.html">conversion_funcs</a></li>
+<li class="toctree-l2 current active"><a class="current reference internal" href="#">datetime_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="generator_funcs.html">generator_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="hash_funcs.html">hash_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="json_funcs.html">json_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="map_funcs.html">map_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="math_funcs.html">math_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="misc_funcs.html">misc_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="predicate_funcs.html">predicate_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="struct_funcs.html">struct_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="url_funcs.html">url_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="window_funcs.html">window_funcs</a></li>
+</ul>
+</details></li>
<li class="toctree-l1"><a class="reference internal" href="../spark_configs_support.html">Supported Spark Configurations</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Project Mechanics</span></p>
@@ -488,9 +507,9 @@ under the License.
<li class="breadcrumb-item"><a href="../index.html" class="nav-link">Comet Contributor Guide</a></li>
- <li class="breadcrumb-item"><a href="../benchmarking.html" class="nav-link">Comet Benchmarking Guide</a></li>
+ <li class="breadcrumb-item"><a href="index.html" class="nav-link">Expression Audits</a></li>
- <li class="breadcrumb-item active" aria-current="page"><span class="ellipsis">Apache DataFusion Comet: Benchmarks Derived From TPC-H</span></li>
+ <li class="breadcrumb-item active" aria-current="page"><span class="ellipsis">datetime_funcs Expression Audits</span></li>
</ul>
</nav>
</div>
@@ -507,70 +526,124 @@ under the License.
<div id="searchbox"></div>
<article class="bd-article">
- <!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
+ <!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
-->
-<section id="apache-datafusion-comet-benchmarks-derived-from-tpc-h">
-<h1>Apache DataFusion Comet: Benchmarks Derived From TPC-H<a class="headerlink" href="#apache-datafusion-comet-benchmarks-derived-from-tpc-h" title="Link to this heading">#</a></h1>
-<p>The following benchmarks were performed on an EKS cluster (<code class="docutils literal notranslate"><span class="pre">r6i.24xlarge</span></code> instances with EBS storage) with data stored in S3.</p>
-<section id="benchmark-results">
-<h2>Benchmark Results<a class="headerlink" href="#benchmark-results" title="Link to this heading">#</a></h2>
-<p>Total time to run all queries (lower is better).</p>
-<p><img alt="" src="../../_images/tpch_allqueries_with_tuned.png" /></p>
-<p>The following charts are based on the tuned run using hash join.</p>
-<p>Per-query breakdown showing the relative performance of Spark and Comet.</p>
-<p><img alt="" src="../../_images/tpch_queries_compare.png" /></p>
-<p>How much Comet accelerates each query in relative terms.</p>
-<p><img alt="" src="../../_images/tpch_queries_speedup_rel.png" /></p>
-<p>How much Comet accelerates each query in absolute terms.</p>
-<p><img alt="" src="../../_images/tpch_queries_speedup_abs.png" /></p>
+<section id="datetime-funcs-expression-audits">
+<h1>datetime_funcs Expression Audits<a class="headerlink" href="#datetime-funcs-expression-audits" title="Link to this heading">#</a></h1>
+<blockquote>
+<div><p>Audit notes for expressions in this category that have been audited. Absence of an entry means the expression has not been audited yet, not that it is unsupported. See the user guide <a class="reference internal" href="../../user-guide/latest/expressions.html"><span class="std std-doc">Spark Expression Support</span></a> for current support status.</p>
+</div></blockquote>
+<section id="curdate">
+<h2>curdate<a class="headerlink" href="#curdate" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Alias of <code class="docutils literal notranslate"><span class="pre">current_date</span></code>; constant-folded to a literal by Spark’s <code class="docutils literal notranslate"><span class="pre">ComputeCurrentTime</span></code> rule before Comet sees the plan.</p></li>
+</ul>
</section>
-<section id="configuration">
-<h2>Configuration<a class="headerlink" href="#configuration" title="Link to this heading">#</a></h2>
-<p>Common:</p>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.executor.instances</span><span class="o">=</span><span class="s">32</span>
-<span class="na">spark.executor.cores</span><span class="o">=</span><span class="s">16</span>
-<span class="na">spark.memory.fraction</span><span class="o">=</span><span class="s">0.6</span>
-<span class="na">spark.memory.storageFraction</span><span class="o">=</span><span class="s">0.2</span>
-<span class="c1"># Kubernetes CPU constraints</span>
-<span class="na">spark.kubernetes.executor.request.cores</span><span class="o">=</span><span class="s">8</span>
-<span class="na">spark.kubernetes.executor.limit.cores</span><span class="o">=</span><span class="s">8</span>
-</pre></div>
-</div>
-<p>Spark:</p>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.executor.memory</span><span class="o">=</span><span class="s">64G</span>
-<span class="na">spark.executor.memoryOverhead</span><span class="o">=</span><span class="s">10G</span>
-</pre></div>
-</div>
-<p>Comet:</p>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.executor.memory</span><span class="o">=</span><span class="s">32G</span>
-<span class="na">spark.executor.memoryOverhead</span><span class="o">=</span><span class="s">10G</span>
-<span class="na">spark.memory.offHeap.enabled</span><span class="o">=</span><span class="s">true</span>
-<span class="na">spark.memory.offHeap.size</span><span class="o">=</span><span class="s">32G</span>
-</pre></div>
-</div>
-<section id="comet-tuned">
-<h3>Comet (Tuned)<a class="headerlink" href="#comet-tuned" title="Link to this heading">#</a></h3>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.comet.exec.replaceSortMergeJoin</span><span class="o">=</span><span class="s">true</span>
-<span class="na">spark.comet.memoryPool.fraction</span><span class="o">=</span><span class="s">0.8</span>
-</pre></div>
-</div>
+<section id="current-date">
+<h2>current_date<a class="headerlink" href="#current-date" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Constant-folded to a literal by Spark’s <code class="docutils literal notranslate"><span class="pre">ComputeCurrentTime</span></code> rule before Comet sees the plan.</p></li>
+</ul>
+</section>
+<section id="current-timestamp">
+<h2>current_timestamp<a class="headerlink" href="#current-timestamp" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Constant-folded to a literal by Spark’s <code class="docutils literal notranslate"><span class="pre">ComputeCurrentTime</span></code> rule before Comet sees the plan.</p></li>
+</ul>
+</section>
+<section id="dayname">
+<h2>dayname<a class="headerlink" href="#dayname" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 4.0+. Implemented natively: maps a <code class="docutils literal notranslate"><span class="pre">DateType</span></code> value to a fixed US-English abbreviated day name (<code class="docutils literal notranslate"><span class="pre">DayOfWeek.getDisplayName(TextStyle.SHORT,</span> <span class="pre">Locale.US)</span></code>), with no session-locale or timezone dependence.</p></li>
+</ul>
+</section>
+<section id="from-utc-timestamp">
+<h2>from_utc_timestamp<a class="headerlink" href="#from-utc-timestamp" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-12): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-12): baseline.</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-12): <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code> widened to <code class="docutils literal notranslate"><span class="pre">StringTypeWithCollation</span></code>; behaviour unchanged for ASCII timezone strings.</p></li>
+<li><p>Known divergence: Comet’s native timezone parser does not accept Spark’s legacy zone forms (<code class="docutils literal notranslate"><span class="pre">GMT+1</span></code>, <code class="docutils literal notranslate"><span class="pre">UTC+1</span></code>, three-letter abbreviations like <code class="docutils literal notranslate"><span class="pre">PST</span></code>). Such timezones throw a native parse error at execution.</p></li>
+</ul>
+</section>
+<section id="make-timestamp-ltz">
+<h2>make_timestamp_ltz<a class="headerlink" href="#make-timestamp-ltz" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>The 6-argument form rewrites to <code class="docutils literal notranslate"><span class="pre">MakeTimestamp</span></code> and runs via the codegen dispatcher. The 2-argument <code class="docutils literal notranslate"><span class="pre">(date,</span> <span class="pre">time)</span></code> form requires the Spark 4.1 TIME type and falls back.</p></li>
+</ul>
+</section>
+<section id="make-timestamp-ntz">
+<h2>make_timestamp_ntz<a class="headerlink" href="#make-timestamp-ntz" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>The 6-argument form rewrites to <code class="docutils literal notranslate"><span class="pre">MakeTimestamp</span></code> and runs via the codegen dispatcher. The 2-argument <code class="docutils literal notranslate"><span class="pre">(date,</span> <span class="pre">time)</span></code> form requires the Spark 4.1 TIME type and falls back.</p></li>
+</ul>
+</section>
+<section id="monthname">
+<h2>monthname<a class="headerlink" href="#monthname" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 4.0+. Implemented natively: maps a <code class="docutils literal notranslate"><span class="pre">DateType</span></code> value to a fixed US-English abbreviated month name (<code class="docutils literal notranslate"><span class="pre">Month.getDisplayName(TextStyle.SHORT,</span> <span class="pre">Locale.US)</span></code>), with no session-locale or timezone dependence.</p></li>
+</ul>
</section>
+<section id="now">
+<h2>now<a class="headerlink" href="#now" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Alias of <code class="docutils literal notranslate"><span class="pre">current_timestamp</span></code>; constant-folded to a literal by Spark’s <code class="docutils literal notranslate"><span class="pre">ComputeCurrentTime</span></code> rule before Comet sees the plan.</p></li>
+</ul>
+</section>
+<section id="to-date">
+<h2>to_date<a class="headerlink" href="#to-date" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Rewrites to <code class="docutils literal notranslate"><span class="pre">Cast</span></code> (no format, native) or <code class="docutils literal notranslate"><span class="pre">Cast(GetTimestamp(...))</span></code> (with format, via the codegen dispatcher) before Comet sees the plan.</p></li>
+</ul>
+</section>
+<section id="to-timestamp">
+<h2>to_timestamp<a class="headerlink" href="#to-timestamp" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Rewrites to <code class="docutils literal notranslate"><span class="pre">Cast</span></code> (no format, native) or <code class="docutils literal notranslate"><span class="pre">GetTimestamp</span></code> (with format, via the codegen dispatcher) before Comet sees the plan.</p></li>
+</ul>
+</section>
+<section id="to-timestamp-ltz">
+<h2>to_timestamp_ltz<a class="headerlink" href="#to-timestamp-ltz" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Rewrites to <code class="docutils literal notranslate"><span class="pre">to_timestamp</span></code> with <code class="docutils literal notranslate"><span class="pre">TimestampType</span></code>; same support as <code class="docutils literal notranslate"><span class="pre">to_timestamp</span></code>.</p></li>
+</ul>
+</section>
+<section id="to-timestamp-ntz">
+<h2>to_timestamp_ntz<a class="headerlink" href="#to-timestamp-ntz" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Rewrites to <code class="docutils literal notranslate"><span class="pre">to_timestamp</span></code> with <code class="docutils literal notranslate"><span class="pre">TimestampNTZType</span></code>; same support as <code class="docutils literal notranslate"><span class="pre">to_timestamp</span></code>.</p></li>
+</ul>
+</section>
+<section id="to-utc-timestamp">
+<h2>to_utc_timestamp<a class="headerlink" href="#to-utc-timestamp" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-12): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-12): baseline.</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-12): <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code> widened to <code class="docutils literal notranslate"><span class="pre">StringTypeWithCollation</span></code>; behaviour unchanged for ASCII timezone strings.</p></li>
+<li><p>Known divergence: Comet’s native timezone parser does not accept Spark’s legacy zone forms (<code class="docutils literal notranslate"><span class="pre">GMT+1</span></code>, <code class="docutils literal notranslate"><span class="pre">UTC+1</span></code>, three-letter abbreviations like <code class="docutils literal notranslate"><span class="pre">PST</span></code>). Such timezones throw a native parse error at execution.</p></li>
+</ul>
+</section>
+<section id="try-make-timestamp">
+<h2>try_make_timestamp<a class="headerlink" href="#try-make-timestamp" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Native for valid inputs; returns wrong values for invalid inputs instead of NULL (https://github.com/apache/datafusion-comet/issues/4554).</p></li>
+</ul>
</section>
</section>
@@ -585,20 +658,20 @@ under the License.
<div class="prev-next-area">
<a class="left-prev"
- href="../benchmarking.html"
+ href="conversion_funcs.html"
title="previous page">
<i class="fa-solid fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">Comet Benchmarking Guide</p>
+ <p class="prev-next-title">conversion_funcs Expression Audits</p>
</div>
</a>
<a class="right-next"
- href="tpc-ds.html"
+ href="generator_funcs.html"
title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
- <p class="prev-next-title">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</p>
+ <p class="prev-next-title">generator_funcs Expression Audits</p>
</div>
<i class="fa-solid fa-angle-right"></i>
</a>
diff --git a/contributor-guide/benchmark-results/tpc-h.html b/contributor-guide/expression-audits/generator_funcs.html
similarity index 72%
copy from contributor-guide/benchmark-results/tpc-h.html
copy to contributor-guide/expression-audits/generator_funcs.html
index b75aa4ff2a..137e87958f 100644
--- a/contributor-guide/benchmark-results/tpc-h.html
+++ b/contributor-guide/expression-audits/generator_funcs.html
@@ -27,7 +27,7 @@ under the License.
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
- <title>Apache DataFusion Comet: Benchmarks Derived From TPC-H — Apache DataFusion Comet documentation</title>
+ <title>generator_funcs Expression Audits — Apache DataFusion Comet documentation</title>
@@ -61,12 +61,12 @@ under the License.
<script src="../../_static/documentation_options.js?v=5929fcd5"></script>
<script src="../../_static/doctools.js?v=9a2dae69"></script>
<script src="../../_static/sphinx_highlight.js?v=dc90522c"></script>
- <script>DOCUMENTATION_OPTIONS.pagename = 'contributor-guide/benchmark-results/tpc-h';</script>
+ <script>DOCUMENTATION_OPTIONS.pagename = 'contributor-guide/expression-audits/generator_funcs';</script>
<script async="true" defer="true" src="https://buttons.github.io/buttons.js"></script>
<link rel="index" title="Index" href="../../genindex.html" />
<link rel="search" title="Search" href="../../search.html" />
- <link rel="next" title="Apache DataFusion Comet: Benchmarks Derived From TPC-DS" href="tpc-ds.html" />
- <link rel="prev" title="Comet Benchmarking Guide" href="../benchmarking.html" />
+ <link rel="next" title="hash_funcs Expression Audits" href="hash_funcs.html" />
+ <link rel="prev" title="datetime_funcs Expression Audits" href="datetime_funcs.html" />
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<meta name="docsearch:language" content="en"/>
<meta name="docsearch:version" content="" />
@@ -418,11 +418,11 @@ under the License.
<li class="toctree-l1"><a class="reference internal" href="../iceberg-spark-tests.html">Iceberg Spark Tests</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Debugging and Performance</span></p>
-<ul class="current nav bd-sidenav">
+<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../debugging.html">Debugging Guide</a></li>
-<li class="toctree-l1 current active has-children"><a class="reference internal" href="../benchmarking.html">Benchmarking Guide</a><details open="open"><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul class="current">
-<li class="toctree-l2 current active"><a class="current reference internal" href="#">Apache DataFusion Comet: Benchmarks Derived From TPC-H</a></li>
-<li class="toctree-l2"><a class="reference internal" href="tpc-ds.html">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</a></li>
+<li class="toctree-l1 has-children"><a class="reference internal" href="../benchmarking.html">Benchmarking Guide</a><details><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul>
+<li class="toctree-l2"><a class="reference internal" href="../benchmark-results/tpc-h.html">Apache DataFusion Comet: Benchmarks Derived From TPC-H</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../benchmark-results/tpc-ds.html">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_macos.html">Comet Benchmarking on macOS</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_aws_ec2.html">Comet Benchmarking in EC2</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_spark_sql_perf.html">TPC-DS Benchmarking with spark-sql-perf</a></li>
@@ -432,8 +432,27 @@ under the License.
<li class="toctree-l1"><a class="reference internal" href="../tracing.html">Tracing</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Reference</span></p>
-<ul class="nav bd-sidenav">
-<li class="toctree-l1"><a class="reference internal" href="../spark_expressions_support.html">Supported Spark Expressions</a></li>
+<ul class="current nav bd-sidenav">
+<li class="toctree-l1 current active has-children"><a class="reference internal" href="index.html">Expression Audits</a><details open="open"><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul class="current">
+<li class="toctree-l2"><a class="reference internal" href="agg_funcs.html">agg_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="array_funcs.html">array_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="bitwise_funcs.html">bitwise_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="collection_funcs.html">collection_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="conditional_funcs.html">conditional_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="conversion_funcs.html">conversion_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="datetime_funcs.html">datetime_funcs</a></li>
+<li class="toctree-l2 current active"><a class="current reference internal" href="#">generator_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="hash_funcs.html">hash_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="json_funcs.html">json_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="map_funcs.html">map_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="math_funcs.html">math_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="misc_funcs.html">misc_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="predicate_funcs.html">predicate_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="struct_funcs.html">struct_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="url_funcs.html">url_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="window_funcs.html">window_funcs</a></li>
+</ul>
+</details></li>
<li class="toctree-l1"><a class="reference internal" href="../spark_configs_support.html">Supported Spark Configurations</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Project Mechanics</span></p>
@@ -488,9 +507,9 @@ under the License.
<li class="breadcrumb-item"><a href="../index.html" class="nav-link">Comet Contributor Guide</a></li>
- <li class="breadcrumb-item"><a href="../benchmarking.html" class="nav-link">Comet Benchmarking Guide</a></li>
+ <li class="breadcrumb-item"><a href="index.html" class="nav-link">Expression Audits</a></li>
- <li class="breadcrumb-item active" aria-current="page"><span class="ellipsis">Apache DataFusion Comet: Benchmarks Derived From TPC-H</span></li>
+ <li class="breadcrumb-item active" aria-current="page"><span class="ellipsis">generator_funcs Expression Audits</span></li>
</ul>
</nav>
</div>
@@ -507,70 +526,52 @@ under the License.
<div id="searchbox"></div>
<article class="bd-article">
- <!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
+ <!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
-->
-<section id="apache-datafusion-comet-benchmarks-derived-from-tpc-h">
-<h1>Apache DataFusion Comet: Benchmarks Derived From TPC-H<a class="headerlink" href="#apache-datafusion-comet-benchmarks-derived-from-tpc-h" title="Link to this heading">#</a></h1>
-<p>The following benchmarks were performed on an EKS cluster (<code class="docutils literal notranslate"><span class="pre">r6i.24xlarge</span></code> instances with EBS storage) with data stored in S3.</p>
-<section id="benchmark-results">
-<h2>Benchmark Results<a class="headerlink" href="#benchmark-results" title="Link to this heading">#</a></h2>
-<p>Total time to run all queries (lower is better).</p>
-<p><img alt="" src="../../_images/tpch_allqueries_with_tuned.png" /></p>
-<p>The following charts are based on the tuned run using hash join.</p>
-<p>Per-query breakdown showing the relative performance of Spark and Comet.</p>
-<p><img alt="" src="../../_images/tpch_queries_compare.png" /></p>
-<p>How much Comet accelerates each query in relative terms.</p>
-<p><img alt="" src="../../_images/tpch_queries_speedup_rel.png" /></p>
-<p>How much Comet accelerates each query in absolute terms.</p>
-<p><img alt="" src="../../_images/tpch_queries_speedup_abs.png" /></p>
+<section id="generator-funcs-expression-audits">
+<h1>generator_funcs Expression Audits<a class="headerlink" href="#generator-funcs-expression-audits" title="Link to this heading">#</a></h1>
+<blockquote>
+<div><p>Audit notes for expressions in this category that have been audited. Absence of an entry means the expression has not been audited yet, not that it is unsupported. See the user guide <a class="reference internal" href="../../user-guide/latest/expressions.html"><span class="std std-doc">Spark Expression Support</span></a> for current support status.</p>
+</div></blockquote>
+<section id="explode">
+<h2>explode<a class="headerlink" href="#explode" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Handled at the operator level as a <code class="docutils literal notranslate"><span class="pre">GenerateExec</span></code> (<code class="docutils literal notranslate"><span class="pre">CometExplodeExec</span></code>), not via the expression serde maps, so it is not auto-detected by the function-registry checkbox logic. Compatible for array inputs; map inputs fall back (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/2837">#2837</a>).</p></li>
+</ul>
</section>
-<section id="configuration">
-<h2>Configuration<a class="headerlink" href="#configuration" title="Link to this heading">#</a></h2>
-<p>Common:</p>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.executor.instances</span><span class="o">=</span><span class="s">32</span>
-<span class="na">spark.executor.cores</span><span class="o">=</span><span class="s">16</span>
-<span class="na">spark.memory.fraction</span><span class="o">=</span><span class="s">0.6</span>
-<span class="na">spark.memory.storageFraction</span><span class="o">=</span><span class="s">0.2</span>
-<span class="c1"># Kubernetes CPU constraints</span>
-<span class="na">spark.kubernetes.executor.request.cores</span><span class="o">=</span><span class="s">8</span>
-<span class="na">spark.kubernetes.executor.limit.cores</span><span class="o">=</span><span class="s">8</span>
-</pre></div>
-</div>
-<p>Spark:</p>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.executor.memory</span><span class="o">=</span><span class="s">64G</span>
-<span class="na">spark.executor.memoryOverhead</span><span class="o">=</span><span class="s">10G</span>
-</pre></div>
-</div>
-<p>Comet:</p>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.executor.memory</span><span class="o">=</span><span class="s">32G</span>
-<span class="na">spark.executor.memoryOverhead</span><span class="o">=</span><span class="s">10G</span>
-<span class="na">spark.memory.offHeap.enabled</span><span class="o">=</span><span class="s">true</span>
-<span class="na">spark.memory.offHeap.size</span><span class="o">=</span><span class="s">32G</span>
-</pre></div>
-</div>
-<section id="comet-tuned">
-<h3>Comet (Tuned)<a class="headerlink" href="#comet-tuned" title="Link to this heading">#</a></h3>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.comet.exec.replaceSortMergeJoin</span><span class="o">=</span><span class="s">true</span>
-<span class="na">spark.comet.memoryPool.fraction</span><span class="o">=</span><span class="s">0.8</span>
-</pre></div>
-</div>
+<section id="explode-outer">
+<h2>explode_outer<a class="headerlink" href="#explode-outer" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Same <code class="docutils literal notranslate"><span class="pre">CometExplodeExec</span></code> path as <code class="docutils literal notranslate"><span class="pre">explode</span></code>, but the <code class="docutils literal notranslate"><span class="pre">outer=true</span></code> case is <code class="docutils literal notranslate"><span class="pre">Incompatible</span></code> (empty arrays are not preserved as null outputs) and falls back unless <code class="docutils literal notra [...]
+</ul>
+</section>
+<section id="posexplode">
+<h2>posexplode<a class="headerlink" href="#posexplode" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Handled at the operator level as a <code class="docutils literal notranslate"><span class="pre">GenerateExec</span></code> (<code class="docutils literal notranslate"><span class="pre">CometExplodeExec</span></code>), like <code class="docutils literal notranslate"><span class="pre">explode</span></code>. Compatible for array inputs; map inputs fall back (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/2837">#2837</a>).</p></li>
+</ul>
</section>
+<section id="posexplode-outer">
+<h2>posexplode_outer<a class="headerlink" href="#posexplode-outer" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Same <code class="docutils literal notranslate"><span class="pre">CometExplodeExec</span></code> path as <code class="docutils literal notranslate"><span class="pre">posexplode</span></code>, but the <code class="docutils literal notranslate"><span class="pre">outer=true</span></code> case is <code class="docutils literal notranslate"><span class="pre">Incompatible</span></code> and falls back unless <code class="docutils literal notranslate"><span class="pre">spark.comet.expr.all [...]
+</ul>
</section>
</section>
@@ -585,20 +586,20 @@ under the License.
<div class="prev-next-area">
<a class="left-prev"
- href="../benchmarking.html"
+ href="datetime_funcs.html"
title="previous page">
<i class="fa-solid fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">Comet Benchmarking Guide</p>
+ <p class="prev-next-title">datetime_funcs Expression Audits</p>
</div>
</a>
<a class="right-next"
- href="tpc-ds.html"
+ href="hash_funcs.html"
title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
- <p class="prev-next-title">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</p>
+ <p class="prev-next-title">hash_funcs Expression Audits</p>
</div>
<i class="fa-solid fa-angle-right"></i>
</a>
diff --git a/contributor-guide/benchmark-results/tpc-ds.html b/contributor-guide/expression-audits/hash_funcs.html
similarity index 59%
copy from contributor-guide/benchmark-results/tpc-ds.html
copy to contributor-guide/expression-audits/hash_funcs.html
index 4d5bf9b1c4..3af21d8fa9 100644
--- a/contributor-guide/benchmark-results/tpc-ds.html
+++ b/contributor-guide/expression-audits/hash_funcs.html
@@ -27,7 +27,7 @@ under the License.
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
- <title>Apache DataFusion Comet: Benchmarks Derived From TPC-DS — Apache DataFusion Comet documentation</title>
+ <title>hash_funcs Expression Audits — Apache DataFusion Comet documentation</title>
@@ -61,12 +61,12 @@ under the License.
<script src="../../_static/documentation_options.js?v=5929fcd5"></script>
<script src="../../_static/doctools.js?v=9a2dae69"></script>
<script src="../../_static/sphinx_highlight.js?v=dc90522c"></script>
- <script>DOCUMENTATION_OPTIONS.pagename = 'contributor-guide/benchmark-results/tpc-ds';</script>
+ <script>DOCUMENTATION_OPTIONS.pagename = 'contributor-guide/expression-audits/hash_funcs';</script>
<script async="true" defer="true" src="https://buttons.github.io/buttons.js"></script>
<link rel="index" title="Index" href="../../genindex.html" />
<link rel="search" title="Search" href="../../search.html" />
- <link rel="next" title="Comet Benchmarking on macOS" href="../benchmarking_macos.html" />
- <link rel="prev" title="Apache DataFusion Comet: Benchmarks Derived From TPC-H" href="tpc-h.html" />
+ <link rel="next" title="json_funcs Expression Audits" href="json_funcs.html" />
+ <link rel="prev" title="generator_funcs Expression Audits" href="generator_funcs.html" />
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<meta name="docsearch:language" content="en"/>
<meta name="docsearch:version" content="" />
@@ -418,11 +418,11 @@ under the License.
<li class="toctree-l1"><a class="reference internal" href="../iceberg-spark-tests.html">Iceberg Spark Tests</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Debugging and Performance</span></p>
-<ul class="current nav bd-sidenav">
+<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../debugging.html">Debugging Guide</a></li>
-<li class="toctree-l1 current active has-children"><a class="reference internal" href="../benchmarking.html">Benchmarking Guide</a><details open="open"><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul class="current">
-<li class="toctree-l2"><a class="reference internal" href="tpc-h.html">Apache DataFusion Comet: Benchmarks Derived From TPC-H</a></li>
-<li class="toctree-l2 current active"><a class="current reference internal" href="#">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</a></li>
+<li class="toctree-l1 has-children"><a class="reference internal" href="../benchmarking.html">Benchmarking Guide</a><details><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul>
+<li class="toctree-l2"><a class="reference internal" href="../benchmark-results/tpc-h.html">Apache DataFusion Comet: Benchmarks Derived From TPC-H</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../benchmark-results/tpc-ds.html">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_macos.html">Comet Benchmarking on macOS</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_aws_ec2.html">Comet Benchmarking in EC2</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_spark_sql_perf.html">TPC-DS Benchmarking with spark-sql-perf</a></li>
@@ -432,8 +432,27 @@ under the License.
<li class="toctree-l1"><a class="reference internal" href="../tracing.html">Tracing</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Reference</span></p>
-<ul class="nav bd-sidenav">
-<li class="toctree-l1"><a class="reference internal" href="../spark_expressions_support.html">Supported Spark Expressions</a></li>
+<ul class="current nav bd-sidenav">
+<li class="toctree-l1 current active has-children"><a class="reference internal" href="index.html">Expression Audits</a><details open="open"><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul class="current">
+<li class="toctree-l2"><a class="reference internal" href="agg_funcs.html">agg_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="array_funcs.html">array_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="bitwise_funcs.html">bitwise_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="collection_funcs.html">collection_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="conditional_funcs.html">conditional_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="conversion_funcs.html">conversion_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="datetime_funcs.html">datetime_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="generator_funcs.html">generator_funcs</a></li>
+<li class="toctree-l2 current active"><a class="current reference internal" href="#">hash_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="json_funcs.html">json_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="map_funcs.html">map_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="math_funcs.html">math_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="misc_funcs.html">misc_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="predicate_funcs.html">predicate_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="struct_funcs.html">struct_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="url_funcs.html">url_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="window_funcs.html">window_funcs</a></li>
+</ul>
+</details></li>
<li class="toctree-l1"><a class="reference internal" href="../spark_configs_support.html">Supported Spark Configurations</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Project Mechanics</span></p>
@@ -488,9 +507,9 @@ under the License.
<li class="breadcrumb-item"><a href="../index.html" class="nav-link">Comet Contributor Guide</a></li>
- <li class="breadcrumb-item"><a href="../benchmarking.html" class="nav-link">Comet Benchmarking Guide</a></li>
+ <li class="breadcrumb-item"><a href="index.html" class="nav-link">Expression Audits</a></li>
- <li class="breadcrumb-item active" aria-current="page"><span class="ellipsis">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</span></li>
+ <li class="breadcrumb-item active" aria-current="page"><span class="ellipsis">hash_funcs Expression Audits</span></li>
</ul>
</nav>
</div>
@@ -507,63 +526,92 @@ under the License.
<div id="searchbox"></div>
<article class="bd-article">
- <!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
+ <!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
-->
-<section id="apache-datafusion-comet-benchmarks-derived-from-tpc-ds">
-<h1>Apache DataFusion Comet: Benchmarks Derived From TPC-DS<a class="headerlink" href="#apache-datafusion-comet-benchmarks-derived-from-tpc-ds" title="Link to this heading">#</a></h1>
-<p>The following benchmarks were performed on an EKS cluster (<code class="docutils literal notranslate"><span class="pre">r6i.24xlarge</span></code> instances with EBS storage) with data stored in S3.</p>
-<section id="benchmark-results">
-<h2>Benchmark Results<a class="headerlink" href="#benchmark-results" title="Link to this heading">#</a></h2>
-<p>Total time to run all queries (lower is better).</p>
-<p><img alt="" src="../../_images/tpcds_allqueries.png" /></p>
-<p>Per-query breakdown showing the relative performance of Spark and Comet.</p>
-<p><img alt="" src="../../_images/tpcds_queries_compare.png" /></p>
-<p>How much Comet accelerates each query in relative terms.</p>
-<p><img alt="" src="../../_images/tpcds_queries_speedup_rel.png" /></p>
-<p>How much Comet accelerates each query in absolute terms.</p>
-<p><img alt="" src="../../_images/tpcds_queries_speedup_abs.png" /></p>
+<section id="hash-funcs-expression-audits">
+<h1>hash_funcs Expression Audits<a class="headerlink" href="#hash-funcs-expression-audits" title="Link to this heading">#</a></h1>
+<blockquote>
+<div><p>Audit notes for expressions in this category that have been audited. Absence of an entry means the expression has not been audited yet, not that it is unsupported. See the user guide <a class="reference internal" href="../../user-guide/latest/expressions.html"><span class="std std-doc">Spark Expression Support</span></a> for current support status.</p>
+</div></blockquote>
+<section id="crc32">
+<h2>crc32<a class="headerlink" href="#crc32" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">Crc32(child)</span> <span class="pre">extends</span> <span class="pre">UnaryExpression</span></code>; <code class="docutils literal notranslate"><span class="pre">inputTypes</span> <span class="pre">=</span> <span class="pre">Seq(BinaryType)</span> <span class="pre">-></span> <span class="pre">LongType</span></code>. Wired as <code class="docutils literal notranslate"><span [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): semantics unchanged; <code class="docutils literal notranslate"><span class="pre">NullIntolerant</span></code> trait replaced by <code class="docutils literal notranslate"><span class="pre">nullIntolerant:</span> <span class="pre">Boolean</span></code> override.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
</section>
-<section id="configuration">
-<h2>Configuration<a class="headerlink" href="#configuration" title="Link to this heading">#</a></h2>
-<p>Common:</p>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.executor.instances</span><span class="o">=</span><span class="s">32</span>
-<span class="na">spark.executor.cores</span><span class="o">=</span><span class="s">16</span>
-<span class="na">spark.memory.fraction</span><span class="o">=</span><span class="s">0.6</span>
-<span class="na">spark.memory.storageFraction</span><span class="o">=</span><span class="s">0.2</span>
-<span class="c1"># Kubernetes CPU constraints</span>
-<span class="na">spark.kubernetes.executor.request.cores</span><span class="o">=</span><span class="s">8</span>
-<span class="na">spark.kubernetes.executor.limit.cores</span><span class="o">=</span><span class="s">8</span>
-</pre></div>
-</div>
-<p>Spark:</p>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.executor.memory</span><span class="o">=</span><span class="s">64G</span>
-<span class="na">spark.executor.memoryOverhead</span><span class="o">=</span><span class="s">10G</span>
-</pre></div>
-</div>
-<p>Comet:</p>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.executor.memory</span><span class="o">=</span><span class="s">64G</span>
-<span class="na">spark.executor.memoryOverhead</span><span class="o">=</span><span class="s">10G</span>
-<span class="na">spark.memory.offHeap.enabled</span><span class="o">=</span><span class="s">true</span>
-<span class="na">spark.memory.offHeap.size</span><span class="o">=</span><span class="s">32G</span>
-<span class="na">spark.comet.memoryPool.fraction</span><span class="o">=</span><span class="s">0.8</span>
-</pre></div>
-</div>
+<section id="hash">
+<h2>hash<a class="headerlink" href="#hash" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">Murmur3Hash(children,</span> <span class="pre">seed)</span> <span class="pre">extends</span> <span class="pre">HashExpression[Int]</span></code>; produces a Murmur3 hash with a configurable Int seed and <code class="docutils literal notranslate"><span class="pre">IntegerType</span></code> result. Comet routes via <code class="docutils literal notranslate"><span class="pre">Come [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): semantics unchanged; some inner helper refactors only.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+<li><p>Known limitation: <code class="docutils literal notranslate"><span class="pre">DecimalType</span></code> children with precision > 18 fall back because Spark hashes them through Java <code class="docutils literal notranslate"><span class="pre">BigDecimal</span></code>; <code class="docutils literal notranslate"><span class="pre">TimeType</span></code> (Spark 4.0+) is also unsupported. The same limitations apply to <code class="docutils literal notranslate"><span class="pre">xxh [...]
+</ul>
+</section>
+<section id="md5">
+<h2>md5<a class="headerlink" href="#md5" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">Md5(child)</span> <span class="pre">extends</span> <span class="pre">UnaryExpression</span></code>; <code class="docutils literal notranslate"><span class="pre">inputTypes</span> <span class="pre">=</span> <span class="pre">Seq(BinaryType)</span> <span class="pre">-></span> <span class="pre">StringType</span></code>. Wired as <code class="docutils literal notranslate"><span [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): semantics unchanged; trait set gains <code class="docutils literal notranslate"><span class="pre">DefaultStringProducingExpression</span></code> and the <code class="docutils literal notranslate"><span class="pre">nullIntolerant:</span> <span class="pre">Boolean</span></code> refactor.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</section>
+<section id="sha">
+<h2>sha<a class="headerlink" href="#sha" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): registry alias of <code class="docutils literal notranslate"><span class="pre">Sha1</span></code>. Same support as <code class="docutils literal notranslate"><span class="pre">sha1</span></code>.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): identical to 3.4.3.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 3.4.3.</p></li>
+</ul>
+</section>
+<section id="sha1">
+<h2>sha1<a class="headerlink" href="#sha1" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">Sha1(child)</span> <span class="pre">extends</span> <span class="pre">UnaryExpression</span> <span class="pre">with</span> <span class="pre">NullIntolerant</span></code>; <code class="docutils literal notranslate"><span class="pre">inputTypes</span> <span class="pre">=</span> <span class="pre">Seq(BinaryType)</span> <span class="pre">-></span> <span class="pre">StringType</s [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): trait set gains <code class="docutils literal notranslate"><span class="pre">DefaultStringProducingExpression</span></code> and <code class="docutils literal notranslate"><span class="pre">NullIntolerant</span></code> is replaced by <code class="docutils literal notranslate"><span class="pre">nullIntolerant:</span> <span class="pre">Boolean</span></code>. Runtime unchanged.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</section>
+<section id="sha2">
+<h2>sha2<a class="headerlink" href="#sha2" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">Sha2(left,</span> <span class="pre">right)</span> <span class="pre">extends</span> <span class="pre">BinaryExpression</span></code>; <code class="docutils literal notranslate"><span class="pre">inputTypes</span> <span class="pre">=</span> <span class="pre">Seq(BinaryType,</span> <span class="pre">IntegerType)</span> <span class="pre">-></span> <span class="pre">StringType</s [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): trait set gains <code class="docutils literal notranslate"><span class="pre">DefaultStringProducingExpression</span></code> and the <code class="docutils literal notranslate"><span class="pre">nullIntolerant:</span> <span class="pre">Boolean</span></code> refactor. Runtime unchanged.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</section>
+<section id="xxhash64">
+<h2>xxhash64<a class="headerlink" href="#xxhash64" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">XxHash64(children,</span> <span class="pre">seed)</span> <span class="pre">extends</span> <span class="pre">HashExpression[Long]</span></code>; produces an xxHash64 hash with a configurable Long seed and <code class="docutils literal notranslate"><span class="pre">LongType</span></code> result. Comet routes via <code class="docutils literal notranslate"><span class="pre">CometX [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): semantics unchanged.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
</section>
</section>
@@ -578,20 +626,20 @@ under the License.
<div class="prev-next-area">
<a class="left-prev"
- href="tpc-h.html"
+ href="generator_funcs.html"
title="previous page">
<i class="fa-solid fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">Apache DataFusion Comet: Benchmarks Derived From TPC-H</p>
+ <p class="prev-next-title">generator_funcs Expression Audits</p>
</div>
</a>
<a class="right-next"
- href="../benchmarking_macos.html"
+ href="json_funcs.html"
title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
- <p class="prev-next-title">Comet Benchmarking on macOS</p>
+ <p class="prev-next-title">json_funcs Expression Audits</p>
</div>
<i class="fa-solid fa-angle-right"></i>
</a>
diff --git a/contributor-guide/benchmark-results/tpc-h.html b/contributor-guide/expression-audits/index.html
similarity index 74%
copy from contributor-guide/benchmark-results/tpc-h.html
copy to contributor-guide/expression-audits/index.html
index b75aa4ff2a..d7ed7d84d3 100644
--- a/contributor-guide/benchmark-results/tpc-h.html
+++ b/contributor-guide/expression-audits/index.html
@@ -27,7 +27,7 @@ under the License.
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
- <title>Apache DataFusion Comet: Benchmarks Derived From TPC-H — Apache DataFusion Comet documentation</title>
+ <title>Expression Audits — Apache DataFusion Comet documentation</title>
@@ -61,12 +61,12 @@ under the License.
<script src="../../_static/documentation_options.js?v=5929fcd5"></script>
<script src="../../_static/doctools.js?v=9a2dae69"></script>
<script src="../../_static/sphinx_highlight.js?v=dc90522c"></script>
- <script>DOCUMENTATION_OPTIONS.pagename = 'contributor-guide/benchmark-results/tpc-h';</script>
+ <script>DOCUMENTATION_OPTIONS.pagename = 'contributor-guide/expression-audits/index';</script>
<script async="true" defer="true" src="https://buttons.github.io/buttons.js"></script>
<link rel="index" title="Index" href="../../genindex.html" />
<link rel="search" title="Search" href="../../search.html" />
- <link rel="next" title="Apache DataFusion Comet: Benchmarks Derived From TPC-DS" href="tpc-ds.html" />
- <link rel="prev" title="Comet Benchmarking Guide" href="../benchmarking.html" />
+ <link rel="next" title="agg_funcs Expression Audits" href="agg_funcs.html" />
+ <link rel="prev" title="Tracing" href="../tracing.html" />
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<meta name="docsearch:language" content="en"/>
<meta name="docsearch:version" content="" />
@@ -418,11 +418,11 @@ under the License.
<li class="toctree-l1"><a class="reference internal" href="../iceberg-spark-tests.html">Iceberg Spark Tests</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Debugging and Performance</span></p>
-<ul class="current nav bd-sidenav">
+<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../debugging.html">Debugging Guide</a></li>
-<li class="toctree-l1 current active has-children"><a class="reference internal" href="../benchmarking.html">Benchmarking Guide</a><details open="open"><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul class="current">
-<li class="toctree-l2 current active"><a class="current reference internal" href="#">Apache DataFusion Comet: Benchmarks Derived From TPC-H</a></li>
-<li class="toctree-l2"><a class="reference internal" href="tpc-ds.html">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</a></li>
+<li class="toctree-l1 has-children"><a class="reference internal" href="../benchmarking.html">Benchmarking Guide</a><details><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul>
+<li class="toctree-l2"><a class="reference internal" href="../benchmark-results/tpc-h.html">Apache DataFusion Comet: Benchmarks Derived From TPC-H</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../benchmark-results/tpc-ds.html">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_macos.html">Comet Benchmarking on macOS</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_aws_ec2.html">Comet Benchmarking in EC2</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_spark_sql_perf.html">TPC-DS Benchmarking with spark-sql-perf</a></li>
@@ -432,8 +432,27 @@ under the License.
<li class="toctree-l1"><a class="reference internal" href="../tracing.html">Tracing</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Reference</span></p>
-<ul class="nav bd-sidenav">
-<li class="toctree-l1"><a class="reference internal" href="../spark_expressions_support.html">Supported Spark Expressions</a></li>
+<ul class="current nav bd-sidenav">
+<li class="toctree-l1 current active has-children"><a class="current reference internal" href="#">Expression Audits</a><details open="open"><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul>
+<li class="toctree-l2"><a class="reference internal" href="agg_funcs.html">agg_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="array_funcs.html">array_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="bitwise_funcs.html">bitwise_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="collection_funcs.html">collection_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="conditional_funcs.html">conditional_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="conversion_funcs.html">conversion_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="datetime_funcs.html">datetime_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="generator_funcs.html">generator_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="hash_funcs.html">hash_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="json_funcs.html">json_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="map_funcs.html">map_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="math_funcs.html">math_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="misc_funcs.html">misc_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="predicate_funcs.html">predicate_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="struct_funcs.html">struct_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="url_funcs.html">url_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="window_funcs.html">window_funcs</a></li>
+</ul>
+</details></li>
<li class="toctree-l1"><a class="reference internal" href="../spark_configs_support.html">Supported Spark Configurations</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Project Mechanics</span></p>
@@ -487,10 +506,7 @@ under the License.
<li class="breadcrumb-item"><a href="../index.html" class="nav-link">Comet Contributor Guide</a></li>
-
- <li class="breadcrumb-item"><a href="../benchmarking.html" class="nav-link">Comet Benchmarking Guide</a></li>
-
- <li class="breadcrumb-item active" aria-current="page"><span class="ellipsis">Apache DataFusion Comet: Benchmarks Derived From TPC-H</span></li>
+ <li class="breadcrumb-item active" aria-current="page"><span class="ellipsis">Expression Audits</span></li>
</ul>
</nav>
</div>
@@ -507,71 +523,49 @@ under the License.
<div id="searchbox"></div>
<article class="bd-article">
- <!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
+ <!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
-->
-<section id="apache-datafusion-comet-benchmarks-derived-from-tpc-h">
-<h1>Apache DataFusion Comet: Benchmarks Derived From TPC-H<a class="headerlink" href="#apache-datafusion-comet-benchmarks-derived-from-tpc-h" title="Link to this heading">#</a></h1>
-<p>The following benchmarks were performed on an EKS cluster (<code class="docutils literal notranslate"><span class="pre">r6i.24xlarge</span></code> instances with EBS storage) with data stored in S3.</p>
-<section id="benchmark-results">
-<h2>Benchmark Results<a class="headerlink" href="#benchmark-results" title="Link to this heading">#</a></h2>
-<p>Total time to run all queries (lower is better).</p>
-<p><img alt="" src="../../_images/tpch_allqueries_with_tuned.png" /></p>
-<p>The following charts are based on the tuned run using hash join.</p>
-<p>Per-query breakdown showing the relative performance of Spark and Comet.</p>
-<p><img alt="" src="../../_images/tpch_queries_compare.png" /></p>
-<p>How much Comet accelerates each query in relative terms.</p>
-<p><img alt="" src="../../_images/tpch_queries_speedup_rel.png" /></p>
-<p>How much Comet accelerates each query in absolute terms.</p>
-<p><img alt="" src="../../_images/tpch_queries_speedup_abs.png" /></p>
-</section>
-<section id="configuration">
-<h2>Configuration<a class="headerlink" href="#configuration" title="Link to this heading">#</a></h2>
-<p>Common:</p>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.executor.instances</span><span class="o">=</span><span class="s">32</span>
-<span class="na">spark.executor.cores</span><span class="o">=</span><span class="s">16</span>
-<span class="na">spark.memory.fraction</span><span class="o">=</span><span class="s">0.6</span>
-<span class="na">spark.memory.storageFraction</span><span class="o">=</span><span class="s">0.2</span>
-<span class="c1"># Kubernetes CPU constraints</span>
-<span class="na">spark.kubernetes.executor.request.cores</span><span class="o">=</span><span class="s">8</span>
-<span class="na">spark.kubernetes.executor.limit.cores</span><span class="o">=</span><span class="s">8</span>
-</pre></div>
-</div>
-<p>Spark:</p>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.executor.memory</span><span class="o">=</span><span class="s">64G</span>
-<span class="na">spark.executor.memoryOverhead</span><span class="o">=</span><span class="s">10G</span>
-</pre></div>
-</div>
-<p>Comet:</p>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.executor.memory</span><span class="o">=</span><span class="s">32G</span>
-<span class="na">spark.executor.memoryOverhead</span><span class="o">=</span><span class="s">10G</span>
-<span class="na">spark.memory.offHeap.enabled</span><span class="o">=</span><span class="s">true</span>
-<span class="na">spark.memory.offHeap.size</span><span class="o">=</span><span class="s">32G</span>
-</pre></div>
-</div>
-<section id="comet-tuned">
-<h3>Comet (Tuned)<a class="headerlink" href="#comet-tuned" title="Link to this heading">#</a></h3>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.comet.exec.replaceSortMergeJoin</span><span class="o">=</span><span class="s">true</span>
-<span class="na">spark.comet.memoryPool.fraction</span><span class="o">=</span><span class="s">0.8</span>
-</pre></div>
+<section id="expression-audits">
+<h1>Expression Audits<a class="headerlink" href="#expression-audits" title="Link to this heading">#</a></h1>
+<p>This section records per-version audit notes for Spark expressions that have been audited for Comet compatibility. These are findings from auditing the Spark implementation across versions (3.4.3, 3.5.8, 4.0.1), not a statement of support status.</p>
+<p>For the authoritative list of which expressions Comet supports, see the user guide <a class="reference internal" href="../../user-guide/latest/expressions.html"><span class="std std-doc">Spark Expression Support</span></a>. To audit an expression, use the <code class="docutils literal notranslate"><span class="pre">audit-comet-expression</span></code> skill, which appends its findings to the relevant category page below.</p>
+<div class="toctree-wrapper compound">
+<ul>
+<li class="toctree-l1"><a class="reference internal" href="agg_funcs.html">agg_funcs</a></li>
+<li class="toctree-l1"><a class="reference internal" href="array_funcs.html">array_funcs</a></li>
+<li class="toctree-l1"><a class="reference internal" href="bitwise_funcs.html">bitwise_funcs</a></li>
+<li class="toctree-l1"><a class="reference internal" href="collection_funcs.html">collection_funcs</a></li>
+<li class="toctree-l1"><a class="reference internal" href="conditional_funcs.html">conditional_funcs</a></li>
+<li class="toctree-l1"><a class="reference internal" href="conversion_funcs.html">conversion_funcs</a></li>
+<li class="toctree-l1"><a class="reference internal" href="datetime_funcs.html">datetime_funcs</a></li>
+<li class="toctree-l1"><a class="reference internal" href="generator_funcs.html">generator_funcs</a></li>
+<li class="toctree-l1"><a class="reference internal" href="hash_funcs.html">hash_funcs</a></li>
+<li class="toctree-l1"><a class="reference internal" href="json_funcs.html">json_funcs</a></li>
+<li class="toctree-l1"><a class="reference internal" href="map_funcs.html">map_funcs</a></li>
+<li class="toctree-l1"><a class="reference internal" href="math_funcs.html">math_funcs</a></li>
+<li class="toctree-l1"><a class="reference internal" href="misc_funcs.html">misc_funcs</a></li>
+<li class="toctree-l1"><a class="reference internal" href="predicate_funcs.html">predicate_funcs</a></li>
+<li class="toctree-l1"><a class="reference internal" href="struct_funcs.html">struct_funcs</a></li>
+<li class="toctree-l1"><a class="reference internal" href="url_funcs.html">url_funcs</a></li>
+<li class="toctree-l1"><a class="reference internal" href="window_funcs.html">window_funcs</a></li>
+</ul>
</div>
-</section>
-</section>
</section>
@@ -585,20 +579,20 @@ under the License.
<div class="prev-next-area">
<a class="left-prev"
- href="../benchmarking.html"
+ href="../tracing.html"
title="previous page">
<i class="fa-solid fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">Comet Benchmarking Guide</p>
+ <p class="prev-next-title">Tracing</p>
</div>
</a>
<a class="right-next"
- href="tpc-ds.html"
+ href="agg_funcs.html"
title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
- <p class="prev-next-title">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</p>
+ <p class="prev-next-title">agg_funcs Expression Audits</p>
</div>
<i class="fa-solid fa-angle-right"></i>
</a>
diff --git a/contributor-guide/benchmark-results/tpc-h.html b/contributor-guide/expression-audits/json_funcs.html
similarity index 72%
copy from contributor-guide/benchmark-results/tpc-h.html
copy to contributor-guide/expression-audits/json_funcs.html
index b75aa4ff2a..a1d0cc4c19 100644
--- a/contributor-guide/benchmark-results/tpc-h.html
+++ b/contributor-guide/expression-audits/json_funcs.html
@@ -27,7 +27,7 @@ under the License.
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
- <title>Apache DataFusion Comet: Benchmarks Derived From TPC-H — Apache DataFusion Comet documentation</title>
+ <title>json_funcs Expression Audits — Apache DataFusion Comet documentation</title>
@@ -61,12 +61,12 @@ under the License.
<script src="../../_static/documentation_options.js?v=5929fcd5"></script>
<script src="../../_static/doctools.js?v=9a2dae69"></script>
<script src="../../_static/sphinx_highlight.js?v=dc90522c"></script>
- <script>DOCUMENTATION_OPTIONS.pagename = 'contributor-guide/benchmark-results/tpc-h';</script>
+ <script>DOCUMENTATION_OPTIONS.pagename = 'contributor-guide/expression-audits/json_funcs';</script>
<script async="true" defer="true" src="https://buttons.github.io/buttons.js"></script>
<link rel="index" title="Index" href="../../genindex.html" />
<link rel="search" title="Search" href="../../search.html" />
- <link rel="next" title="Apache DataFusion Comet: Benchmarks Derived From TPC-DS" href="tpc-ds.html" />
- <link rel="prev" title="Comet Benchmarking Guide" href="../benchmarking.html" />
+ <link rel="next" title="map_funcs Expression Audits" href="map_funcs.html" />
+ <link rel="prev" title="hash_funcs Expression Audits" href="hash_funcs.html" />
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<meta name="docsearch:language" content="en"/>
<meta name="docsearch:version" content="" />
@@ -418,11 +418,11 @@ under the License.
<li class="toctree-l1"><a class="reference internal" href="../iceberg-spark-tests.html">Iceberg Spark Tests</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Debugging and Performance</span></p>
-<ul class="current nav bd-sidenav">
+<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../debugging.html">Debugging Guide</a></li>
-<li class="toctree-l1 current active has-children"><a class="reference internal" href="../benchmarking.html">Benchmarking Guide</a><details open="open"><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul class="current">
-<li class="toctree-l2 current active"><a class="current reference internal" href="#">Apache DataFusion Comet: Benchmarks Derived From TPC-H</a></li>
-<li class="toctree-l2"><a class="reference internal" href="tpc-ds.html">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</a></li>
+<li class="toctree-l1 has-children"><a class="reference internal" href="../benchmarking.html">Benchmarking Guide</a><details><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul>
+<li class="toctree-l2"><a class="reference internal" href="../benchmark-results/tpc-h.html">Apache DataFusion Comet: Benchmarks Derived From TPC-H</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../benchmark-results/tpc-ds.html">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_macos.html">Comet Benchmarking on macOS</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_aws_ec2.html">Comet Benchmarking in EC2</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_spark_sql_perf.html">TPC-DS Benchmarking with spark-sql-perf</a></li>
@@ -432,8 +432,27 @@ under the License.
<li class="toctree-l1"><a class="reference internal" href="../tracing.html">Tracing</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Reference</span></p>
-<ul class="nav bd-sidenav">
-<li class="toctree-l1"><a class="reference internal" href="../spark_expressions_support.html">Supported Spark Expressions</a></li>
+<ul class="current nav bd-sidenav">
+<li class="toctree-l1 current active has-children"><a class="reference internal" href="index.html">Expression Audits</a><details open="open"><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul class="current">
+<li class="toctree-l2"><a class="reference internal" href="agg_funcs.html">agg_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="array_funcs.html">array_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="bitwise_funcs.html">bitwise_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="collection_funcs.html">collection_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="conditional_funcs.html">conditional_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="conversion_funcs.html">conversion_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="datetime_funcs.html">datetime_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="generator_funcs.html">generator_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="hash_funcs.html">hash_funcs</a></li>
+<li class="toctree-l2 current active"><a class="current reference internal" href="#">json_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="map_funcs.html">map_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="math_funcs.html">math_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="misc_funcs.html">misc_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="predicate_funcs.html">predicate_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="struct_funcs.html">struct_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="url_funcs.html">url_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="window_funcs.html">window_funcs</a></li>
+</ul>
+</details></li>
<li class="toctree-l1"><a class="reference internal" href="../spark_configs_support.html">Supported Spark Configurations</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Project Mechanics</span></p>
@@ -488,9 +507,9 @@ under the License.
<li class="breadcrumb-item"><a href="../index.html" class="nav-link">Comet Contributor Guide</a></li>
- <li class="breadcrumb-item"><a href="../benchmarking.html" class="nav-link">Comet Benchmarking Guide</a></li>
+ <li class="breadcrumb-item"><a href="index.html" class="nav-link">Expression Audits</a></li>
- <li class="breadcrumb-item active" aria-current="page"><span class="ellipsis">Apache DataFusion Comet: Benchmarks Derived From TPC-H</span></li>
+ <li class="breadcrumb-item active" aria-current="page"><span class="ellipsis">json_funcs Expression Audits</span></li>
</ul>
</nav>
</div>
@@ -507,70 +526,50 @@ under the License.
<div id="searchbox"></div>
<article class="bd-article">
- <!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
+ <!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
-->
-<section id="apache-datafusion-comet-benchmarks-derived-from-tpc-h">
-<h1>Apache DataFusion Comet: Benchmarks Derived From TPC-H<a class="headerlink" href="#apache-datafusion-comet-benchmarks-derived-from-tpc-h" title="Link to this heading">#</a></h1>
-<p>The following benchmarks were performed on an EKS cluster (<code class="docutils literal notranslate"><span class="pre">r6i.24xlarge</span></code> instances with EBS storage) with data stored in S3.</p>
-<section id="benchmark-results">
-<h2>Benchmark Results<a class="headerlink" href="#benchmark-results" title="Link to this heading">#</a></h2>
-<p>Total time to run all queries (lower is better).</p>
-<p><img alt="" src="../../_images/tpch_allqueries_with_tuned.png" /></p>
-<p>The following charts are based on the tuned run using hash join.</p>
-<p>Per-query breakdown showing the relative performance of Spark and Comet.</p>
-<p><img alt="" src="../../_images/tpch_queries_compare.png" /></p>
-<p>How much Comet accelerates each query in relative terms.</p>
-<p><img alt="" src="../../_images/tpch_queries_speedup_rel.png" /></p>
-<p>How much Comet accelerates each query in absolute terms.</p>
-<p><img alt="" src="../../_images/tpch_queries_speedup_abs.png" /></p>
+<section id="json-funcs-expression-audits">
+<h1>json_funcs Expression Audits<a class="headerlink" href="#json-funcs-expression-audits" title="Link to this heading">#</a></h1>
+<blockquote>
+<div><p>Audit notes for expressions in this category that have been audited. Absence of an entry means the expression has not been audited yet, not that it is unsupported. See the user guide <a class="reference internal" href="../../user-guide/latest/expressions.html"><span class="std std-doc">Spark Expression Support</span></a> for current support status.</p>
+</div></blockquote>
+<section id="from-json">
+<h2>from_json<a class="headerlink" href="#from-json" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Partial native support, marked <code class="docutils literal notranslate"><span class="pre">Incompatible</span></code> (requires explicit schema).</p></li>
+</ul>
</section>
-<section id="configuration">
-<h2>Configuration<a class="headerlink" href="#configuration" title="Link to this heading">#</a></h2>
-<p>Common:</p>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.executor.instances</span><span class="o">=</span><span class="s">32</span>
-<span class="na">spark.executor.cores</span><span class="o">=</span><span class="s">16</span>
-<span class="na">spark.memory.fraction</span><span class="o">=</span><span class="s">0.6</span>
-<span class="na">spark.memory.storageFraction</span><span class="o">=</span><span class="s">0.2</span>
-<span class="c1"># Kubernetes CPU constraints</span>
-<span class="na">spark.kubernetes.executor.request.cores</span><span class="o">=</span><span class="s">8</span>
-<span class="na">spark.kubernetes.executor.limit.cores</span><span class="o">=</span><span class="s">8</span>
-</pre></div>
-</div>
-<p>Spark:</p>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.executor.memory</span><span class="o">=</span><span class="s">64G</span>
-<span class="na">spark.executor.memoryOverhead</span><span class="o">=</span><span class="s">10G</span>
-</pre></div>
-</div>
-<p>Comet:</p>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.executor.memory</span><span class="o">=</span><span class="s">32G</span>
-<span class="na">spark.executor.memoryOverhead</span><span class="o">=</span><span class="s">10G</span>
-<span class="na">spark.memory.offHeap.enabled</span><span class="o">=</span><span class="s">true</span>
-<span class="na">spark.memory.offHeap.size</span><span class="o">=</span><span class="s">32G</span>
-</pre></div>
-</div>
-<section id="comet-tuned">
-<h3>Comet (Tuned)<a class="headerlink" href="#comet-tuned" title="Link to this heading">#</a></h3>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.comet.exec.replaceSortMergeJoin</span><span class="o">=</span><span class="s">true</span>
-<span class="na">spark.comet.memoryPool.fraction</span><span class="o">=</span><span class="s">0.8</span>
-</pre></div>
-</div>
+<section id="get-json-object">
+<h2>get_json_object<a class="headerlink" href="#get-json-object" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">BinaryExpression</span> <span class="pre">with</span> <span class="pre">ExpectsInputTypes</span> <span class="pre">with</span> <span class="pre">CodegenFallback</span></code>; <code class="docutils literal notranslate"><span class="pre">inputTypes</span> <span class="pre">=</span> <span class="pre">Seq(StringType,</span> <span class="pre">StringType)</span> <span class="pre">-& [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): the eval is extracted into a <code class="docutils literal notranslate"><span class="pre">GetJsonObjectEvaluator</span></code> helper (no behaviour change). The trait set now mixes in <code class="docutils literal notranslate"><span class="pre">DefaultStringProducingExpression</span></code>, and <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code> is widened to <code class="docutils literal notranslate"><span class [...]
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+<li><p>Known incompatibility: Spark accepts single-quoted JSON and unescaped control characters; Comet’s native parser (built on <code class="docutils literal notranslate"><span class="pre">serde_json</span></code>) rejects both, so those inputs require <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.GetJsonObject.allowIncompatible=true</span></code> and may still produce different results. Non-default Spark 4.0 string collations are not propagated (ht [...]
+</ul>
</section>
+<section id="to-json">
+<h2>to_json<a class="headerlink" href="#to-json" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Partial native support; options and map/array inputs fall back.</p></li>
+</ul>
</section>
</section>
@@ -585,20 +584,20 @@ under the License.
<div class="prev-next-area">
<a class="left-prev"
- href="../benchmarking.html"
+ href="hash_funcs.html"
title="previous page">
<i class="fa-solid fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">Comet Benchmarking Guide</p>
+ <p class="prev-next-title">hash_funcs Expression Audits</p>
</div>
</a>
<a class="right-next"
- href="tpc-ds.html"
+ href="map_funcs.html"
title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
- <p class="prev-next-title">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</p>
+ <p class="prev-next-title">map_funcs Expression Audits</p>
</div>
<i class="fa-solid fa-angle-right"></i>
</a>
diff --git a/contributor-guide/benchmark-results/tpc-ds.html b/contributor-guide/expression-audits/map_funcs.html
similarity index 56%
copy from contributor-guide/benchmark-results/tpc-ds.html
copy to contributor-guide/expression-audits/map_funcs.html
index 4d5bf9b1c4..9dc814bf30 100644
--- a/contributor-guide/benchmark-results/tpc-ds.html
+++ b/contributor-guide/expression-audits/map_funcs.html
@@ -27,7 +27,7 @@ under the License.
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
- <title>Apache DataFusion Comet: Benchmarks Derived From TPC-DS — Apache DataFusion Comet documentation</title>
+ <title>map_funcs Expression Audits — Apache DataFusion Comet documentation</title>
@@ -61,12 +61,12 @@ under the License.
<script src="../../_static/documentation_options.js?v=5929fcd5"></script>
<script src="../../_static/doctools.js?v=9a2dae69"></script>
<script src="../../_static/sphinx_highlight.js?v=dc90522c"></script>
- <script>DOCUMENTATION_OPTIONS.pagename = 'contributor-guide/benchmark-results/tpc-ds';</script>
+ <script>DOCUMENTATION_OPTIONS.pagename = 'contributor-guide/expression-audits/map_funcs';</script>
<script async="true" defer="true" src="https://buttons.github.io/buttons.js"></script>
<link rel="index" title="Index" href="../../genindex.html" />
<link rel="search" title="Search" href="../../search.html" />
- <link rel="next" title="Comet Benchmarking on macOS" href="../benchmarking_macos.html" />
- <link rel="prev" title="Apache DataFusion Comet: Benchmarks Derived From TPC-H" href="tpc-h.html" />
+ <link rel="next" title="math_funcs Expression Audits" href="math_funcs.html" />
+ <link rel="prev" title="json_funcs Expression Audits" href="json_funcs.html" />
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<meta name="docsearch:language" content="en"/>
<meta name="docsearch:version" content="" />
@@ -418,11 +418,11 @@ under the License.
<li class="toctree-l1"><a class="reference internal" href="../iceberg-spark-tests.html">Iceberg Spark Tests</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Debugging and Performance</span></p>
-<ul class="current nav bd-sidenav">
+<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../debugging.html">Debugging Guide</a></li>
-<li class="toctree-l1 current active has-children"><a class="reference internal" href="../benchmarking.html">Benchmarking Guide</a><details open="open"><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul class="current">
-<li class="toctree-l2"><a class="reference internal" href="tpc-h.html">Apache DataFusion Comet: Benchmarks Derived From TPC-H</a></li>
-<li class="toctree-l2 current active"><a class="current reference internal" href="#">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</a></li>
+<li class="toctree-l1 has-children"><a class="reference internal" href="../benchmarking.html">Benchmarking Guide</a><details><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul>
+<li class="toctree-l2"><a class="reference internal" href="../benchmark-results/tpc-h.html">Apache DataFusion Comet: Benchmarks Derived From TPC-H</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../benchmark-results/tpc-ds.html">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_macos.html">Comet Benchmarking on macOS</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_aws_ec2.html">Comet Benchmarking in EC2</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_spark_sql_perf.html">TPC-DS Benchmarking with spark-sql-perf</a></li>
@@ -432,8 +432,27 @@ under the License.
<li class="toctree-l1"><a class="reference internal" href="../tracing.html">Tracing</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Reference</span></p>
-<ul class="nav bd-sidenav">
-<li class="toctree-l1"><a class="reference internal" href="../spark_expressions_support.html">Supported Spark Expressions</a></li>
+<ul class="current nav bd-sidenav">
+<li class="toctree-l1 current active has-children"><a class="reference internal" href="index.html">Expression Audits</a><details open="open"><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul class="current">
+<li class="toctree-l2"><a class="reference internal" href="agg_funcs.html">agg_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="array_funcs.html">array_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="bitwise_funcs.html">bitwise_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="collection_funcs.html">collection_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="conditional_funcs.html">conditional_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="conversion_funcs.html">conversion_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="datetime_funcs.html">datetime_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="generator_funcs.html">generator_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="hash_funcs.html">hash_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="json_funcs.html">json_funcs</a></li>
+<li class="toctree-l2 current active"><a class="current reference internal" href="#">map_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="math_funcs.html">math_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="misc_funcs.html">misc_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="predicate_funcs.html">predicate_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="struct_funcs.html">struct_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="url_funcs.html">url_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="window_funcs.html">window_funcs</a></li>
+</ul>
+</details></li>
<li class="toctree-l1"><a class="reference internal" href="../spark_configs_support.html">Supported Spark Configurations</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Project Mechanics</span></p>
@@ -488,9 +507,9 @@ under the License.
<li class="breadcrumb-item"><a href="../index.html" class="nav-link">Comet Contributor Guide</a></li>
- <li class="breadcrumb-item"><a href="../benchmarking.html" class="nav-link">Comet Benchmarking Guide</a></li>
+ <li class="breadcrumb-item"><a href="index.html" class="nav-link">Expression Audits</a></li>
- <li class="breadcrumb-item active" aria-current="page"><span class="ellipsis">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</span></li>
+ <li class="breadcrumb-item active" aria-current="page"><span class="ellipsis">map_funcs Expression Audits</span></li>
</ul>
</nav>
</div>
@@ -507,63 +526,101 @@ under the License.
<div id="searchbox"></div>
<article class="bd-article">
- <!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
+ <!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
-->
-<section id="apache-datafusion-comet-benchmarks-derived-from-tpc-ds">
-<h1>Apache DataFusion Comet: Benchmarks Derived From TPC-DS<a class="headerlink" href="#apache-datafusion-comet-benchmarks-derived-from-tpc-ds" title="Link to this heading">#</a></h1>
-<p>The following benchmarks were performed on an EKS cluster (<code class="docutils literal notranslate"><span class="pre">r6i.24xlarge</span></code> instances with EBS storage) with data stored in S3.</p>
-<section id="benchmark-results">
-<h2>Benchmark Results<a class="headerlink" href="#benchmark-results" title="Link to this heading">#</a></h2>
-<p>Total time to run all queries (lower is better).</p>
-<p><img alt="" src="../../_images/tpcds_allqueries.png" /></p>
-<p>Per-query breakdown showing the relative performance of Spark and Comet.</p>
-<p><img alt="" src="../../_images/tpcds_queries_compare.png" /></p>
-<p>How much Comet accelerates each query in relative terms.</p>
-<p><img alt="" src="../../_images/tpcds_queries_speedup_rel.png" /></p>
-<p>How much Comet accelerates each query in absolute terms.</p>
-<p><img alt="" src="../../_images/tpcds_queries_speedup_abs.png" /></p>
+<section id="map-funcs-expression-audits">
+<h1>map_funcs Expression Audits<a class="headerlink" href="#map-funcs-expression-audits" title="Link to this heading">#</a></h1>
+<blockquote>
+<div><p>Audit notes for expressions in this category that have been audited. Absence of an entry means the expression has not been audited yet, not that it is unsupported. See the user guide <a class="reference internal" href="../../user-guide/latest/expressions.html"><span class="std std-doc">Spark Expression Support</span></a> for current support status.</p>
+</div></blockquote>
+<section id="element-at">
+<h2>element_at<a class="headerlink" href="#element-at" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">ElementAt(left,</span> <span class="pre">right,</span> <span class="pre">defaultValueOutOfBound,</span> <span class="pre">failOnError)</span> <span class="pre">extends</span> <span class="pre">GetMapValueUtil</span></code>; the parser routes <code class="docutils literal notranslate"><span class="pre">element_at(<array>,</span> <span class="pre">...)</span></code> to one [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): adds <code class="docutils literal notranslate"><span class="pre">nullIntolerant:</span> <span class="pre">Boolean</span></code> field; semantics unchanged.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
</section>
-<section id="configuration">
-<h2>Configuration<a class="headerlink" href="#configuration" title="Link to this heading">#</a></h2>
-<p>Common:</p>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.executor.instances</span><span class="o">=</span><span class="s">32</span>
-<span class="na">spark.executor.cores</span><span class="o">=</span><span class="s">16</span>
-<span class="na">spark.memory.fraction</span><span class="o">=</span><span class="s">0.6</span>
-<span class="na">spark.memory.storageFraction</span><span class="o">=</span><span class="s">0.2</span>
-<span class="c1"># Kubernetes CPU constraints</span>
-<span class="na">spark.kubernetes.executor.request.cores</span><span class="o">=</span><span class="s">8</span>
-<span class="na">spark.kubernetes.executor.limit.cores</span><span class="o">=</span><span class="s">8</span>
-</pre></div>
-</div>
-<p>Spark:</p>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.executor.memory</span><span class="o">=</span><span class="s">64G</span>
-<span class="na">spark.executor.memoryOverhead</span><span class="o">=</span><span class="s">10G</span>
-</pre></div>
-</div>
-<p>Comet:</p>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.executor.memory</span><span class="o">=</span><span class="s">64G</span>
-<span class="na">spark.executor.memoryOverhead</span><span class="o">=</span><span class="s">10G</span>
-<span class="na">spark.memory.offHeap.enabled</span><span class="o">=</span><span class="s">true</span>
-<span class="na">spark.memory.offHeap.size</span><span class="o">=</span><span class="s">32G</span>
-<span class="na">spark.comet.memoryPool.fraction</span><span class="o">=</span><span class="s">0.8</span>
-</pre></div>
-</div>
+<section id="map-contains-key">
+<h2>map_contains_key<a class="headerlink" href="#map-contains-key" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">MapContainsKey(left,</span> <span class="pre">right)</span> <span class="pre">extends</span> <span class="pre">RuntimeReplaceable</span> <span class="pre">with</span> <span class="pre">InheritAnalysisRules</span></code>; the analyzer rewrites to <code class="docutils literal notranslate"><span class="pre">ArrayContains(MapKeys(left),</span> <span class="pre">right)</span></code [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): semantics unchanged; minor trait refactors.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</section>
+<section id="map-entries">
+<h2>map_entries<a class="headerlink" href="#map-entries" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">MapEntries(child)</span></code> returns an array of structs <code class="docutils literal notranslate"><span class="pre"><key,</span> <span class="pre">value></span></code>. Wired to native <code class="docutils literal notranslate"><span class="pre">map_entries</span></code>.</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): semantics unchanged.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</section>
+<section id="map-from-arrays">
+<h2>map_from_arrays<a class="headerlink" href="#map-from-arrays" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">MapFromArrays(left,</span> <span class="pre">right)</span> <span class="pre">extends</span> <span class="pre">BinaryExpression</span> <span class="pre">with</span> <span class="pre">NullIntolerant</span></code>; Spark uses <code class="docutils literal notranslate"><span class="pre">ArrayBasedMapBuilder</span></code> to detect duplicate keys (subject to <code class="docutils li [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): semantics unchanged; <code class="docutils literal notranslate"><span class="pre">NullIntolerant</span></code> trait replaced by <code class="docutils literal notranslate"><span class="pre">nullIntolerant:</span> <span class="pre">Boolean</span></code>.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</section>
+<section id="map-from-entries">
+<h2>map_from_entries<a class="headerlink" href="#map-from-entries" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">MapFromEntries(child)</span> <span class="pre">extends</span> <span class="pre">UnaryExpression</span> <span class="pre">with</span> <span class="pre">NullIntolerant</span></code>; expects an array of structs and produces a map. Wired as <code class="docutils literal notranslate"><span class="pre">CometScalarFunction("map_from_entries")</span></code>.</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): semantics unchanged; trait refactor.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+<li><p>Known limitation: input arrays where the struct’s key or value type contains <code class="docutils literal notranslate"><span class="pre">BinaryType</span></code> are marked <code class="docutils literal notranslate"><span class="pre">Incompatible</span></code> and fall back unless <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.MapFromEntries.allowIncompatible=true</span></code>.</p></li>
+</ul>
+</section>
+<section id="map-keys">
+<h2>map_keys<a class="headerlink" href="#map-keys" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">MapKeys(child)</span></code> returns the map’s keys as an array. Wired to native <code class="docutils literal notranslate"><span class="pre">map_keys</span></code>.</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): semantics unchanged.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</section>
+<section id="map-values">
+<h2>map_values<a class="headerlink" href="#map-values" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">MapValues(child)</span></code> returns the map’s values as an array. Wired to native <code class="docutils literal notranslate"><span class="pre">map_values</span></code>.</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): semantics unchanged.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</section>
+<section id="str-to-map">
+<h2>str_to_map<a class="headerlink" href="#str-to-map" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">StringToMap(text,</span> <span class="pre">pairDelim,</span> <span class="pre">keyValueDelim)</span> <span class="pre">extends</span> <span class="pre">TernaryExpression</span></code>; splits <code class="docutils literal notranslate"><span class="pre">text</span></code> on <code class="docutils literal notranslate"><span class="pre">pairDelim</span></code>, then each pair on < [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code> widened to <code class="docutils literal notranslate"><span class="pre">StringTypeNonCSAICollation</span></code>; uses <code class="docutils literal notranslate"><span class="pre">CollationAwareUTF8String.splitSQL</span></code> with a <code class="docutils literal notranslate"><span class="pre">collationId</span></code>. Runtime unchanged for <code class="docutil [...]
+<li><p>Spark 4.1.1 (audited 2026-05-27): adds the <code class="docutils literal notranslate"><span class="pre">legacySplitTruncate</span></code> flag (driven by <code class="docutils literal notranslate"><span class="pre">spark.sql.legacy.truncateForEmptyRegexSplit</span></code>) to both <code class="docutils literal notranslate"><span class="pre">splitSQL</span></code> calls (https://github.com/apache/datafusion-comet/issues/4477). The Comet native impl does not honour this flag; behavi [...]
+</ul>
</section>
</section>
@@ -578,20 +635,20 @@ under the License.
<div class="prev-next-area">
<a class="left-prev"
- href="tpc-h.html"
+ href="json_funcs.html"
title="previous page">
<i class="fa-solid fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">Apache DataFusion Comet: Benchmarks Derived From TPC-H</p>
+ <p class="prev-next-title">json_funcs Expression Audits</p>
</div>
</a>
<a class="right-next"
- href="../benchmarking_macos.html"
+ href="math_funcs.html"
title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
- <p class="prev-next-title">Comet Benchmarking on macOS</p>
+ <p class="prev-next-title">math_funcs Expression Audits</p>
</div>
<i class="fa-solid fa-angle-right"></i>
</a>
diff --git a/contributor-guide/expression-audits/math_funcs.html b/contributor-guide/expression-audits/math_funcs.html
new file mode 100644
index 0000000000..1d678f5532
--- /dev/null
+++ b/contributor-guide/expression-audits/math_funcs.html
@@ -0,0 +1,988 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+
+<!DOCTYPE html>
+
+
+<html lang="en" data-content_root="../../" data-theme="light">
+
+ <head>
+ <meta charset="utf-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
+
+ <title>math_funcs Expression Audits — Apache DataFusion Comet documentation</title>
+
+
+
+ <script data-cfasync="false">
+ document.documentElement.dataset.mode = localStorage.getItem("mode") || "light";
+ document.documentElement.dataset.theme = localStorage.getItem("theme") || "light";
+ </script>
+ <!--
+ this give us a css class that will be invisible only if js is disabled
+ -->
+ <noscript>
+ <style>
+ .pst-js-only { display: none !important; }
+
+ </style>
+ </noscript>
+
+ <!-- Loaded before other Sphinx assets -->
+ <link href="../../_static/styles/theme.css?digest=8878045cc6db502f8baf" rel="stylesheet" />
+<link href="../../_static/styles/pydata-sphinx-theme.css?digest=8878045cc6db502f8baf" rel="stylesheet" />
+
+ <link rel="stylesheet" type="text/css" href="../../_static/pygments.css?v=8f2a1f02" />
+ <link rel="stylesheet" type="text/css" href="../../_static/theme_overrides.css?v=cd442bcd" />
+
+ <!-- So that users can add custom icons -->
+ <script src="../../_static/scripts/fontawesome.js?digest=8878045cc6db502f8baf"></script>
+ <!-- Pre-loaded scripts that we'll load fully later -->
+ <link rel="preload" as="script" href="../../_static/scripts/bootstrap.js?digest=8878045cc6db502f8baf" />
+<link rel="preload" as="script" href="../../_static/scripts/pydata-sphinx-theme.js?digest=8878045cc6db502f8baf" />
+
+ <script src="../../_static/documentation_options.js?v=5929fcd5"></script>
+ <script src="../../_static/doctools.js?v=9a2dae69"></script>
+ <script src="../../_static/sphinx_highlight.js?v=dc90522c"></script>
+ <script>DOCUMENTATION_OPTIONS.pagename = 'contributor-guide/expression-audits/math_funcs';</script>
+ <script async="true" defer="true" src="https://buttons.github.io/buttons.js"></script>
+ <link rel="index" title="Index" href="../../genindex.html" />
+ <link rel="search" title="Search" href="../../search.html" />
+ <link rel="next" title="misc_funcs Expression Audits" href="misc_funcs.html" />
+ <link rel="prev" title="map_funcs Expression Audits" href="map_funcs.html" />
+ <meta name="viewport" content="width=device-width, initial-scale=1"/>
+ <meta name="docsearch:language" content="en"/>
+ <meta name="docsearch:version" content="" />
+ </head>
+
+
+ <body data-bs-spy="scroll" data-bs-target=".bd-toc-nav" data-offset="180" data-bs-root-margin="0px 0px -60%" data-default-mode="light">
+
+
+
+ <div id="pst-skip-link" class="skip-link d-print-none"><a href="#main-content">Skip to main content</a></div>
+
+ <div id="pst-scroll-pixel-helper"></div>
+
+ <button type="button" class="btn rounded-pill" id="pst-back-to-top">
+ <i class="fa-solid fa-arrow-up"></i>Back to top</button>
+
+
+ <dialog id="pst-search-dialog">
+
+<form class="bd-search d-flex align-items-center"
+ action="../../search.html"
+ method="get">
+ <i class="fa-solid fa-magnifying-glass"></i>
+ <input type="search"
+ class="form-control"
+ name="q"
+ placeholder="Search the docs ..."
+ aria-label="Search the docs ..."
+ autocomplete="off"
+ autocorrect="off"
+ autocapitalize="off"
+ spellcheck="false"/>
+ <span class="search-button__kbd-shortcut"><kbd class="kbd-shortcut__modifier">Ctrl</kbd>+<kbd>K</kbd></span>
+</form>
+ </dialog>
+
+ <div class="pst-async-banner-revealer d-none">
+ <aside id="bd-header-version-warning" class="d-none d-print-none" aria-label="Version warning"></aside>
+</div>
+
+
+ <header class="bd-header navbar navbar-expand-lg bd-navbar d-print-none">
+<div class="bd-header__inner bd-page-width">
+ <button class="pst-navbar-icon sidebar-toggle primary-toggle" aria-label="Site navigation">
+ <span class="fa-solid fa-bars"></span>
+ </button>
+
+
+ <div class="col-lg-3 navbar-header-items__start">
+
+ <div class="navbar-item">
+
+
+
+
+
+<a class="navbar-brand logo" href="../../index.html">
+
+
+
+
+
+
+
+
+ <img src="../../_static/DataFusionComet-Logo-Light.png" class="logo__image only-light" alt="Apache DataFusion Comet documentation - Home"/>
+ <img src="../../_static/DataFusionComet-Logo-Dark.png" class="logo__image only-dark pst-js-only" alt="Apache DataFusion Comet documentation - Home"/>
+
+
+</a></div>
+
+ </div>
+
+ <div class="col-lg-9 navbar-header-items">
+
+ <div class="me-auto navbar-header-items__center">
+
+ <div class="navbar-item">
+<nav>
+ <ul class="bd-navbar-elements navbar-nav">
+
+<li class="nav-item ">
+ <a class="nav-link nav-internal" href="../../user-guide/index.html">
+ User Guide
+ </a>
+</li>
+
+
+<li class="nav-item current active">
+ <a class="nav-link nav-internal" href="../index.html">
+ Contributor Guide
+ </a>
+</li>
+
+
+<li class="nav-item ">
+ <a class="nav-link nav-internal" href="../../changelog/index.html">
+ Changelog
+ </a>
+</li>
+
+
+<li class="nav-item ">
+ <a class="nav-link nav-internal" href="../../about/gluten_comparison.html">
+ Comparison with Gluten
+ </a>
+</li>
+
+
+<li class="nav-item ">
+ <a class="nav-link nav-internal" href="../../about/versioning_policy.html">
+ Versioning Policy
+ </a>
+</li>
+
+ <li class="nav-item dropdown">
+ <button class="btn dropdown-toggle nav-item" type="button"
+ data-bs-toggle="dropdown" aria-expanded="false"
+ aria-controls="pst-nav-more-links">
+ More
+ </button>
+ <ul id="pst-nav-more-links" class="dropdown-menu">
+
+<li class=" ">
+ <a class="nav-link dropdown-item nav-internal" href="../../asf/index.html">
+ ASF Links
+ </a>
+</li>
+
+ </ul>
+ </li>
+
+ </ul>
+</nav></div>
+
+ </div>
+
+
+ <div class="navbar-header-items__end">
+
+ <div class="navbar-item navbar-persistent--container">
+
+
+<button class="btn search-button-field search-button__button pst-js-only" title="Search" aria-label="Search" data-bs-placement="bottom" data-bs-toggle="tooltip">
+ <i class="fa-solid fa-magnifying-glass"></i>
+ <span class="search-button__default-text">Search</span>
+ <span class="search-button__kbd-shortcut"><kbd class="kbd-shortcut__modifier">Ctrl</kbd>+<kbd class="kbd-shortcut__modifier">K</kbd></span>
+</button>
+ </div>
+
+
+ <div class="navbar-item"><ul class="navbar-icon-links"
+ aria-label="Icon Links">
+ <li class="nav-item">
+
+
+
+
+
+
+
+
+ <a href="https://github.com/apache/datafusion-comet" title="GitHub" class="nav-link pst-navbar-icon" rel="noopener" target="_blank" data-bs-toggle="tooltip" data-bs-placement="bottom"><i class="fa-brands fa-github fa-lg" aria-hidden="true"></i>
+ <span class="sr-only">GitHub</span></a>
+ </li>
+</ul></div>
+
+ <div class="navbar-item">
+
+<button class="btn btn-sm nav-link pst-navbar-icon theme-switch-button pst-js-only" aria-label="Color mode" data-bs-title="Color mode" data-bs-placement="bottom" data-bs-toggle="tooltip">
+ <i class="theme-switch fa-solid fa-sun fa-lg" data-mode="light" title="Light"></i>
+ <i class="theme-switch fa-solid fa-moon fa-lg" data-mode="dark" title="Dark"></i>
+ <i class="theme-switch fa-solid fa-circle-half-stroke fa-lg" data-mode="auto" title="System Settings"></i>
+</button></div>
+
+ </div>
+
+ </div>
+
+
+ <div class="navbar-persistent--mobile">
+
+<button class="btn search-button-field search-button__button pst-js-only" title="Search" aria-label="Search" data-bs-placement="bottom" data-bs-toggle="tooltip">
+ <i class="fa-solid fa-magnifying-glass"></i>
+ <span class="search-button__default-text">Search</span>
+ <span class="search-button__kbd-shortcut"><kbd class="kbd-shortcut__modifier">Ctrl</kbd>+<kbd class="kbd-shortcut__modifier">K</kbd></span>
+</button>
+ </div>
+
+
+
+</div>
+
+ </header>
+
+
+ <div class="bd-container">
+ <div class="bd-container__inner bd-page-width">
+
+
+
+ <dialog id="pst-primary-sidebar-modal"></dialog>
+ <div id="pst-primary-sidebar" class="bd-sidebar-primary bd-sidebar">
+
+
+
+ <div class="sidebar-header-items sidebar-primary__section">
+
+
+ <div class="sidebar-header-items__center">
+
+
+
+ <div class="navbar-item">
+<nav>
+ <ul class="bd-navbar-elements navbar-nav">
+
+<li class="nav-item ">
+ <a class="nav-link nav-internal" href="../../user-guide/index.html">
+ User Guide
+ </a>
+</li>
+
+
+<li class="nav-item current active">
+ <a class="nav-link nav-internal" href="../index.html">
+ Contributor Guide
+ </a>
+</li>
+
+
+<li class="nav-item ">
+ <a class="nav-link nav-internal" href="../../changelog/index.html">
+ Changelog
+ </a>
+</li>
+
+
+<li class="nav-item ">
+ <a class="nav-link nav-internal" href="../../about/gluten_comparison.html">
+ Comparison with Gluten
+ </a>
+</li>
+
+
+<li class="nav-item ">
+ <a class="nav-link nav-internal" href="../../about/versioning_policy.html">
+ Versioning Policy
+ </a>
+</li>
+
+
+<li class="nav-item ">
+ <a class="nav-link nav-internal" href="../../asf/index.html">
+ ASF Links
+ </a>
+</li>
+
+ </ul>
+</nav></div>
+
+
+ </div>
+
+
+
+ <div class="sidebar-header-items__end">
+
+ <div class="navbar-item"><ul class="navbar-icon-links"
+ aria-label="Icon Links">
+ <li class="nav-item">
+
+
+
+
+
+
+
+
+ <a href="https://github.com/apache/datafusion-comet" title="GitHub" class="nav-link pst-navbar-icon" rel="noopener" target="_blank" data-bs-toggle="tooltip" data-bs-placement="bottom"><i class="fa-brands fa-github fa-lg" aria-hidden="true"></i>
+ <span class="sr-only">GitHub</span></a>
+ </li>
+</ul></div>
+
+ <div class="navbar-item">
+
+<button class="btn btn-sm nav-link pst-navbar-icon theme-switch-button pst-js-only" aria-label="Color mode" data-bs-title="Color mode" data-bs-placement="bottom" data-bs-toggle="tooltip">
+ <i class="theme-switch fa-solid fa-sun fa-lg" data-mode="light" title="Light"></i>
+ <i class="theme-switch fa-solid fa-moon fa-lg" data-mode="dark" title="Dark"></i>
+ <i class="theme-switch fa-solid fa-circle-half-stroke fa-lg" data-mode="auto" title="System Settings"></i>
+</button></div>
+
+ </div>
+
+ </div>
+
+ <div class="sidebar-primary-items__start sidebar-primary__section">
+ <div class="sidebar-primary-item"><!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+
+
+
+
+<nav class="bd-links" id="bd-docs-nav" aria-label="Main navigation">
+ <div class="bd-toc-item active">
+ <p aria-level="2" class="caption" role="heading"><span class="caption-text">Getting Started</span></p>
+<ul class="nav bd-sidenav">
+<li class="toctree-l1"><a class="reference internal" href="../contributing.html">Getting Started</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../development.html">Development Guide</a></li>
+</ul>
+<p aria-level="2" class="caption" role="heading"><span class="caption-text">Project Architecture</span></p>
+<ul class="nav bd-sidenav">
+<li class="toctree-l1"><a class="reference internal" href="../plugin_overview.html">Comet Plugin Architecture</a></li>
+
+<li class="toctree-l1"><a class="reference internal" href="../ffi.html">Arrow FFI</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../jvm_shuffle.html">JVM Shuffle</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../native_shuffle.html">Native Shuffle</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../sql_error_propagation.html">ANSI Error Propagation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../s3-credential-provider-design.html">S3 Credential Provider Design</a></li>
+</ul>
+<p aria-level="2" class="caption" role="heading"><span class="caption-text">Adding Functionality</span></p>
+<ul class="nav bd-sidenav">
+<li class="toctree-l1"><a class="reference internal" href="../adding_a_new_operator.html">Adding a New Operator</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../adding_a_new_expression.html">Adding a New Expression</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../adding_a_new_spark_version.html">Adding a New Spark Version</a></li>
+</ul>
+<p aria-level="2" class="caption" role="heading"><span class="caption-text">Testing</span></p>
+<ul class="nav bd-sidenav">
+<li class="toctree-l1"><a class="reference internal" href="../sql-file-tests.html">Comet SQL Tests</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../spark-sql-tests.html">Spark SQL Tests</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../iceberg-spark-tests.html">Iceberg Spark Tests</a></li>
+</ul>
+<p aria-level="2" class="caption" role="heading"><span class="caption-text">Debugging and Performance</span></p>
+<ul class="nav bd-sidenav">
+<li class="toctree-l1"><a class="reference internal" href="../debugging.html">Debugging Guide</a></li>
+<li class="toctree-l1 has-children"><a class="reference internal" href="../benchmarking.html">Benchmarking Guide</a><details><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul>
+<li class="toctree-l2"><a class="reference internal" href="../benchmark-results/tpc-h.html">Apache DataFusion Comet: Benchmarks Derived From TPC-H</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../benchmark-results/tpc-ds.html">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../benchmarking_macos.html">Comet Benchmarking on macOS</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../benchmarking_aws_ec2.html">Comet Benchmarking in EC2</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../benchmarking_spark_sql_perf.html">TPC-DS Benchmarking with spark-sql-perf</a></li>
+</ul>
+</details></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tracing.html">Tracing</a></li>
+</ul>
+<p aria-level="2" class="caption" role="heading"><span class="caption-text">Reference</span></p>
+<ul class="current nav bd-sidenav">
+<li class="toctree-l1 current active has-children"><a class="reference internal" href="index.html">Expression Audits</a><details open="open"><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul class="current">
+<li class="toctree-l2"><a class="reference internal" href="agg_funcs.html">agg_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="array_funcs.html">array_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="bitwise_funcs.html">bitwise_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="collection_funcs.html">collection_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="conditional_funcs.html">conditional_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="conversion_funcs.html">conversion_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="datetime_funcs.html">datetime_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="generator_funcs.html">generator_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="hash_funcs.html">hash_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="json_funcs.html">json_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="map_funcs.html">map_funcs</a></li>
+<li class="toctree-l2 current active"><a class="current reference internal" href="#">math_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="misc_funcs.html">misc_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="predicate_funcs.html">predicate_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="struct_funcs.html">struct_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="url_funcs.html">url_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="window_funcs.html">window_funcs</a></li>
+</ul>
+</details></li>
+<li class="toctree-l1"><a class="reference internal" href="../spark_configs_support.html">Supported Spark Configurations</a></li>
+</ul>
+<p aria-level="2" class="caption" role="heading"><span class="caption-text">Project Mechanics</span></p>
+<ul class="nav bd-sidenav">
+<li class="toctree-l1"><a class="reference internal" href="../bug_triage.html">Bug Triage</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../release_process.html">Release Process</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../roadmap.html">Roadmap</a></li>
+<li class="toctree-l1"><a class="reference external" href="https://github.com/apache/datafusion-comet">Github and Issue Tracker</a></li>
+</ul>
+
+ </div>
+</nav>
+</div>
+ </div>
+
+
+ <div class="sidebar-primary-items__end sidebar-primary__section">
+ <div class="sidebar-primary-item">
+<div id="ethical-ad-placement"
+ class="flat"
+ data-ea-publisher="readthedocs"
+ data-ea-type="readthedocs-sidebar"
+ data-ea-manual="true">
+</div></div>
+ </div>
+
+
+ </div>
+
+ <main id="main-content" class="bd-main" role="main">
+
+
+ <div class="bd-content">
+ <div class="bd-article-container">
+
+ <div class="bd-header-article d-print-none">
+<div class="header-article-items header-article__inner">
+
+ <div class="header-article-items__start">
+
+ <div class="header-article-item">
+
+<nav aria-label="Breadcrumb" class="d-print-none">
+ <ul class="bd-breadcrumbs">
+
+ <li class="breadcrumb-item breadcrumb-home">
+ <a href="../../index.html" class="nav-link" aria-label="Home">
+ <i class="fa-solid fa-home"></i>
+ </a>
+ </li>
+
+ <li class="breadcrumb-item"><a href="../index.html" class="nav-link">Comet Contributor Guide</a></li>
+
+
+ <li class="breadcrumb-item"><a href="index.html" class="nav-link">Expression Audits</a></li>
+
+ <li class="breadcrumb-item active" aria-current="page"><span class="ellipsis">math_funcs Expression Audits</span></li>
+ </ul>
+</nav>
+</div>
+
+ </div>
+
+
+</div>
+</div>
+
+
+
+
+<div id="searchbox"></div>
+ <article class="bd-article">
+
+ <!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+<section id="math-funcs-expression-audits">
+<h1>math_funcs Expression Audits<a class="headerlink" href="#math-funcs-expression-audits" title="Link to this heading">#</a></h1>
+<blockquote>
+<div><p>Audit notes for expressions in this category that have been audited. Absence of an entry means the expression has not been audited yet, not that it is unsupported. See the user guide <a class="reference internal" href="../../user-guide/latest/expressions.html"><span class="std std-doc">Spark Expression Support</span></a> for current support status.</p>
+</div></blockquote>
+<section id="id1">
+<h2><code class="docutils literal notranslate"><span class="pre">%</span></code><a class="headerlink" href="#id1" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">Remainder(left,</span> <span class="pre">right,</span> <span class="pre">evalMode)</span></code> signature identical across versions. Native path uses Rust <code class="docutils literal notranslate"><span class="pre">spark_modulo</span></code> UDF; non-ANSI returns NULL on divide-by-zero, ANSI raises <code class="docutils literal notranslate"><span class="pre">DIVIDE [...]
+</ul>
+</section>
+<section id="id2">
+<h2><code class="docutils literal notranslate"><span class="pre">*</span></code><a class="headerlink" href="#id2" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">Multiply(left,</span> <span class="pre">right,</span> <span class="pre">evalMode)</span></code> signature identical. Decimal results exceeding <code class="docutils literal notranslate"><span class="pre">DECIMAL128_MAX_PRECISION</span></code> go through <code class="docutils literal notranslate"><span class="pre">WideDecimalBinaryExpr</span></code> (Decimal256 interm [...]
+</ul>
+</section>
+<section id="id3">
+<h2><code class="docutils literal notranslate"><span class="pre">+</span></code><a class="headerlink" href="#id3" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">Add(left,</span> <span class="pre">right,</span> <span class="pre">evalMode)</span></code> with the same Decimal / ANSI plumbing as <code class="docutils literal notranslate"><span class="pre">*</span></code>. <code class="docutils literal notranslate"><span class="pre">Date</span> <span class="pre">+</span> <span class="pre">Int8/16/32</span></code> dispatches to th [...]
+</ul>
+</section>
+<section id="id4">
+<h2><code class="docutils literal notranslate"><span class="pre">-</span></code><a class="headerlink" href="#id4" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">Subtract(left,</span> <span class="pre">right,</span> <span class="pre">evalMode)</span></code> mirrors <code class="docutils literal notranslate"><span class="pre">+</span></code>. <code class="docutils literal notranslate"><span class="pre">Date</span> <span class="pre">-</span> <span class="pre">Int8/16/32</span></code> uses the Rust <code class="docutils literal [...]
+</ul>
+</section>
+<section id="id5">
+<h2><code class="docutils literal notranslate"><span class="pre">/</span></code><a class="headerlink" href="#id5" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">Divide(left,</span> <span class="pre">right,</span> <span class="pre">evalMode)</span></code>. Non-ANSI mode wraps the divisor in <code class="docutils literal notranslate"><span class="pre">If(EqualTo(right,</span> <span class="pre">0),</span> <span class="pre">null,</span> <span class="pre">right)</span></code> so DataFusion never throws. Decimal output is wrapped [...]
+</ul>
+</section>
+<section id="abs">
+<h2>abs<a class="headerlink" href="#abs" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">Abs(child,</span> <span class="pre">failOnError)</span></code> over <code class="docutils literal notranslate"><span class="pre">NumericType</span></code> plus the two interval types. <code class="docutils literal notranslate"><span class="pre">failOnError</span></code> (ANSI) is propagated to the native <code class="docutils literal notranslate"><span class="pre">ab [...]
+</ul>
+</section>
+<section id="acos">
+<h2>acos<a class="headerlink" href="#acos" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">UnaryMathExpression(math.acos,</span> <span class="pre">"ACOS")</span></code> unchanged across versions; wired as <code class="docutils literal notranslate"><span class="pre">CometScalarFunction("acos")</span></code> to DataFusion’s <code class="docutils literal notranslate"><span class="pre">acos</span></code> UDF. NaN for <code class="docutils l [...]
+</ul>
+</section>
+<section id="acosh">
+<h2>acosh<a class="headerlink" href="#acosh" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): custom <code class="docutils literal notranslate"><span class="pre">StrictMath.log(x</span> <span class="pre">+</span> <span class="pre">sqrt(x*x</span> <span class="pre">-</span> <span class="pre">1))</span></code> unchanged across versions. NaN for <code class="docutils literal notranslate"><span class="pre">x</span> <span class="pre"><</span> <span class="pre">1</span></code>. Routes to DataFusion’s <code class="docutils [...]
+</ul>
+</section>
+<section id="asin">
+<h2>asin<a class="headerlink" href="#asin" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">UnaryMathExpression(math.asin,</span> <span class="pre">"ASIN")</span></code> unchanged. NaN for <code class="docutils literal notranslate"><span class="pre">|x|</span> <span class="pre">></span> <span class="pre">1</span></code>.</p></li>
+</ul>
+</section>
+<section id="asinh">
+<h2>asinh<a class="headerlink" href="#asinh" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): special-cases <code class="docutils literal notranslate"><span class="pre">Double.NegativeInfinity</span></code> to avoid <code class="docutils literal notranslate"><span class="pre">log(NaN)</span></code>, otherwise <code class="docutils literal notranslate"><span class="pre">StrictMath.log(x</span> <span class="pre">+</span> <span class="pre">sqrt(x*x</span> <span class="pre">+</span> <span class="pre">1))</span></code>. Ide [...]
+</ul>
+</section>
+<section id="atan">
+<h2>atan<a class="headerlink" href="#atan" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">UnaryMathExpression(math.atan,</span> <span class="pre">"ATAN")</span></code> unchanged.</p></li>
+</ul>
+</section>
+<section id="atan2">
+<h2>atan2<a class="headerlink" href="#atan2" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">BinaryMathExpression(math.atan2,</span> <span class="pre">"ATAN2")</span></code> with both inputs adjusted by <code class="docutils literal notranslate"><span class="pre">+0.0</span></code> to flip <code class="docutils literal notranslate"><span class="pre">-0.0</span></code> to <code class="docutils literal notranslate"><span class="pre">+0.0</span></code [...]
+</ul>
+</section>
+<section id="atanh">
+<h2>atanh<a class="headerlink" href="#atanh" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): custom <code class="docutils literal notranslate"><span class="pre">0.5</span> <span class="pre">*</span> <span class="pre">(log1p(x)</span> <span class="pre">-</span> <span class="pre">log1p(-x))</span></code> (SPARK-28519). NaN for <code class="docutils literal notranslate"><span class="pre">|x|</span> <span class="pre">></span> <span class="pre">1</span></code>, +/-Infinity for <code class="docutils literal notranslate"> [...]
+</ul>
+</section>
+<section id="bin">
+<h2>bin<a class="headerlink" href="#bin" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">Bin(child)</span></code> over <code class="docutils literal notranslate"><span class="pre">LongType</span> <span class="pre">-></span> <span class="pre">StringType</span></code>. Spark 4.x gains <code class="docutils literal notranslate"><span class="pre">DefaultStringProducingExpression</span></code> and the <code class="docutils literal notranslate"><span class= [...]
+</ul>
+</section>
+<section id="cbrt">
+<h2>cbrt<a class="headerlink" href="#cbrt" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): passthrough to DataFusion <code class="docutils literal notranslate"><span class="pre">cbrt</span></code>.</p></li>
+</ul>
+</section>
+<section id="ceil">
+<h2>ceil<a class="headerlink" href="#ceil" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): one-arg <code class="docutils literal notranslate"><span class="pre">ceil(expr)</span></code> supported (<code class="docutils literal notranslate"><span class="pre">LongType</span></code> / <code class="docutils literal notranslate"><span class="pre">DoubleType</span></code> / <code class="docutils literal notranslate"><span class="pre">DecimalType</span></code> with scale >= 0). Decimal with negative scale falls back at c [...]
+</ul>
+</section>
+<section id="ceiling">
+<h2>ceiling<a class="headerlink" href="#ceiling" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): registry alias for <code class="docutils literal notranslate"><span class="pre">Ceil</span></code>. Same support as <code class="docutils literal notranslate"><span class="pre">ceil</span></code>.</p></li>
+</ul>
+</section>
+<section id="cos">
+<h2>cos<a class="headerlink" href="#cos" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">UnaryMathExpression(math.cos,</span> <span class="pre">"COS")</span></code> unchanged across versions.</p></li>
+</ul>
+</section>
+<section id="cosh">
+<h2>cosh<a class="headerlink" href="#cosh" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">UnaryMathExpression(math.cosh,</span> <span class="pre">"COSH")</span></code> unchanged.</p></li>
+</ul>
+</section>
+<section id="cot">
+<h2>cot<a class="headerlink" href="#cot" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): custom <code class="docutils literal notranslate"><span class="pre">1</span> <span class="pre">/</span> <span class="pre">math.tan(x)</span></code>. DataFusion’s <code class="docutils literal notranslate"><span class="pre">cot</span></code> is also <code class="docutils literal notranslate"><span class="pre">1.0</span> <span class="pre">/</span> <span class="pre">tan(x)</span></code>, so the result matches.</p></li>
+</ul>
+</section>
+<section id="csc">
+<h2>csc<a class="headerlink" href="#csc" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): custom <code class="docutils literal notranslate"><span class="pre">1</span> <span class="pre">/</span> <span class="pre">math.sin(x)</span></code>. Routed to datafusion-spark’s <code class="docutils literal notranslate"><span class="pre">SparkCsc</span></code> (registered in <code class="docutils literal notranslate"><span class="pre">jni_api.rs</span></code>).</p></li>
+</ul>
+</section>
+<section id="degrees">
+<h2>degrees<a class="headerlink" href="#degrees" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">UnaryMathExpression(math.toDegrees,</span> <span class="pre">"DEGREES")</span></code> unchanged across versions.</p></li>
+</ul>
+</section>
+<section id="div">
+<h2>div<a class="headerlink" href="#div" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">IntegralDivide(left,</span> <span class="pre">right,</span> <span class="pre">evalMode)</span></code>. Non-decimal operands are cast to <code class="docutils literal notranslate"><span class="pre">DecimalType(19,</span> <span class="pre">0)</span></code>; result is recomputed per <code class="docutils literal notranslate"><span class="pre">IntegralDivide.resultDecima [...]
+</ul>
+</section>
+<section id="e">
+<h2>e<a class="headerlink" href="#e" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Foldable; rewritten to a literal by ConstantFolding (like <code class="docutils literal notranslate"><span class="pre">pi</span></code>).</p></li>
+</ul>
+</section>
+<section id="exp">
+<h2>exp<a class="headerlink" href="#exp" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">UnaryMathExpression(StrictMath.exp,</span> <span class="pre">"EXP")</span></code> unchanged. ULP-level differences vs DataFusion <code class="docutils literal notranslate"><span class="pre">exp</span></code> are possible but unflagged.</p></li>
+</ul>
+</section>
+<section id="expm1">
+<h2>expm1<a class="headerlink" href="#expm1" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">UnaryMathExpression(StrictMath.expm1,</span> <span class="pre">"EXPM1")</span></code> unchanged.</p></li>
+</ul>
+</section>
+<section id="factorial">
+<h2>factorial<a class="headerlink" href="#factorial" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>3.4.3 (audited 2026-05-15): identical to v3.5.8.</p></li>
+<li><p>3.5.8 (audited 2026-05-15): canonical reference; <code class="docutils literal notranslate"><span class="pre">extends</span> <span class="pre">UnaryExpression</span> <span class="pre">with</span> <span class="pre">ImplicitCastInputTypes</span> <span class="pre">with</span> <span class="pre">NullIntolerant</span></code>. Returns NULL for NULL input or values outside <code class="docutils literal notranslate"><span class="pre">[0,</span> <span class="pre">20]</span></code>.</p></li>
+<li><p>4.0.1 (audited 2026-05-15): <code class="docutils literal notranslate"><span class="pre">NullIntolerant</span></code> trait replaced by <code class="docutils literal notranslate"><span class="pre">nullIntolerant:</span> <span class="pre">Boolean</span></code> method override; behavior unchanged.</p></li>
+<li><p>4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</section>
+<section id="floor">
+<h2>floor<a class="headerlink" href="#floor" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): mirror of <code class="docutils literal notranslate"><span class="pre">ceil</span></code>. Two-arg <code class="docutils literal notranslate"><span class="pre">floor(expr,</span> <span class="pre">scale)</span></code> form (<code class="docutils literal notranslate"><span class="pre">RoundFloor</span></code>) falls back to Spark.</p></li>
+</ul>
+</section>
+<section id="greatest">
+<h2>greatest<a class="headerlink" href="#greatest" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): NULL-skipping variadic. Wired as <code class="docutils literal notranslate"><span class="pre">CometScalarFunction("greatest")</span></code> to DataFusion’s <code class="docutils literal notranslate"><span class="pre">GreatestFunc</span></code>. Comet does not gate input types, so interval inputs and other Spark-only orderings rely on the native UDF accepting them; no explicit fallback path.</p></li>
+</ul>
+</section>
+<section id="hex">
+<h2>hex<a class="headerlink" href="#hex" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): accepts <code class="docutils literal notranslate"><span class="pre">LongType</span></code> / <code class="docutils literal notranslate"><span class="pre">BinaryType</span></code> / <code class="docutils literal notranslate"><span class="pre">StringType</span></code>. Spark 4.x widens <code class="docutils literal notranslate"><span class="pre">StringType</span></code> to <code class="docutils literal notranslate"><span class= [...]
+</ul>
+</section>
+<section id="least">
+<h2>least<a class="headerlink" href="#least" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): mirror of <code class="docutils literal notranslate"><span class="pre">greatest</span></code>; same caveats. Spark 4.1.1 adds <code class="docutils literal notranslate"><span class="pre">contextIndependentFoldable</span></code> (no Comet impact).</p></li>
+</ul>
+</section>
+<section id="ln">
+<h2>ln<a class="headerlink" href="#ln" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): registry alias for <code class="docutils literal notranslate"><span class="pre">Log</span></code>. Comet wires through <code class="docutils literal notranslate"><span class="pre">CometLog</span></code> to DataFusion <code class="docutils literal notranslate"><span class="pre">ln</span></code> with a <code class="docutils literal notranslate"><span class="pre">nullIfNegative</span></code> rewrite to match Spark’s NULL behaviou [...]
+</ul>
+</section>
+<section id="log">
+<h2>log<a class="headerlink" href="#log" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): one-arg <code class="docutils literal notranslate"><span class="pre">log(x)</span></code> -> <code class="docutils literal notranslate"><span class="pre">CometLog</span></code> (DataFusion <code class="docutils literal notranslate"><span class="pre">ln</span></code>); two-arg <code class="docutils literal notranslate"><span class="pre">log(base,</span> <span class="pre">x)</span></code> -> <code class="docutils literal n [...]
+</ul>
+</section>
+<section id="log10">
+<h2>log10<a class="headerlink" href="#log10" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">UnaryLogExpression(StrictMath.log10,</span> <span class="pre">"LOG10")</span></code>; returns NULL for <code class="docutils literal notranslate"><span class="pre">x</span> <span class="pre"><=</span> <span class="pre">0</span></code>. Possible ULP differences from <code class="docutils literal notranslate"><span class="pre">StrictMath</span></code>.</p></li>
+</ul>
+</section>
+<section id="log2">
+<h2>log2<a class="headerlink" href="#log2" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">UnaryLogExpression(StrictMath.log(x)</span> <span class="pre">/</span> <span class="pre">StrictMath.log(2),</span> <span class="pre">"LOG2")</span></code>; returns NULL for <code class="docutils literal notranslate"><span class="pre">x</span> <span class="pre"><=</span> <span class="pre">0</span></code>.</p></li>
+</ul>
+</section>
+<section id="mod">
+<h2>mod<a class="headerlink" href="#mod" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): registry alias for <code class="docutils literal notranslate"><span class="pre">Remainder</span></code>. Same support as <code class="docutils literal notranslate"><span class="pre">%</span></code>.</p></li>
+</ul>
+</section>
+<section id="negative">
+<h2>negative<a class="headerlink" href="#negative" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">UnaryMinus(child,</span> <span class="pre">failOnError)</span></code> -> Rust <code class="docutils literal notranslate"><span class="pre">NegativeExpr</span></code>. ANSI overflow is detected for <code class="docutils literal notranslate"><span class="pre">Int8/Int16/Int32/Int64</span></code> and <code class="docutils literal notranslate"><span class="pre">Interv [...]
+</ul>
+</section>
+<section id="pi">
+<h2>pi<a class="headerlink" href="#pi" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">LeafMathExpression(math.Pi,</span> <span class="pre">"PI")</span></code>; foldable, so Spark <code class="docutils literal notranslate"><span class="pre">ConstantFolding</span></code> rewrites it to a <code class="docutils literal notranslate"><span class="pre">Literal</span></code> before Comet sees the plan. The <code class="docutils literal notranslate"> [...]
+</ul>
+</section>
+<section id="positive">
+<h2>positive<a class="headerlink" href="#positive" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">UnaryPositive(child)</span></code> is a regular expression. There is no Comet serde for <code class="docutils literal notranslate"><span class="pre">UnaryPositive</span></code>, so projections containing <code class="docutils literal notranslate"><span class="pre">+col</span></code> silently disable Comet for the projection on 3.4/3.5.</p></li>
+<li><p>Spark 4.0.1, 4.1.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">UnaryPositive</span></code> is <code class="docutils literal notranslate"><span class="pre">RuntimeReplaceable</span></code> with <code class="docutils literal notranslate"><span class="pre">replacement</span> <span class="pre">=</span> <span class="pre">child</span></code>; the optimizer removes it before Comet sees the plan, so the gap is transparent on 4.x.</p></li>
+</ul>
+</section>
+<section id="pow">
+<h2>pow<a class="headerlink" href="#pow" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">Pow(left,</span> <span class="pre">right)</span> <span class="pre">extends</span> <span class="pre">BinaryMathExpression(StrictMath.pow,</span> <span class="pre">"POWER")</span></code>; routes to DataFusion <code class="docutils literal notranslate"><span class="pre">pow</span></code>. ULP-level differences possible.</p></li>
+</ul>
+</section>
+<section id="power">
+<h2>power<a class="headerlink" href="#power" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): registry alias for <code class="docutils literal notranslate"><span class="pre">Pow</span></code>. Same support as <code class="docutils literal notranslate"><span class="pre">pow</span></code>.</p></li>
+</ul>
+</section>
+<section id="radians">
+<h2>radians<a class="headerlink" href="#radians" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">UnaryMathExpression(math.toRadians,</span> <span class="pre">"RADIANS")</span></code> unchanged across versions.</p></li>
+</ul>
+</section>
+<section id="rand">
+<h2>rand<a class="headerlink" href="#rand" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>See <code class="docutils literal notranslate"><span class="pre">misc_funcs</span> <span class="pre">/</span> <span class="pre">rand</span></code>.</p></li>
+</ul>
+</section>
+<section id="randn">
+<h2>randn<a class="headerlink" href="#randn" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>See <code class="docutils literal notranslate"><span class="pre">misc_funcs</span> <span class="pre">/</span> <span class="pre">randn</span></code>.</p></li>
+</ul>
+</section>
+<section id="rint">
+<h2>rint<a class="headerlink" href="#rint" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">UnaryMathExpression(math.rint,</span> <span class="pre">"ROUND")</span></code> with <code class="docutils literal notranslate"><span class="pre">funcName</span> <span class="pre">=</span> <span class="pre">"rint"</span></code>. Passthrough to DataFusion <code class="docutils literal notranslate"><span class="pre">rint</span></code> (round-half-to- [...]
+</ul>
+</section>
+<section id="round">
+<h2>round<a class="headerlink" href="#round" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): HALF_UP rounding for integer / decimal. Float / Double child types always fall back to Spark because <code class="docutils literal notranslate"><span class="pre">BigDecimal</span></code>-via-<code class="docutils literal notranslate"><span class="pre">toString</span></code> rounding cannot be precisely matched (documented inline in <code class="docutils literal notranslate"><span class="pre">CometRound</span></code>). ANSI <co [...]
+</ul>
+</section>
+<section id="sec">
+<h2>sec<a class="headerlink" href="#sec" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): custom <code class="docutils literal notranslate"><span class="pre">1</span> <span class="pre">/</span> <span class="pre">math.cos(x)</span></code>. Routed to datafusion-spark’s <code class="docutils literal notranslate"><span class="pre">SparkSec</span></code>.</p></li>
+</ul>
+</section>
+<section id="shiftleft">
+<h2>shiftleft<a class="headerlink" href="#shiftleft" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>See <code class="docutils literal notranslate"><span class="pre">bitwise_funcs</span> <span class="pre">/</span> <span class="pre"><<</span></code> (audited in PR #4479). Same support as the operator alias added in 4.0.</p></li>
+</ul>
+</section>
+<section id="sign">
+<h2>sign<a class="headerlink" href="#sign" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): registry alias for <code class="docutils literal notranslate"><span class="pre">Signum</span></code>. Same support as <code class="docutils literal notranslate"><span class="pre">signum</span></code>.</p></li>
+</ul>
+</section>
+<section id="signum">
+<h2>signum<a class="headerlink" href="#signum" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">Signum(child)</span></code> over <code class="docutils literal notranslate"><span class="pre">DoubleType</span></code>. Spark also restricts to the two interval types via <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code>; Comet handles only the <code class="docutils literal notranslate"><span class="pre">Double</span></code> case v [...]
+</ul>
+</section>
+<section id="sin">
+<h2>sin<a class="headerlink" href="#sin" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">UnaryMathExpression(math.sin,</span> <span class="pre">"SIN")</span></code> unchanged.</p></li>
+</ul>
+</section>
+<section id="sinh">
+<h2>sinh<a class="headerlink" href="#sinh" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">UnaryMathExpression(math.sinh,</span> <span class="pre">"SINH")</span></code> unchanged.</p></li>
+</ul>
+</section>
+<section id="sqrt">
+<h2>sqrt<a class="headerlink" href="#sqrt" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">UnaryMathExpression(math.sqrt,</span> <span class="pre">"SQRT")</span></code> unchanged.</p></li>
+</ul>
+</section>
+<section id="tan">
+<h2>tan<a class="headerlink" href="#tan" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">UnaryMathExpression(math.tan,</span> <span class="pre">"TAN")</span></code> unchanged.</p></li>
+</ul>
+</section>
+<section id="tanh">
+<h2>tanh<a class="headerlink" href="#tanh" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">UnaryMathExpression(math.tanh,</span> <span class="pre">"TANH")</span></code> unchanged.</p></li>
+</ul>
+</section>
+<section id="try-add">
+<h2>try_add<a class="headerlink" href="#try-add" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">TryAdd</span></code> is <code class="docutils literal notranslate"><span class="pre">RuntimeReplaceable</span></code> and rewrites to <code class="docutils literal notranslate"><span class="pre">Add(..,</span> <span class="pre">EvalMode.TRY)</span></code> for numeric inputs (datetime / interval go through <code class="docutils literal notranslate"><span class="pre">T [...]
+</ul>
+</section>
+<section id="try-divide">
+<h2>try_divide<a class="headerlink" href="#try-divide" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">TryDivide</span></code> rewrites to <code class="docutils literal notranslate"><span class="pre">Divide(..,</span> <span class="pre">EvalMode.TRY)</span></code>. The <code class="docutils literal notranslate"><span class="pre">nullIfWhenPrimitive</span></code> wrapper swaps zero divisors to NULL; integer / float divide uses <code class="docutils literal notranslate"> [...]
+</ul>
+</section>
+<section id="try-multiply">
+<h2>try_multiply<a class="headerlink" href="#try-multiply" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): rewrites to <code class="docutils literal notranslate"><span class="pre">Multiply(..,</span> <span class="pre">EvalMode.TRY)</span></code>. Integer path uses <code class="docutils literal notranslate"><span class="pre">checked_mul</span></code>; decimal uses <code class="docutils literal notranslate"><span class="pre">WideDecimalBinaryExpr</span></code> with <code class="docutils literal notranslate"><span class="pre">EvalMode [...]
+</ul>
+</section>
+<section id="try-subtract">
+<h2>try_subtract<a class="headerlink" href="#try-subtract" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): rewrites to <code class="docutils literal notranslate"><span class="pre">Subtract(..,</span> <span class="pre">EvalMode.TRY)</span></code>. Integer path uses <code class="docutils literal notranslate"><span class="pre">checked_sub</span></code>; decimal uses <code class="docutils literal notranslate"><span class="pre">WideDecimalBinaryExpr</span></code> as needed.</p></li>
+</ul>
+</section>
+<section id="unhex">
+<h2>unhex<a class="headerlink" href="#unhex" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">Unhex(child,</span> <span class="pre">failOnError)</span></code>. Spark 4.x widens input to <code class="docutils literal notranslate"><span class="pre">StringTypeWithCollation</span></code> and wraps the inner call in try/catch; Comet <code class="docutils literal notranslate"><span class="pre">CometUnhex</span></code> forwards <code class="docutils literal notransl [...]
+</ul>
+</section>
+<section id="width-bucket">
+<h2>width_bucket<a class="headerlink" href="#width-bucket" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.5.8 (audited 2026-05-27): introduced; not available in 3.4.3.</p></li>
+<li><p>Spark 4.0.1, 4.1.1 (audited 2026-05-27): same semantics; <code class="docutils literal notranslate"><span class="pre">NullIntolerant</span></code> -> <code class="docutils literal notranslate"><span class="pre">nullIntolerant:</span> <span class="pre">Boolean</span></code> refactor.</p></li>
+<li><p>Known limitation: wired via per-version <code class="docutils literal notranslate"><span class="pre">CometExprShim</span></code> rather than a <code class="docutils literal notranslate"><span class="pre">CometExpressionSerde</span></code>, so it bypasses the support-level framework and the auto-generated compatibility doc (https://github.com/apache/datafusion-comet/issues/4485). Native path uses datafusion-spark <code class="docutils literal notranslate"><span class="pre">SparkWid [...]
+</ul>
+</section>
+</section>
+
+
+ </article>
+
+
+
+
+
+ <footer class="prev-next-footer d-print-none">
+
+<div class="prev-next-area">
+ <a class="left-prev"
+ href="map_funcs.html"
+ title="previous page">
+ <i class="fa-solid fa-angle-left"></i>
+ <div class="prev-next-info">
+ <p class="prev-next-subtitle">previous</p>
+ <p class="prev-next-title">map_funcs Expression Audits</p>
+ </div>
+ </a>
+ <a class="right-next"
+ href="misc_funcs.html"
+ title="next page">
+ <div class="prev-next-info">
+ <p class="prev-next-subtitle">next</p>
+ <p class="prev-next-title">misc_funcs Expression Audits</p>
+ </div>
+ <i class="fa-solid fa-angle-right"></i>
+ </a>
+</div>
+ </footer>
+
+ </div>
+
+
+
+
+ </div>
+ <footer class="bd-footer-content">
+
+ </footer>
+
+ </main>
+ </div>
+ </div>
+
+ <!-- Scripts loaded after <body> so the DOM is not blocked -->
+ <script defer src="../../_static/scripts/bootstrap.js?digest=8878045cc6db502f8baf"></script>
+<script defer src="../../_static/scripts/pydata-sphinx-theme.js?digest=8878045cc6db502f8baf"></script>
+
+<!-- Based on pydata_sphinx_theme/footer.html -->
+<footer class="footer mt-5 mt-md-0">
+ <div class="container">
+
+ <div class="footer-item">
+ <p>Apache DataFusion, Apache DataFusion Comet, Apache, the Apache feather logo, and the Apache DataFusion project logo</p>
+ <p>are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.</p>
+ </div>
+ </div>
+</footer>
+
+
+ </body>
+</html>
\ No newline at end of file
diff --git a/contributor-guide/benchmark-results/tpc-h.html b/contributor-guide/expression-audits/misc_funcs.html
similarity index 59%
copy from contributor-guide/benchmark-results/tpc-h.html
copy to contributor-guide/expression-audits/misc_funcs.html
index b75aa4ff2a..f77c73463e 100644
--- a/contributor-guide/benchmark-results/tpc-h.html
+++ b/contributor-guide/expression-audits/misc_funcs.html
@@ -27,7 +27,7 @@ under the License.
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
- <title>Apache DataFusion Comet: Benchmarks Derived From TPC-H — Apache DataFusion Comet documentation</title>
+ <title>misc_funcs Expression Audits — Apache DataFusion Comet documentation</title>
@@ -61,12 +61,12 @@ under the License.
<script src="../../_static/documentation_options.js?v=5929fcd5"></script>
<script src="../../_static/doctools.js?v=9a2dae69"></script>
<script src="../../_static/sphinx_highlight.js?v=dc90522c"></script>
- <script>DOCUMENTATION_OPTIONS.pagename = 'contributor-guide/benchmark-results/tpc-h';</script>
+ <script>DOCUMENTATION_OPTIONS.pagename = 'contributor-guide/expression-audits/misc_funcs';</script>
<script async="true" defer="true" src="https://buttons.github.io/buttons.js"></script>
<link rel="index" title="Index" href="../../genindex.html" />
<link rel="search" title="Search" href="../../search.html" />
- <link rel="next" title="Apache DataFusion Comet: Benchmarks Derived From TPC-DS" href="tpc-ds.html" />
- <link rel="prev" title="Comet Benchmarking Guide" href="../benchmarking.html" />
+ <link rel="next" title="predicate_funcs Expression Audits" href="predicate_funcs.html" />
+ <link rel="prev" title="math_funcs Expression Audits" href="math_funcs.html" />
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<meta name="docsearch:language" content="en"/>
<meta name="docsearch:version" content="" />
@@ -418,11 +418,11 @@ under the License.
<li class="toctree-l1"><a class="reference internal" href="../iceberg-spark-tests.html">Iceberg Spark Tests</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Debugging and Performance</span></p>
-<ul class="current nav bd-sidenav">
+<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../debugging.html">Debugging Guide</a></li>
-<li class="toctree-l1 current active has-children"><a class="reference internal" href="../benchmarking.html">Benchmarking Guide</a><details open="open"><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul class="current">
-<li class="toctree-l2 current active"><a class="current reference internal" href="#">Apache DataFusion Comet: Benchmarks Derived From TPC-H</a></li>
-<li class="toctree-l2"><a class="reference internal" href="tpc-ds.html">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</a></li>
+<li class="toctree-l1 has-children"><a class="reference internal" href="../benchmarking.html">Benchmarking Guide</a><details><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul>
+<li class="toctree-l2"><a class="reference internal" href="../benchmark-results/tpc-h.html">Apache DataFusion Comet: Benchmarks Derived From TPC-H</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../benchmark-results/tpc-ds.html">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_macos.html">Comet Benchmarking on macOS</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_aws_ec2.html">Comet Benchmarking in EC2</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_spark_sql_perf.html">TPC-DS Benchmarking with spark-sql-perf</a></li>
@@ -432,8 +432,27 @@ under the License.
<li class="toctree-l1"><a class="reference internal" href="../tracing.html">Tracing</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Reference</span></p>
-<ul class="nav bd-sidenav">
-<li class="toctree-l1"><a class="reference internal" href="../spark_expressions_support.html">Supported Spark Expressions</a></li>
+<ul class="current nav bd-sidenav">
+<li class="toctree-l1 current active has-children"><a class="reference internal" href="index.html">Expression Audits</a><details open="open"><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul class="current">
+<li class="toctree-l2"><a class="reference internal" href="agg_funcs.html">agg_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="array_funcs.html">array_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="bitwise_funcs.html">bitwise_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="collection_funcs.html">collection_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="conditional_funcs.html">conditional_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="conversion_funcs.html">conversion_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="datetime_funcs.html">datetime_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="generator_funcs.html">generator_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="hash_funcs.html">hash_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="json_funcs.html">json_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="map_funcs.html">map_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="math_funcs.html">math_funcs</a></li>
+<li class="toctree-l2 current active"><a class="current reference internal" href="#">misc_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="predicate_funcs.html">predicate_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="struct_funcs.html">struct_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="url_funcs.html">url_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="window_funcs.html">window_funcs</a></li>
+</ul>
+</details></li>
<li class="toctree-l1"><a class="reference internal" href="../spark_configs_support.html">Supported Spark Configurations</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Project Mechanics</span></p>
@@ -488,9 +507,9 @@ under the License.
<li class="breadcrumb-item"><a href="../index.html" class="nav-link">Comet Contributor Guide</a></li>
- <li class="breadcrumb-item"><a href="../benchmarking.html" class="nav-link">Comet Benchmarking Guide</a></li>
+ <li class="breadcrumb-item"><a href="index.html" class="nav-link">Expression Audits</a></li>
- <li class="breadcrumb-item active" aria-current="page"><span class="ellipsis">Apache DataFusion Comet: Benchmarks Derived From TPC-H</span></li>
+ <li class="breadcrumb-item active" aria-current="page"><span class="ellipsis">misc_funcs Expression Audits</span></li>
</ul>
</nav>
</div>
@@ -507,70 +526,111 @@ under the License.
<div id="searchbox"></div>
<article class="bd-article">
- <!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
+ <!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
-->
-<section id="apache-datafusion-comet-benchmarks-derived-from-tpc-h">
-<h1>Apache DataFusion Comet: Benchmarks Derived From TPC-H<a class="headerlink" href="#apache-datafusion-comet-benchmarks-derived-from-tpc-h" title="Link to this heading">#</a></h1>
-<p>The following benchmarks were performed on an EKS cluster (<code class="docutils literal notranslate"><span class="pre">r6i.24xlarge</span></code> instances with EBS storage) with data stored in S3.</p>
-<section id="benchmark-results">
-<h2>Benchmark Results<a class="headerlink" href="#benchmark-results" title="Link to this heading">#</a></h2>
-<p>Total time to run all queries (lower is better).</p>
-<p><img alt="" src="../../_images/tpch_allqueries_with_tuned.png" /></p>
-<p>The following charts are based on the tuned run using hash join.</p>
-<p>Per-query breakdown showing the relative performance of Spark and Comet.</p>
-<p><img alt="" src="../../_images/tpch_queries_compare.png" /></p>
-<p>How much Comet accelerates each query in relative terms.</p>
-<p><img alt="" src="../../_images/tpch_queries_speedup_rel.png" /></p>
-<p>How much Comet accelerates each query in absolute terms.</p>
-<p><img alt="" src="../../_images/tpch_queries_speedup_abs.png" /></p>
+<section id="misc-funcs-expression-audits">
+<h1>misc_funcs Expression Audits<a class="headerlink" href="#misc-funcs-expression-audits" title="Link to this heading">#</a></h1>
+<blockquote>
+<div><p>Audit notes for expressions in this category that have been audited. Absence of an entry means the expression has not been audited yet, not that it is unsupported. See the user guide <a class="reference internal" href="../../user-guide/latest/expressions.html"><span class="std std-doc">Spark Expression Support</span></a> for current support status.</p>
+</div></blockquote>
+<section id="current-catalog">
+<h2>current_catalog<a class="headerlink" href="#current-catalog" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Resolved to a literal by the analyzer (<code class="docutils literal notranslate"><span class="pre">ReplaceCurrentLike</span></code>).</p></li>
+</ul>
</section>
-<section id="configuration">
-<h2>Configuration<a class="headerlink" href="#configuration" title="Link to this heading">#</a></h2>
-<p>Common:</p>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.executor.instances</span><span class="o">=</span><span class="s">32</span>
-<span class="na">spark.executor.cores</span><span class="o">=</span><span class="s">16</span>
-<span class="na">spark.memory.fraction</span><span class="o">=</span><span class="s">0.6</span>
-<span class="na">spark.memory.storageFraction</span><span class="o">=</span><span class="s">0.2</span>
-<span class="c1"># Kubernetes CPU constraints</span>
-<span class="na">spark.kubernetes.executor.request.cores</span><span class="o">=</span><span class="s">8</span>
-<span class="na">spark.kubernetes.executor.limit.cores</span><span class="o">=</span><span class="s">8</span>
-</pre></div>
-</div>
-<p>Spark:</p>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.executor.memory</span><span class="o">=</span><span class="s">64G</span>
-<span class="na">spark.executor.memoryOverhead</span><span class="o">=</span><span class="s">10G</span>
-</pre></div>
-</div>
-<p>Comet:</p>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.executor.memory</span><span class="o">=</span><span class="s">32G</span>
-<span class="na">spark.executor.memoryOverhead</span><span class="o">=</span><span class="s">10G</span>
-<span class="na">spark.memory.offHeap.enabled</span><span class="o">=</span><span class="s">true</span>
-<span class="na">spark.memory.offHeap.size</span><span class="o">=</span><span class="s">32G</span>
-</pre></div>
-</div>
-<section id="comet-tuned">
-<h3>Comet (Tuned)<a class="headerlink" href="#comet-tuned" title="Link to this heading">#</a></h3>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.comet.exec.replaceSortMergeJoin</span><span class="o">=</span><span class="s">true</span>
-<span class="na">spark.comet.memoryPool.fraction</span><span class="o">=</span><span class="s">0.8</span>
-</pre></div>
-</div>
+<section id="current-database">
+<h2>current_database<a class="headerlink" href="#current-database" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Resolved to a literal by the analyzer (<code class="docutils literal notranslate"><span class="pre">ReplaceCurrentLike</span></code>).</p></li>
+</ul>
+</section>
+<section id="current-schema">
+<h2>current_schema<a class="headerlink" href="#current-schema" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Alias of <code class="docutils literal notranslate"><span class="pre">current_database</span></code>; resolved to a literal by the analyzer.</p></li>
+</ul>
+</section>
+<section id="current-user">
+<h2>current_user<a class="headerlink" href="#current-user" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Resolved to a literal by the analyzer; same as <code class="docutils literal notranslate"><span class="pre">user</span></code>.</p></li>
+</ul>
+</section>
+<section id="monotonically-increasing-id">
+<h2>monotonically_increasing_id<a class="headerlink" href="#monotonically-increasing-id" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): byte-for-byte identical to 4.1.1. <code class="docutils literal notranslate"><span class="pre">MonotonicallyIncreasingID()</span> <span class="pre">extends</span> <span class="pre">LeafExpression</span> <span class="pre">with</span> <span class="pre">Stateful</span></code>; produces a Long that encodes the partition id in the upper 31 bits and a per-partition row counter in the lower 33 bits. Comet emits an empty <code class="docutils literal notr [...]
+<li><p>Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): identical to 3.4.3.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 3.4.3.</p></li>
+</ul>
</section>
+<section id="rand">
+<h2>rand<a class="headerlink" href="#rand" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">Rand(child,</span> <span class="pre">hideSeed)</span> <span class="pre">extends</span> <span class="pre">RDG</span></code> (an <code class="docutils literal notranslate"><span class="pre">UnaryExpression</span> <span class="pre">with</span> <span class="pre">ExpectsInputTypes</span> <span class="pre">with</span> <span class="pre">Nondeterministic</span> <span class="pre">with</ [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">RDG</span></code> is refactored from an <code class="docutils literal notranslate"><span class="pre">abstract</span> <span class="pre">class</span></code> into a trait, and <code class="docutils literal notranslate"><span class="pre">Rand</span></code> now extends a new <code class="docutils literal notranslate"><span class="pre">NondeterministicUnaryRDG</span></code> base. <code class=" [...]
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+<li><p>Comet limitation: the seed argument must be a literal (column-reference seeds are rejected via <code class="docutils literal notranslate"><span class="pre">getSupportLevel</span></code>). Pre-4.0 Spark would otherwise silently fail at runtime; 4.0+ rejects at analysis time before the expression reaches Comet.</p></li>
+</ul>
+</section>
+<section id="randn">
+<h2>randn<a class="headerlink" href="#randn" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): same base as <code class="docutils literal notranslate"><span class="pre">Rand</span></code>; differs only in the eval body (<code class="docutils literal notranslate"><span class="pre">nextGaussian()</span></code> instead of <code class="docutils literal notranslate"><span class="pre">nextDouble()</span></code>), producing values from the standard normal distribution.</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): same refactor as <code class="docutils literal notranslate"><span class="pre">Rand</span></code>; runtime unchanged.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+<li><p>Comet limitation: same as <code class="docutils literal notranslate"><span class="pre">rand</span></code> — the seed argument must be a literal.</p></li>
+</ul>
+</section>
+<section id="session-user">
+<h2>session_user<a class="headerlink" href="#session-user" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Alias of <code class="docutils literal notranslate"><span class="pre">current_user</span></code>; resolved to a literal by the analyzer.</p></li>
+</ul>
+</section>
+<section id="spark-partition-id">
+<h2>spark_partition_id<a class="headerlink" href="#spark-partition-id" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): byte-for-byte identical to 4.1.1. <code class="docutils literal notranslate"><span class="pre">SparkPartitionID()</span> <span class="pre">extends</span> <span class="pre">LeafExpression</span> <span class="pre">with</span> <span class="pre">Nondeterministic</span></code>; returns the integer index of the partition being processed. Comet emits an empty <code class="docutils literal notranslate"><span class="pre">SparkPartitionId</span></code> prot [...]
+<li><p>Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): identical to 3.4.3.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 3.4.3.</p></li>
+</ul>
+</section>
+<section id="typeof">
+<h2>typeof<a class="headerlink" href="#typeof" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Foldable; resolved to a literal before Comet sees the plan.</p></li>
+</ul>
+</section>
+<section id="user">
+<h2>user<a class="headerlink" href="#user" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">CurrentUser()</span> <span class="pre">extends</span> <span class="pre">LeafExpression</span> <span class="pre">with</span> <span class="pre">Unevaluable</span></code>; the analyzer’s <code class="docutils literal notranslate"><span class="pre">ResolveCurrentLike</span></code> rule replaces it with a <code class="docutils literal notranslate"><span class="pre">StringType</span></code> li [...]
+<li><p>Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): identical to 3.4.3 except the resulting literal carries the default string collation.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
</section>
</section>
@@ -585,20 +645,20 @@ under the License.
<div class="prev-next-area">
<a class="left-prev"
- href="../benchmarking.html"
+ href="math_funcs.html"
title="previous page">
<i class="fa-solid fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">Comet Benchmarking Guide</p>
+ <p class="prev-next-title">math_funcs Expression Audits</p>
</div>
</a>
<a class="right-next"
- href="tpc-ds.html"
+ href="predicate_funcs.html"
title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
- <p class="prev-next-title">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</p>
+ <p class="prev-next-title">predicate_funcs Expression Audits</p>
</div>
<i class="fa-solid fa-angle-right"></i>
</a>
diff --git a/contributor-guide/expression-audits/predicate_funcs.html b/contributor-guide/expression-audits/predicate_funcs.html
new file mode 100644
index 0000000000..b3397944ee
--- /dev/null
+++ b/contributor-guide/expression-audits/predicate_funcs.html
@@ -0,0 +1,781 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+
+<!DOCTYPE html>
+
+
+<html lang="en" data-content_root="../../" data-theme="light">
+
+ <head>
+ <meta charset="utf-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
+
+ <title>predicate_funcs Expression Audits — Apache DataFusion Comet documentation</title>
+
+
+
+ <script data-cfasync="false">
+ document.documentElement.dataset.mode = localStorage.getItem("mode") || "light";
+ document.documentElement.dataset.theme = localStorage.getItem("theme") || "light";
+ </script>
+ <!--
+ this give us a css class that will be invisible only if js is disabled
+ -->
+ <noscript>
+ <style>
+ .pst-js-only { display: none !important; }
+
+ </style>
+ </noscript>
+
+ <!-- Loaded before other Sphinx assets -->
+ <link href="../../_static/styles/theme.css?digest=8878045cc6db502f8baf" rel="stylesheet" />
+<link href="../../_static/styles/pydata-sphinx-theme.css?digest=8878045cc6db502f8baf" rel="stylesheet" />
+
+ <link rel="stylesheet" type="text/css" href="../../_static/pygments.css?v=8f2a1f02" />
+ <link rel="stylesheet" type="text/css" href="../../_static/theme_overrides.css?v=cd442bcd" />
+
+ <!-- So that users can add custom icons -->
+ <script src="../../_static/scripts/fontawesome.js?digest=8878045cc6db502f8baf"></script>
+ <!-- Pre-loaded scripts that we'll load fully later -->
+ <link rel="preload" as="script" href="../../_static/scripts/bootstrap.js?digest=8878045cc6db502f8baf" />
+<link rel="preload" as="script" href="../../_static/scripts/pydata-sphinx-theme.js?digest=8878045cc6db502f8baf" />
+
+ <script src="../../_static/documentation_options.js?v=5929fcd5"></script>
+ <script src="../../_static/doctools.js?v=9a2dae69"></script>
+ <script src="../../_static/sphinx_highlight.js?v=dc90522c"></script>
+ <script>DOCUMENTATION_OPTIONS.pagename = 'contributor-guide/expression-audits/predicate_funcs';</script>
+ <script async="true" defer="true" src="https://buttons.github.io/buttons.js"></script>
+ <link rel="index" title="Index" href="../../genindex.html" />
+ <link rel="search" title="Search" href="../../search.html" />
+ <link rel="next" title="struct_funcs Expression Audits" href="struct_funcs.html" />
+ <link rel="prev" title="misc_funcs Expression Audits" href="misc_funcs.html" />
+ <meta name="viewport" content="width=device-width, initial-scale=1"/>
+ <meta name="docsearch:language" content="en"/>
+ <meta name="docsearch:version" content="" />
+ </head>
+
+
+ <body data-bs-spy="scroll" data-bs-target=".bd-toc-nav" data-offset="180" data-bs-root-margin="0px 0px -60%" data-default-mode="light">
+
+
+
+ <div id="pst-skip-link" class="skip-link d-print-none"><a href="#main-content">Skip to main content</a></div>
+
+ <div id="pst-scroll-pixel-helper"></div>
+
+ <button type="button" class="btn rounded-pill" id="pst-back-to-top">
+ <i class="fa-solid fa-arrow-up"></i>Back to top</button>
+
+
+ <dialog id="pst-search-dialog">
+
+<form class="bd-search d-flex align-items-center"
+ action="../../search.html"
+ method="get">
+ <i class="fa-solid fa-magnifying-glass"></i>
+ <input type="search"
+ class="form-control"
+ name="q"
+ placeholder="Search the docs ..."
+ aria-label="Search the docs ..."
+ autocomplete="off"
+ autocorrect="off"
+ autocapitalize="off"
+ spellcheck="false"/>
+ <span class="search-button__kbd-shortcut"><kbd class="kbd-shortcut__modifier">Ctrl</kbd>+<kbd>K</kbd></span>
+</form>
+ </dialog>
+
+ <div class="pst-async-banner-revealer d-none">
+ <aside id="bd-header-version-warning" class="d-none d-print-none" aria-label="Version warning"></aside>
+</div>
+
+
+ <header class="bd-header navbar navbar-expand-lg bd-navbar d-print-none">
+<div class="bd-header__inner bd-page-width">
+ <button class="pst-navbar-icon sidebar-toggle primary-toggle" aria-label="Site navigation">
+ <span class="fa-solid fa-bars"></span>
+ </button>
+
+
+ <div class="col-lg-3 navbar-header-items__start">
+
+ <div class="navbar-item">
+
+
+
+
+
+<a class="navbar-brand logo" href="../../index.html">
+
+
+
+
+
+
+
+
+ <img src="../../_static/DataFusionComet-Logo-Light.png" class="logo__image only-light" alt="Apache DataFusion Comet documentation - Home"/>
+ <img src="../../_static/DataFusionComet-Logo-Dark.png" class="logo__image only-dark pst-js-only" alt="Apache DataFusion Comet documentation - Home"/>
+
+
+</a></div>
+
+ </div>
+
+ <div class="col-lg-9 navbar-header-items">
+
+ <div class="me-auto navbar-header-items__center">
+
+ <div class="navbar-item">
+<nav>
+ <ul class="bd-navbar-elements navbar-nav">
+
+<li class="nav-item ">
+ <a class="nav-link nav-internal" href="../../user-guide/index.html">
+ User Guide
+ </a>
+</li>
+
+
+<li class="nav-item current active">
+ <a class="nav-link nav-internal" href="../index.html">
+ Contributor Guide
+ </a>
+</li>
+
+
+<li class="nav-item ">
+ <a class="nav-link nav-internal" href="../../changelog/index.html">
+ Changelog
+ </a>
+</li>
+
+
+<li class="nav-item ">
+ <a class="nav-link nav-internal" href="../../about/gluten_comparison.html">
+ Comparison with Gluten
+ </a>
+</li>
+
+
+<li class="nav-item ">
+ <a class="nav-link nav-internal" href="../../about/versioning_policy.html">
+ Versioning Policy
+ </a>
+</li>
+
+ <li class="nav-item dropdown">
+ <button class="btn dropdown-toggle nav-item" type="button"
+ data-bs-toggle="dropdown" aria-expanded="false"
+ aria-controls="pst-nav-more-links">
+ More
+ </button>
+ <ul id="pst-nav-more-links" class="dropdown-menu">
+
+<li class=" ">
+ <a class="nav-link dropdown-item nav-internal" href="../../asf/index.html">
+ ASF Links
+ </a>
+</li>
+
+ </ul>
+ </li>
+
+ </ul>
+</nav></div>
+
+ </div>
+
+
+ <div class="navbar-header-items__end">
+
+ <div class="navbar-item navbar-persistent--container">
+
+
+<button class="btn search-button-field search-button__button pst-js-only" title="Search" aria-label="Search" data-bs-placement="bottom" data-bs-toggle="tooltip">
+ <i class="fa-solid fa-magnifying-glass"></i>
+ <span class="search-button__default-text">Search</span>
+ <span class="search-button__kbd-shortcut"><kbd class="kbd-shortcut__modifier">Ctrl</kbd>+<kbd class="kbd-shortcut__modifier">K</kbd></span>
+</button>
+ </div>
+
+
+ <div class="navbar-item"><ul class="navbar-icon-links"
+ aria-label="Icon Links">
+ <li class="nav-item">
+
+
+
+
+
+
+
+
+ <a href="https://github.com/apache/datafusion-comet" title="GitHub" class="nav-link pst-navbar-icon" rel="noopener" target="_blank" data-bs-toggle="tooltip" data-bs-placement="bottom"><i class="fa-brands fa-github fa-lg" aria-hidden="true"></i>
+ <span class="sr-only">GitHub</span></a>
+ </li>
+</ul></div>
+
+ <div class="navbar-item">
+
+<button class="btn btn-sm nav-link pst-navbar-icon theme-switch-button pst-js-only" aria-label="Color mode" data-bs-title="Color mode" data-bs-placement="bottom" data-bs-toggle="tooltip">
+ <i class="theme-switch fa-solid fa-sun fa-lg" data-mode="light" title="Light"></i>
+ <i class="theme-switch fa-solid fa-moon fa-lg" data-mode="dark" title="Dark"></i>
+ <i class="theme-switch fa-solid fa-circle-half-stroke fa-lg" data-mode="auto" title="System Settings"></i>
+</button></div>
+
+ </div>
+
+ </div>
+
+
+ <div class="navbar-persistent--mobile">
+
+<button class="btn search-button-field search-button__button pst-js-only" title="Search" aria-label="Search" data-bs-placement="bottom" data-bs-toggle="tooltip">
+ <i class="fa-solid fa-magnifying-glass"></i>
+ <span class="search-button__default-text">Search</span>
+ <span class="search-button__kbd-shortcut"><kbd class="kbd-shortcut__modifier">Ctrl</kbd>+<kbd class="kbd-shortcut__modifier">K</kbd></span>
+</button>
+ </div>
+
+
+
+</div>
+
+ </header>
+
+
+ <div class="bd-container">
+ <div class="bd-container__inner bd-page-width">
+
+
+
+ <dialog id="pst-primary-sidebar-modal"></dialog>
+ <div id="pst-primary-sidebar" class="bd-sidebar-primary bd-sidebar">
+
+
+
+ <div class="sidebar-header-items sidebar-primary__section">
+
+
+ <div class="sidebar-header-items__center">
+
+
+
+ <div class="navbar-item">
+<nav>
+ <ul class="bd-navbar-elements navbar-nav">
+
+<li class="nav-item ">
+ <a class="nav-link nav-internal" href="../../user-guide/index.html">
+ User Guide
+ </a>
+</li>
+
+
+<li class="nav-item current active">
+ <a class="nav-link nav-internal" href="../index.html">
+ Contributor Guide
+ </a>
+</li>
+
+
+<li class="nav-item ">
+ <a class="nav-link nav-internal" href="../../changelog/index.html">
+ Changelog
+ </a>
+</li>
+
+
+<li class="nav-item ">
+ <a class="nav-link nav-internal" href="../../about/gluten_comparison.html">
+ Comparison with Gluten
+ </a>
+</li>
+
+
+<li class="nav-item ">
+ <a class="nav-link nav-internal" href="../../about/versioning_policy.html">
+ Versioning Policy
+ </a>
+</li>
+
+
+<li class="nav-item ">
+ <a class="nav-link nav-internal" href="../../asf/index.html">
+ ASF Links
+ </a>
+</li>
+
+ </ul>
+</nav></div>
+
+
+ </div>
+
+
+
+ <div class="sidebar-header-items__end">
+
+ <div class="navbar-item"><ul class="navbar-icon-links"
+ aria-label="Icon Links">
+ <li class="nav-item">
+
+
+
+
+
+
+
+
+ <a href="https://github.com/apache/datafusion-comet" title="GitHub" class="nav-link pst-navbar-icon" rel="noopener" target="_blank" data-bs-toggle="tooltip" data-bs-placement="bottom"><i class="fa-brands fa-github fa-lg" aria-hidden="true"></i>
+ <span class="sr-only">GitHub</span></a>
+ </li>
+</ul></div>
+
+ <div class="navbar-item">
+
+<button class="btn btn-sm nav-link pst-navbar-icon theme-switch-button pst-js-only" aria-label="Color mode" data-bs-title="Color mode" data-bs-placement="bottom" data-bs-toggle="tooltip">
+ <i class="theme-switch fa-solid fa-sun fa-lg" data-mode="light" title="Light"></i>
+ <i class="theme-switch fa-solid fa-moon fa-lg" data-mode="dark" title="Dark"></i>
+ <i class="theme-switch fa-solid fa-circle-half-stroke fa-lg" data-mode="auto" title="System Settings"></i>
+</button></div>
+
+ </div>
+
+ </div>
+
+ <div class="sidebar-primary-items__start sidebar-primary__section">
+ <div class="sidebar-primary-item"><!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+
+
+
+
+<nav class="bd-links" id="bd-docs-nav" aria-label="Main navigation">
+ <div class="bd-toc-item active">
+ <p aria-level="2" class="caption" role="heading"><span class="caption-text">Getting Started</span></p>
+<ul class="nav bd-sidenav">
+<li class="toctree-l1"><a class="reference internal" href="../contributing.html">Getting Started</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../development.html">Development Guide</a></li>
+</ul>
+<p aria-level="2" class="caption" role="heading"><span class="caption-text">Project Architecture</span></p>
+<ul class="nav bd-sidenav">
+<li class="toctree-l1"><a class="reference internal" href="../plugin_overview.html">Comet Plugin Architecture</a></li>
+
+<li class="toctree-l1"><a class="reference internal" href="../ffi.html">Arrow FFI</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../jvm_shuffle.html">JVM Shuffle</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../native_shuffle.html">Native Shuffle</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../sql_error_propagation.html">ANSI Error Propagation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../s3-credential-provider-design.html">S3 Credential Provider Design</a></li>
+</ul>
+<p aria-level="2" class="caption" role="heading"><span class="caption-text">Adding Functionality</span></p>
+<ul class="nav bd-sidenav">
+<li class="toctree-l1"><a class="reference internal" href="../adding_a_new_operator.html">Adding a New Operator</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../adding_a_new_expression.html">Adding a New Expression</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../adding_a_new_spark_version.html">Adding a New Spark Version</a></li>
+</ul>
+<p aria-level="2" class="caption" role="heading"><span class="caption-text">Testing</span></p>
+<ul class="nav bd-sidenav">
+<li class="toctree-l1"><a class="reference internal" href="../sql-file-tests.html">Comet SQL Tests</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../spark-sql-tests.html">Spark SQL Tests</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../iceberg-spark-tests.html">Iceberg Spark Tests</a></li>
+</ul>
+<p aria-level="2" class="caption" role="heading"><span class="caption-text">Debugging and Performance</span></p>
+<ul class="nav bd-sidenav">
+<li class="toctree-l1"><a class="reference internal" href="../debugging.html">Debugging Guide</a></li>
+<li class="toctree-l1 has-children"><a class="reference internal" href="../benchmarking.html">Benchmarking Guide</a><details><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul>
+<li class="toctree-l2"><a class="reference internal" href="../benchmark-results/tpc-h.html">Apache DataFusion Comet: Benchmarks Derived From TPC-H</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../benchmark-results/tpc-ds.html">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../benchmarking_macos.html">Comet Benchmarking on macOS</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../benchmarking_aws_ec2.html">Comet Benchmarking in EC2</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../benchmarking_spark_sql_perf.html">TPC-DS Benchmarking with spark-sql-perf</a></li>
+</ul>
+</details></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tracing.html">Tracing</a></li>
+</ul>
+<p aria-level="2" class="caption" role="heading"><span class="caption-text">Reference</span></p>
+<ul class="current nav bd-sidenav">
+<li class="toctree-l1 current active has-children"><a class="reference internal" href="index.html">Expression Audits</a><details open="open"><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul class="current">
+<li class="toctree-l2"><a class="reference internal" href="agg_funcs.html">agg_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="array_funcs.html">array_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="bitwise_funcs.html">bitwise_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="collection_funcs.html">collection_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="conditional_funcs.html">conditional_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="conversion_funcs.html">conversion_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="datetime_funcs.html">datetime_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="generator_funcs.html">generator_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="hash_funcs.html">hash_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="json_funcs.html">json_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="map_funcs.html">map_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="math_funcs.html">math_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="misc_funcs.html">misc_funcs</a></li>
+<li class="toctree-l2 current active"><a class="current reference internal" href="#">predicate_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="struct_funcs.html">struct_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="url_funcs.html">url_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="window_funcs.html">window_funcs</a></li>
+</ul>
+</details></li>
+<li class="toctree-l1"><a class="reference internal" href="../spark_configs_support.html">Supported Spark Configurations</a></li>
+</ul>
+<p aria-level="2" class="caption" role="heading"><span class="caption-text">Project Mechanics</span></p>
+<ul class="nav bd-sidenav">
+<li class="toctree-l1"><a class="reference internal" href="../bug_triage.html">Bug Triage</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../release_process.html">Release Process</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../roadmap.html">Roadmap</a></li>
+<li class="toctree-l1"><a class="reference external" href="https://github.com/apache/datafusion-comet">Github and Issue Tracker</a></li>
+</ul>
+
+ </div>
+</nav>
+</div>
+ </div>
+
+
+ <div class="sidebar-primary-items__end sidebar-primary__section">
+ <div class="sidebar-primary-item">
+<div id="ethical-ad-placement"
+ class="flat"
+ data-ea-publisher="readthedocs"
+ data-ea-type="readthedocs-sidebar"
+ data-ea-manual="true">
+</div></div>
+ </div>
+
+
+ </div>
+
+ <main id="main-content" class="bd-main" role="main">
+
+
+ <div class="bd-content">
+ <div class="bd-article-container">
+
+ <div class="bd-header-article d-print-none">
+<div class="header-article-items header-article__inner">
+
+ <div class="header-article-items__start">
+
+ <div class="header-article-item">
+
+<nav aria-label="Breadcrumb" class="d-print-none">
+ <ul class="bd-breadcrumbs">
+
+ <li class="breadcrumb-item breadcrumb-home">
+ <a href="../../index.html" class="nav-link" aria-label="Home">
+ <i class="fa-solid fa-home"></i>
+ </a>
+ </li>
+
+ <li class="breadcrumb-item"><a href="../index.html" class="nav-link">Comet Contributor Guide</a></li>
+
+
+ <li class="breadcrumb-item"><a href="index.html" class="nav-link">Expression Audits</a></li>
+
+ <li class="breadcrumb-item active" aria-current="page"><span class="ellipsis">predicate_funcs Expression Audits</span></li>
+ </ul>
+</nav>
+</div>
+
+ </div>
+
+
+</div>
+</div>
+
+
+
+
+<div id="searchbox"></div>
+ <article class="bd-article">
+
+ <!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+<section id="predicate-funcs-expression-audits">
+<h1>predicate_funcs Expression Audits<a class="headerlink" href="#predicate-funcs-expression-audits" title="Link to this heading">#</a></h1>
+<blockquote>
+<div><p>Audit notes for expressions in this category that have been audited. Absence of an entry means the expression has not been audited yet, not that it is unsupported. See the user guide <a class="reference internal" href="../../user-guide/latest/expressions.html"><span class="std std-doc">Spark Expression Support</span></a> for current support status.</p>
+</div></blockquote>
+<section id="id1">
+<h2><code class="docutils literal notranslate"><span class="pre">!</span></code><a class="headerlink" href="#id1" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): registry alias of <code class="docutils literal notranslate"><span class="pre">Not</span></code>. Same support as <code class="docutils literal notranslate"><span class="pre">not</span></code>.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): identical to 3.4.3.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 3.4.3.</p></li>
+</ul>
+</section>
+<section id="id2">
+<h2><code class="docutils literal notranslate"><span class="pre"><</span></code><a class="headerlink" href="#id2" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">LessThan(left,</span> <span class="pre">right)</span> <span class="pre">extends</span> <span class="pre">BinaryComparison</span></code>. Comet routes via <code class="docutils literal notranslate"><span class="pre">CometLessThan</span></code> to the proto’s <code class="docutils literal notranslate"><span class="pre">lt</span></code> binary expression.</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): semantics unchanged.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</section>
+<section id="id3">
+<h2><code class="docutils literal notranslate"><span class="pre"><=</span></code><a class="headerlink" href="#id3" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">LessThanOrEqual(left,</span> <span class="pre">right)</span> <span class="pre">extends</span> <span class="pre">BinaryComparison</span></code>. Comet routes via <code class="docutils literal notranslate"><span class="pre">CometLessThanOrEqual</span></code> to the proto’s <code class="docutils literal notranslate"><span class="pre">lt_eq</span></code> binary expression.</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): semantics unchanged.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</section>
+<section id="id4">
+<h2><code class="docutils literal notranslate"><span class="pre"><=></span></code><a class="headerlink" href="#id4" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">EqualNullSafe(left,</span> <span class="pre">right)</span> <span class="pre">extends</span> <span class="pre">BinaryComparison</span></code>; treats two NULLs as equal and a NULL with a non-NULL as not equal. Comet routes via <code class="docutils literal notranslate"><span class="pre">CometEqualNullSafe</span></code> to the proto’s <code class="docutils literal notranslate"><s [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): semantics unchanged.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</section>
+<section id="id5">
+<h2><code class="docutils literal notranslate"><span class="pre">=</span></code><a class="headerlink" href="#id5" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">EqualTo(left,</span> <span class="pre">right)</span> <span class="pre">extends</span> <span class="pre">BinaryComparison</span></code>. Comet routes via <code class="docutils literal notranslate"><span class="pre">CometEqualTo</span></code> to the proto’s <code class="docutils literal notranslate"><span class="pre">eq</span></code> binary expression.</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): semantics unchanged.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</section>
+<section id="id6">
+<h2><code class="docutils literal notranslate"><span class="pre">==</span></code><a class="headerlink" href="#id6" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): registry alias of <code class="docutils literal notranslate"><span class="pre">EqualTo</span></code>. Same support as <code class="docutils literal notranslate"><span class="pre">=</span></code>.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): identical to 3.4.3.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 3.4.3.</p></li>
+</ul>
+</section>
+<section id="id7">
+<h2><code class="docutils literal notranslate"><span class="pre">></span></code><a class="headerlink" href="#id7" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">GreaterThan(left,</span> <span class="pre">right)</span> <span class="pre">extends</span> <span class="pre">BinaryComparison</span></code>. Comet routes via <code class="docutils literal notranslate"><span class="pre">CometGreaterThan</span></code> to the proto’s <code class="docutils literal notranslate"><span class="pre">gt</span></code> binary expression.</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): semantics unchanged.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</section>
+<section id="id8">
+<h2><code class="docutils literal notranslate"><span class="pre">>=</span></code><a class="headerlink" href="#id8" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">GreaterThanOrEqual(left,</span> <span class="pre">right)</span> <span class="pre">extends</span> <span class="pre">BinaryComparison</span></code>. Comet routes via <code class="docutils literal notranslate"><span class="pre">CometGreaterThanOrEqual</span></code> to the proto’s <code class="docutils literal notranslate"><span class="pre">gt_eq</span></code> binary expression.</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): semantics unchanged.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</section>
+<section id="and">
+<h2>and<a class="headerlink" href="#and" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">And(left,</span> <span class="pre">right)</span> <span class="pre">extends</span> <span class="pre">BinaryOperator</span> <span class="pre">with</span> <span class="pre">Predicate</span></code>; short-circuit left-to-right. Comet routes via <code class="docutils literal notranslate"><span class="pre">CometAnd</span></code> to the proto’s <code class="docutils literal notranslat [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): semantics unchanged.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</section>
+<section id="between">
+<h2>between<a class="headerlink" href="#between" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): the SQL form <code class="docutils literal notranslate"><span class="pre">expr</span> <span class="pre">BETWEEN</span> <span class="pre">low</span> <span class="pre">AND</span> <span class="pre">high</span></code> is rewritten at the parser level to <code class="docutils literal notranslate"><span class="pre">expr</span> <span class="pre">>=</span> <span class="pre">low</span> <span class="pre">AND</span> <span class="pre">expr</span> <span cla [...]
+<li><p>Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3.</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): identical to 3.4.3.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 3.4.3.</p></li>
+</ul>
+</section>
+<section id="ilike">
+<h2>ilike<a class="headerlink" href="#ilike" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">ILike(left,</span> <span class="pre">right,</span> <span class="pre">escapeChar)</span> <span class="pre">extends</span> <span class="pre">RuntimeReplaceable</span></code>; the analyzer rewrites to <code class="docutils literal notranslate"><span class="pre">Like(Lower(left),</span> <span class="pre">Lower(right),</span> <span class="pre">escapeChar)</span></code>. Comet handle [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</section>
+<section id="in">
+<h2>in<a class="headerlink" href="#in" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">In(value,</span> <span class="pre">list:</span> <span class="pre">Seq[Expression])</span> <span class="pre">extends</span> <span class="pre">Predicate</span></code>; NULL-sensitive set membership (NULL in any element of the list yields NULL when no match found). Comet routes via <code class="docutils literal notranslate"><span class="pre">CometIn</span></code> to the proto’s <c [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): semantics unchanged.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</section>
+<section id="isnan">
+<h2>isnan<a class="headerlink" href="#isnan" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">IsNaN(child)</span> <span class="pre">extends</span> <span class="pre">UnaryExpression</span></code>; returns <code class="docutils literal notranslate"><span class="pre">BooleanType</span></code>; <code class="docutils literal notranslate"><span class="pre">false</span></code> for NULL or non-float types. Comet routes via <code class="docutils literal notranslate"><span class= [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): semantics unchanged.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</section>
+<section id="isnotnull">
+<h2>isnotnull<a class="headerlink" href="#isnotnull" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">IsNotNull(child)</span> <span class="pre">extends</span> <span class="pre">UnaryExpression</span> <span class="pre">with</span> <span class="pre">Predicate</span></code>. Comet routes via <code class="docutils literal notranslate"><span class="pre">CometIsNotNull</span></code> to the proto’s <code class="docutils literal notranslate"><span class="pre">is_not_null</span></code> [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): semantics unchanged.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</section>
+<section id="isnull">
+<h2>isnull<a class="headerlink" href="#isnull" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">IsNull(child)</span> <span class="pre">extends</span> <span class="pre">UnaryExpression</span> <span class="pre">with</span> <span class="pre">Predicate</span></code>. Comet routes via <code class="docutils literal notranslate"><span class="pre">CometIsNull</span></code> to the proto’s <code class="docutils literal notranslate"><span class="pre">is_null</span></code> unary expr [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): semantics unchanged.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</section>
+<section id="like">
+<h2>like<a class="headerlink" href="#like" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>See <code class="docutils literal notranslate"><span class="pre">string_funcs</span> <span class="pre">/</span> <span class="pre">like</span></code> (audited in PR #4461). <code class="docutils literal notranslate"><span class="pre">CometLike</span></code> only supports the default <code class="docutils literal notranslate"><span class="pre">\\</span></code> escape character.</p></li>
+</ul>
+</section>
+<section id="not">
+<h2>not<a class="headerlink" href="#not" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">Not(child)</span> <span class="pre">extends</span> <span class="pre">UnaryExpression</span> <span class="pre">with</span> <span class="pre">Predicate</span></code>. Comet routes via <code class="docutils literal notranslate"><span class="pre">CometNot</span></code>, which optimizes a few special cases: <code class="docutils literal notranslate"><span class="pre">Not(EqualTo)</s [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): semantics unchanged.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</section>
+<section id="or">
+<h2>or<a class="headerlink" href="#or" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">Or(left,</span> <span class="pre">right)</span> <span class="pre">extends</span> <span class="pre">BinaryOperator</span> <span class="pre">with</span> <span class="pre">Predicate</span></code>; short-circuit left-to-right. Comet routes via <code class="docutils literal notranslate"><span class="pre">CometOr</span></code> to the proto’s <code class="docutils literal notranslate" [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): semantics unchanged.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+</ul>
+</section>
+<section id="rlike">
+<h2>rlike<a class="headerlink" href="#rlike" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>See <code class="docutils literal notranslate"><span class="pre">string_funcs</span> <span class="pre">/</span> <span class="pre">regexp_replace</span></code> and the <code class="docutils literal notranslate"><span class="pre">CometRLike</span></code> notes (audited in PR #4461). Uses the Rust <code class="docutils literal notranslate"><span class="pre">regex</span></code> crate, which differs from Java’s <code class="docutils literal notranslate"><span class="pre">Pattern</span> [...]
+</ul>
+</section>
+</section>
+
+
+ </article>
+
+
+
+
+
+ <footer class="prev-next-footer d-print-none">
+
+<div class="prev-next-area">
+ <a class="left-prev"
+ href="misc_funcs.html"
+ title="previous page">
+ <i class="fa-solid fa-angle-left"></i>
+ <div class="prev-next-info">
+ <p class="prev-next-subtitle">previous</p>
+ <p class="prev-next-title">misc_funcs Expression Audits</p>
+ </div>
+ </a>
+ <a class="right-next"
+ href="struct_funcs.html"
+ title="next page">
+ <div class="prev-next-info">
+ <p class="prev-next-subtitle">next</p>
+ <p class="prev-next-title">struct_funcs Expression Audits</p>
+ </div>
+ <i class="fa-solid fa-angle-right"></i>
+ </a>
+</div>
+ </footer>
+
+ </div>
+
+
+
+
+ </div>
+ <footer class="bd-footer-content">
+
+ </footer>
+
+ </main>
+ </div>
+ </div>
+
+ <!-- Scripts loaded after <body> so the DOM is not blocked -->
+ <script defer src="../../_static/scripts/bootstrap.js?digest=8878045cc6db502f8baf"></script>
+<script defer src="../../_static/scripts/pydata-sphinx-theme.js?digest=8878045cc6db502f8baf"></script>
+
+<!-- Based on pydata_sphinx_theme/footer.html -->
+<footer class="footer mt-5 mt-md-0">
+ <div class="container">
+
+ <div class="footer-item">
+ <p>Apache DataFusion, Apache DataFusion Comet, Apache, the Apache feather logo, and the Apache DataFusion project logo</p>
+ <p>are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.</p>
+ </div>
+ </div>
+</footer>
+
+
+ </body>
+</html>
\ No newline at end of file
diff --git a/contributor-guide/benchmark-results/tpc-ds.html b/contributor-guide/expression-audits/struct_funcs.html
similarity index 68%
copy from contributor-guide/benchmark-results/tpc-ds.html
copy to contributor-guide/expression-audits/struct_funcs.html
index 4d5bf9b1c4..3a30469b4c 100644
--- a/contributor-guide/benchmark-results/tpc-ds.html
+++ b/contributor-guide/expression-audits/struct_funcs.html
@@ -27,7 +27,7 @@ under the License.
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
- <title>Apache DataFusion Comet: Benchmarks Derived From TPC-DS — Apache DataFusion Comet documentation</title>
+ <title>struct_funcs Expression Audits — Apache DataFusion Comet documentation</title>
@@ -61,12 +61,12 @@ under the License.
<script src="../../_static/documentation_options.js?v=5929fcd5"></script>
<script src="../../_static/doctools.js?v=9a2dae69"></script>
<script src="../../_static/sphinx_highlight.js?v=dc90522c"></script>
- <script>DOCUMENTATION_OPTIONS.pagename = 'contributor-guide/benchmark-results/tpc-ds';</script>
+ <script>DOCUMENTATION_OPTIONS.pagename = 'contributor-guide/expression-audits/struct_funcs';</script>
<script async="true" defer="true" src="https://buttons.github.io/buttons.js"></script>
<link rel="index" title="Index" href="../../genindex.html" />
<link rel="search" title="Search" href="../../search.html" />
- <link rel="next" title="Comet Benchmarking on macOS" href="../benchmarking_macos.html" />
- <link rel="prev" title="Apache DataFusion Comet: Benchmarks Derived From TPC-H" href="tpc-h.html" />
+ <link rel="next" title="url_funcs Expression Audits" href="url_funcs.html" />
+ <link rel="prev" title="predicate_funcs Expression Audits" href="predicate_funcs.html" />
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<meta name="docsearch:language" content="en"/>
<meta name="docsearch:version" content="" />
@@ -418,11 +418,11 @@ under the License.
<li class="toctree-l1"><a class="reference internal" href="../iceberg-spark-tests.html">Iceberg Spark Tests</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Debugging and Performance</span></p>
-<ul class="current nav bd-sidenav">
+<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../debugging.html">Debugging Guide</a></li>
-<li class="toctree-l1 current active has-children"><a class="reference internal" href="../benchmarking.html">Benchmarking Guide</a><details open="open"><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul class="current">
-<li class="toctree-l2"><a class="reference internal" href="tpc-h.html">Apache DataFusion Comet: Benchmarks Derived From TPC-H</a></li>
-<li class="toctree-l2 current active"><a class="current reference internal" href="#">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</a></li>
+<li class="toctree-l1 has-children"><a class="reference internal" href="../benchmarking.html">Benchmarking Guide</a><details><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul>
+<li class="toctree-l2"><a class="reference internal" href="../benchmark-results/tpc-h.html">Apache DataFusion Comet: Benchmarks Derived From TPC-H</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../benchmark-results/tpc-ds.html">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_macos.html">Comet Benchmarking on macOS</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_aws_ec2.html">Comet Benchmarking in EC2</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_spark_sql_perf.html">TPC-DS Benchmarking with spark-sql-perf</a></li>
@@ -432,8 +432,27 @@ under the License.
<li class="toctree-l1"><a class="reference internal" href="../tracing.html">Tracing</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Reference</span></p>
-<ul class="nav bd-sidenav">
-<li class="toctree-l1"><a class="reference internal" href="../spark_expressions_support.html">Supported Spark Expressions</a></li>
+<ul class="current nav bd-sidenav">
+<li class="toctree-l1 current active has-children"><a class="reference internal" href="index.html">Expression Audits</a><details open="open"><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul class="current">
+<li class="toctree-l2"><a class="reference internal" href="agg_funcs.html">agg_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="array_funcs.html">array_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="bitwise_funcs.html">bitwise_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="collection_funcs.html">collection_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="conditional_funcs.html">conditional_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="conversion_funcs.html">conversion_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="datetime_funcs.html">datetime_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="generator_funcs.html">generator_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="hash_funcs.html">hash_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="json_funcs.html">json_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="map_funcs.html">map_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="math_funcs.html">math_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="misc_funcs.html">misc_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="predicate_funcs.html">predicate_funcs</a></li>
+<li class="toctree-l2 current active"><a class="current reference internal" href="#">struct_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="url_funcs.html">url_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="window_funcs.html">window_funcs</a></li>
+</ul>
+</details></li>
<li class="toctree-l1"><a class="reference internal" href="../spark_configs_support.html">Supported Spark Configurations</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Project Mechanics</span></p>
@@ -488,9 +507,9 @@ under the License.
<li class="breadcrumb-item"><a href="../index.html" class="nav-link">Comet Contributor Guide</a></li>
- <li class="breadcrumb-item"><a href="../benchmarking.html" class="nav-link">Comet Benchmarking Guide</a></li>
+ <li class="breadcrumb-item"><a href="index.html" class="nav-link">Expression Audits</a></li>
- <li class="breadcrumb-item active" aria-current="page"><span class="ellipsis">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</span></li>
+ <li class="breadcrumb-item active" aria-current="page"><span class="ellipsis">struct_funcs Expression Audits</span></li>
</ul>
</nav>
</div>
@@ -507,63 +526,47 @@ under the License.
<div id="searchbox"></div>
<article class="bd-article">
- <!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
+ <!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
-->
-<section id="apache-datafusion-comet-benchmarks-derived-from-tpc-ds">
-<h1>Apache DataFusion Comet: Benchmarks Derived From TPC-DS<a class="headerlink" href="#apache-datafusion-comet-benchmarks-derived-from-tpc-ds" title="Link to this heading">#</a></h1>
-<p>The following benchmarks were performed on an EKS cluster (<code class="docutils literal notranslate"><span class="pre">r6i.24xlarge</span></code> instances with EBS storage) with data stored in S3.</p>
-<section id="benchmark-results">
-<h2>Benchmark Results<a class="headerlink" href="#benchmark-results" title="Link to this heading">#</a></h2>
-<p>Total time to run all queries (lower is better).</p>
-<p><img alt="" src="../../_images/tpcds_allqueries.png" /></p>
-<p>Per-query breakdown showing the relative performance of Spark and Comet.</p>
-<p><img alt="" src="../../_images/tpcds_queries_compare.png" /></p>
-<p>How much Comet accelerates each query in relative terms.</p>
-<p><img alt="" src="../../_images/tpcds_queries_speedup_rel.png" /></p>
-<p>How much Comet accelerates each query in absolute terms.</p>
-<p><img alt="" src="../../_images/tpcds_queries_speedup_abs.png" /></p>
+<section id="struct-funcs-expression-audits">
+<h1>struct_funcs Expression Audits<a class="headerlink" href="#struct-funcs-expression-audits" title="Link to this heading">#</a></h1>
+<blockquote>
+<div><p>Audit notes for expressions in this category that have been audited. Absence of an entry means the expression has not been audited yet, not that it is unsupported. See the user guide <a class="reference internal" href="../../user-guide/latest/expressions.html"><span class="std std-doc">Spark Expression Support</span></a> for current support status.</p>
+</div></blockquote>
+<section id="named-struct">
+<h2>named_struct<a class="headerlink" href="#named-struct" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">CreateNamedStruct(children)</span></code> with <code class="docutils literal notranslate"><span class="pre">NoThrow</span></code>; <code class="docutils literal notranslate"><span class="pre">children</span></code> is <code class="docutils literal notranslate"><span class="pre">Seq(name1,</span> <span class="pre">val1,</span> <span class="pre">name2,</span> <span class="pre">va [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): semantics unchanged; an <code class="docutils literal notranslate"><span class="pre">override</span> <span class="pre">def</span> <span class="pre">stateful:</span> <span class="pre">Boolean</span> <span class="pre">=</span> <span class="pre">true</span></code> flag is added for the optimizer. Parser-level Alias unwrapping (<code class="docutils literal notranslate"><span class="pre">case</span> <span class="pre">(a:</span> <span class="pre">Alias [...]
+<li><p>Spark 4.1.1 (audited 2026-05-27): semantics unchanged; adds <code class="docutils literal notranslate"><span class="pre">override</span> <span class="pre">def</span> <span class="pre">contextIndependentFoldable:</span> <span class="pre">Boolean</span> <span class="pre">=</span> <span class="pre">children.forall(_.contextIndependentFoldable)</span></code> for the new constant-folding pass; no impact on Comet conversion.</p></li>
+<li><p>Comet limitation: a <code class="docutils literal notranslate"><span class="pre">CreateNamedStruct</span></code> whose names contain duplicates falls back to Spark. (Spark’s analyzer also tolerates duplicates at the column-name level, but the proto layer would lose them, so the fallback is the safe choice.)</p></li>
+</ul>
</section>
-<section id="configuration">
-<h2>Configuration<a class="headerlink" href="#configuration" title="Link to this heading">#</a></h2>
-<p>Common:</p>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.executor.instances</span><span class="o">=</span><span class="s">32</span>
-<span class="na">spark.executor.cores</span><span class="o">=</span><span class="s">16</span>
-<span class="na">spark.memory.fraction</span><span class="o">=</span><span class="s">0.6</span>
-<span class="na">spark.memory.storageFraction</span><span class="o">=</span><span class="s">0.2</span>
-<span class="c1"># Kubernetes CPU constraints</span>
-<span class="na">spark.kubernetes.executor.request.cores</span><span class="o">=</span><span class="s">8</span>
-<span class="na">spark.kubernetes.executor.limit.cores</span><span class="o">=</span><span class="s">8</span>
-</pre></div>
-</div>
-<p>Spark:</p>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.executor.memory</span><span class="o">=</span><span class="s">64G</span>
-<span class="na">spark.executor.memoryOverhead</span><span class="o">=</span><span class="s">10G</span>
-</pre></div>
-</div>
-<p>Comet:</p>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.executor.memory</span><span class="o">=</span><span class="s">64G</span>
-<span class="na">spark.executor.memoryOverhead</span><span class="o">=</span><span class="s">10G</span>
-<span class="na">spark.memory.offHeap.enabled</span><span class="o">=</span><span class="s">true</span>
-<span class="na">spark.memory.offHeap.size</span><span class="o">=</span><span class="s">32G</span>
-<span class="na">spark.comet.memoryPool.fraction</span><span class="o">=</span><span class="s">0.8</span>
-</pre></div>
-</div>
+<section id="struct">
+<h2>struct<a class="headerlink" href="#struct" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): the SQL function <code class="docutils literal notranslate"><span class="pre">struct(a,</span> <span class="pre">b,</span> <span class="pre">c)</span></code> is built by <code class="docutils literal notranslate"><span class="pre">CreateStruct.create</span></code>, which lowers to a <code class="docutils literal notranslate"><span class="pre">CreateNamedStruct</span></code> with synthetic field names <code class="docutils literal notranslate"><spa [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): same lowering; identical to 3.5.8.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-05-27): same lowering; identical to 3.5.8.</p></li>
+</ul>
</section>
</section>
@@ -578,20 +581,20 @@ under the License.
<div class="prev-next-area">
<a class="left-prev"
- href="tpc-h.html"
+ href="predicate_funcs.html"
title="previous page">
<i class="fa-solid fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">Apache DataFusion Comet: Benchmarks Derived From TPC-H</p>
+ <p class="prev-next-title">predicate_funcs Expression Audits</p>
</div>
</a>
<a class="right-next"
- href="../benchmarking_macos.html"
+ href="url_funcs.html"
title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
- <p class="prev-next-title">Comet Benchmarking on macOS</p>
+ <p class="prev-next-title">url_funcs Expression Audits</p>
</div>
<i class="fa-solid fa-angle-right"></i>
</a>
diff --git a/contributor-guide/benchmark-results/tpc-ds.html b/contributor-guide/expression-audits/url_funcs.html
similarity index 78%
copy from contributor-guide/benchmark-results/tpc-ds.html
copy to contributor-guide/expression-audits/url_funcs.html
index 4d5bf9b1c4..0cb96946a7 100644
--- a/contributor-guide/benchmark-results/tpc-ds.html
+++ b/contributor-guide/expression-audits/url_funcs.html
@@ -27,7 +27,7 @@ under the License.
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
- <title>Apache DataFusion Comet: Benchmarks Derived From TPC-DS — Apache DataFusion Comet documentation</title>
+ <title>url_funcs Expression Audits — Apache DataFusion Comet documentation</title>
@@ -61,12 +61,12 @@ under the License.
<script src="../../_static/documentation_options.js?v=5929fcd5"></script>
<script src="../../_static/doctools.js?v=9a2dae69"></script>
<script src="../../_static/sphinx_highlight.js?v=dc90522c"></script>
- <script>DOCUMENTATION_OPTIONS.pagename = 'contributor-guide/benchmark-results/tpc-ds';</script>
+ <script>DOCUMENTATION_OPTIONS.pagename = 'contributor-guide/expression-audits/url_funcs';</script>
<script async="true" defer="true" src="https://buttons.github.io/buttons.js"></script>
<link rel="index" title="Index" href="../../genindex.html" />
<link rel="search" title="Search" href="../../search.html" />
- <link rel="next" title="Comet Benchmarking on macOS" href="../benchmarking_macos.html" />
- <link rel="prev" title="Apache DataFusion Comet: Benchmarks Derived From TPC-H" href="tpc-h.html" />
+ <link rel="next" title="window_funcs Expression Audits" href="window_funcs.html" />
+ <link rel="prev" title="struct_funcs Expression Audits" href="struct_funcs.html" />
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<meta name="docsearch:language" content="en"/>
<meta name="docsearch:version" content="" />
@@ -418,11 +418,11 @@ under the License.
<li class="toctree-l1"><a class="reference internal" href="../iceberg-spark-tests.html">Iceberg Spark Tests</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Debugging and Performance</span></p>
-<ul class="current nav bd-sidenav">
+<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../debugging.html">Debugging Guide</a></li>
-<li class="toctree-l1 current active has-children"><a class="reference internal" href="../benchmarking.html">Benchmarking Guide</a><details open="open"><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul class="current">
-<li class="toctree-l2"><a class="reference internal" href="tpc-h.html">Apache DataFusion Comet: Benchmarks Derived From TPC-H</a></li>
-<li class="toctree-l2 current active"><a class="current reference internal" href="#">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</a></li>
+<li class="toctree-l1 has-children"><a class="reference internal" href="../benchmarking.html">Benchmarking Guide</a><details><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul>
+<li class="toctree-l2"><a class="reference internal" href="../benchmark-results/tpc-h.html">Apache DataFusion Comet: Benchmarks Derived From TPC-H</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../benchmark-results/tpc-ds.html">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_macos.html">Comet Benchmarking on macOS</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_aws_ec2.html">Comet Benchmarking in EC2</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_spark_sql_perf.html">TPC-DS Benchmarking with spark-sql-perf</a></li>
@@ -432,8 +432,27 @@ under the License.
<li class="toctree-l1"><a class="reference internal" href="../tracing.html">Tracing</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Reference</span></p>
-<ul class="nav bd-sidenav">
-<li class="toctree-l1"><a class="reference internal" href="../spark_expressions_support.html">Supported Spark Expressions</a></li>
+<ul class="current nav bd-sidenav">
+<li class="toctree-l1 current active has-children"><a class="reference internal" href="index.html">Expression Audits</a><details open="open"><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul class="current">
+<li class="toctree-l2"><a class="reference internal" href="agg_funcs.html">agg_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="array_funcs.html">array_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="bitwise_funcs.html">bitwise_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="collection_funcs.html">collection_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="conditional_funcs.html">conditional_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="conversion_funcs.html">conversion_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="datetime_funcs.html">datetime_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="generator_funcs.html">generator_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="hash_funcs.html">hash_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="json_funcs.html">json_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="map_funcs.html">map_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="math_funcs.html">math_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="misc_funcs.html">misc_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="predicate_funcs.html">predicate_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="struct_funcs.html">struct_funcs</a></li>
+<li class="toctree-l2 current active"><a class="current reference internal" href="#">url_funcs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="window_funcs.html">window_funcs</a></li>
+</ul>
+</details></li>
<li class="toctree-l1"><a class="reference internal" href="../spark_configs_support.html">Supported Spark Configurations</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Project Mechanics</span></p>
@@ -488,9 +507,9 @@ under the License.
<li class="breadcrumb-item"><a href="../index.html" class="nav-link">Comet Contributor Guide</a></li>
- <li class="breadcrumb-item"><a href="../benchmarking.html" class="nav-link">Comet Benchmarking Guide</a></li>
+ <li class="breadcrumb-item"><a href="index.html" class="nav-link">Expression Audits</a></li>
- <li class="breadcrumb-item active" aria-current="page"><span class="ellipsis">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</span></li>
+ <li class="breadcrumb-item active" aria-current="page"><span class="ellipsis">url_funcs Expression Audits</span></li>
</ul>
</nav>
</div>
@@ -507,63 +526,50 @@ under the License.
<div id="searchbox"></div>
<article class="bd-article">
- <!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
+ <!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
-->
-<section id="apache-datafusion-comet-benchmarks-derived-from-tpc-ds">
-<h1>Apache DataFusion Comet: Benchmarks Derived From TPC-DS<a class="headerlink" href="#apache-datafusion-comet-benchmarks-derived-from-tpc-ds" title="Link to this heading">#</a></h1>
-<p>The following benchmarks were performed on an EKS cluster (<code class="docutils literal notranslate"><span class="pre">r6i.24xlarge</span></code> instances with EBS storage) with data stored in S3.</p>
-<section id="benchmark-results">
-<h2>Benchmark Results<a class="headerlink" href="#benchmark-results" title="Link to this heading">#</a></h2>
-<p>Total time to run all queries (lower is better).</p>
-<p><img alt="" src="../../_images/tpcds_allqueries.png" /></p>
-<p>Per-query breakdown showing the relative performance of Spark and Comet.</p>
-<p><img alt="" src="../../_images/tpcds_queries_compare.png" /></p>
-<p>How much Comet accelerates each query in relative terms.</p>
-<p><img alt="" src="../../_images/tpcds_queries_speedup_rel.png" /></p>
-<p>How much Comet accelerates each query in absolute terms.</p>
-<p><img alt="" src="../../_images/tpcds_queries_speedup_abs.png" /></p>
+<section id="url-funcs-expression-audits">
+<h1>url_funcs Expression Audits<a class="headerlink" href="#url-funcs-expression-audits" title="Link to this heading">#</a></h1>
+<blockquote>
+<div><p>Audit notes for expressions in this category that have been audited. Absence of an entry means the expression has not been audited yet, not that it is unsupported. See the user guide <a class="reference internal" href="../../user-guide/latest/expressions.html"><span class="std std-doc">Spark Expression Support</span></a> for current support status.</p>
+</div></blockquote>
+<section id="try-url-decode">
+<h2>try_url_decode<a class="headerlink" href="#try-url-decode" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>4.0.1, 2026-05-05</p></li>
+</ul>
</section>
-<section id="configuration">
-<h2>Configuration<a class="headerlink" href="#configuration" title="Link to this heading">#</a></h2>
-<p>Common:</p>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.executor.instances</span><span class="o">=</span><span class="s">32</span>
-<span class="na">spark.executor.cores</span><span class="o">=</span><span class="s">16</span>
-<span class="na">spark.memory.fraction</span><span class="o">=</span><span class="s">0.6</span>
-<span class="na">spark.memory.storageFraction</span><span class="o">=</span><span class="s">0.2</span>
-<span class="c1"># Kubernetes CPU constraints</span>
-<span class="na">spark.kubernetes.executor.request.cores</span><span class="o">=</span><span class="s">8</span>
-<span class="na">spark.kubernetes.executor.limit.cores</span><span class="o">=</span><span class="s">8</span>
-</pre></div>
-</div>
-<p>Spark:</p>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.executor.memory</span><span class="o">=</span><span class="s">64G</span>
-<span class="na">spark.executor.memoryOverhead</span><span class="o">=</span><span class="s">10G</span>
-</pre></div>
-</div>
-<p>Comet:</p>
-<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.executor.memory</span><span class="o">=</span><span class="s">64G</span>
-<span class="na">spark.executor.memoryOverhead</span><span class="o">=</span><span class="s">10G</span>
-<span class="na">spark.memory.offHeap.enabled</span><span class="o">=</span><span class="s">true</span>
-<span class="na">spark.memory.offHeap.size</span><span class="o">=</span><span class="s">32G</span>
-<span class="na">spark.comet.memoryPool.fraction</span><span class="o">=</span><span class="s">0.8</span>
-</pre></div>
-</div>
+<section id="url-decode">
+<h2>url_decode<a class="headerlink" href="#url-decode" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>3.4.3, 2026-04-29</p></li>
+<li><p>3.5.8, 2026-04-29</p></li>
+<li><p>4.0.1, 2026-04-29</p></li>
+</ul>
+</section>
+<section id="url-encode">
+<h2>url_encode<a class="headerlink" href="#url-encode" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>3.4.3, 2026-04-29</p></li>
+<li><p>3.5.8, 2026-04-29</p></li>
+<li><p>4.0.1, 2026-04-29</p></li>
+</ul>
</section>
</section>
@@ -578,20 +584,20 @@ under the License.
<div class="prev-next-area">
<a class="left-prev"
- href="tpc-h.html"
+ href="struct_funcs.html"
title="previous page">
<i class="fa-solid fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">Apache DataFusion Comet: Benchmarks Derived From TPC-H</p>
+ <p class="prev-next-title">struct_funcs Expression Audits</p>
</div>
</a>
<a class="right-next"
- href="../benchmarking_macos.html"
+ href="window_funcs.html"
title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
- <p class="prev-next-title">Comet Benchmarking on macOS</p>
+ <p class="prev-next-title">window_funcs Expression Audits</p>
</div>
<i class="fa-solid fa-angle-right"></i>
</a>
diff --git a/contributor-guide/benchmark-results/tpc-ds.html b/contributor-guide/expression-audits/window_funcs.html
similarity index 78%
copy from contributor-guide/benchmark-results/tpc-ds.html
copy to contributor-guide/expression-audits/window_funcs.html
index 4d5bf9b1c4..dfb57181a8 100644
--- a/contributor-guide/benchmark-results/tpc-ds.html
+++ b/contributor-guide/expression-audits/window_funcs.html
@@ -27,7 +27,7 @@ under the License.
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
- <title>Apache DataFusion Comet: Benchmarks Derived From TPC-DS — Apache DataFusion Comet documentation</title>
+ <title>window_funcs Expression Audits — Apache DataFusion Comet documentation</title>
@@ -61,12 +61,12 @@ under the License.
<script src="../../_static/documentation_options.js?v=5929fcd5"></script>
<script src="../../_static/doctools.js?v=9a2dae69"></script>
<script src="../../_static/sphinx_highlight.js?v=dc90522c"></script>
- <script>DOCUMENTATION_OPTIONS.pagename = 'contributor-guide/benchmark-results/tpc-ds';</script>
+ <script>DOCUMENTATION_OPTIONS.pagename = 'contributor-guide/expression-audits/window_funcs';</script>
<script async="true" defer="true" src="https://buttons.github.io/buttons.js"></script>
<link rel="index" title="Index" href="../../genindex.html" />
<link rel="search" title="Search" href="../../search.html" />
- <link rel="next" title="Comet Benchmarking on macOS" href="../benchmarking_macos.html" />
- <link rel="prev" title="Apache DataFusion Comet: Benchmarks Derived From TPC-H" href="tpc-h.html" />
+ <link rel="next" title="Supported Spark Configurations" href="../spark_configs_support.html" />
+ <link rel="prev" title="url_funcs Expression Audits" href="url_funcs.html" />
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<meta name="docsearch:language" content="en"/>
<meta name="docsearch:version" content="" />
@@ -418,11 +418,11 @@ under the License.
<li class="toctree-l1"><a class="reference internal" href="../iceberg-spark-tests.html">Iceberg Spark Tests</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Debugging and Performance</span></p>
-<ul class="current nav bd-sidenav">
+<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../debugging.html">Debugging Guide</a></li>
-<li class="toctree-l1 current active has-children"><a class="reference internal" href="../benchmarking.html">Benchmarking Guide</a><details open="open"><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul class="current">
-<li class="toctree-l2"><a class="reference internal" href="tpc-h.html">Apache DataFusion Comet: Benchmarks Derived From TPC-H</a></li>
-<li class="toctree-l2 current active"><a class="current reference internal" href="#">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</a></li>
+<li class="toctree-l1 has-children"><a class="reference internal" href="../benchmarking.html">Benchmarking Guide</a><details><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul>
+<li class="toctree-l2"><a class="reference internal" href="../benchmark-results/tpc-h.html">Apache DataFusion Comet: Benchmarks Derived From TPC-H</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../benchmark-results/tpc-ds.html">Apache DataFusion Comet: Benchmarks Derived From TPC-DS</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_macos.html">Comet Benchmarking on macOS</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_aws_ec2.html">Comet Benchmarking in EC2</a></li>
<li class="toctree-l2"><a class="reference internal" href="../benchmarking_spark_sql_perf.html">TPC-DS Benchmarking with spark-sql-perf</a></li>
@@ -432,8 +432,27 @@ under the License.
<li class="toctree-l1"><a class="reference internal" href="../tracing.html">Tracing</a></li>
... 3508 lines suppressed ...
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org