You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/03 13:45:57 UTC
(datafusion-comet) branch asf-site updated: Publish built docs triggered by 9c0fde09c075212c99e295c39f44ae67e9ba7414
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 7bfc722a1e Publish built docs triggered by 9c0fde09c075212c99e295c39f44ae67e9ba7414
7bfc722a1e is described below
commit 7bfc722a1eaa493dc061066e67cc639a88af3f72
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Sat Oct 3 13:45:53 2026 +0000
Publish built docs triggered by 9c0fde09c075212c99e295c39f44ae67e9ba7414
---
...id-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg | 2 +-
_sources/contributor-guide/native_shuffle.md.txt | 4 +--
.../expressions/spark-3.4/misc.md.txt | 2 +-
.../expressions/spark-3.5/misc.md.txt | 2 +-
.../expressions/spark-4.0/misc.md.txt | 2 +-
.../expressions/spark-4.1/misc.md.txt | 2 +-
.../expressions/spark-4.2/misc.md.txt | 2 +-
.../latest/compatibility/floating-point.md.txt | 33 +++++++++++++---------
.../latest/compatibility/operators.md.txt | 22 ++++++---------
_sources/user-guide/latest/configs.md.txt | 2 +-
_sources/user-guide/latest/pyarrow-udfs.md.txt | 11 ++++++++
_sources/user-guide/latest/tuning/operators.md.txt | 21 +++++++-------
contributor-guide/native_shuffle.html | 4 +--
searchindex.js | 2 +-
.../compatibility/expressions/spark-3.4/misc.html | 2 +-
.../compatibility/expressions/spark-3.5/misc.html | 2 +-
.../compatibility/expressions/spark-4.0/misc.html | 2 +-
.../compatibility/expressions/spark-4.1/misc.html | 2 +-
.../compatibility/expressions/spark-4.2/misc.html | 2 +-
.../latest/compatibility/floating-point.html | 29 +++++++++++--------
user-guide/latest/compatibility/operators.html | 21 +++++---------
user-guide/latest/configs.html | 2 +-
user-guide/latest/pyarrow-udfs.html | 11 ++++++++
user-guide/latest/tuning/operators.html | 19 +++++++------
24 files changed, 112 insertions(+), 91 deletions(-)
diff --git a/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg b/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg
index 34016f92b0..538ca86f6f 100644
--- a/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg
+++ b/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg
@@ -1 +1 @@
-<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 966" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
+<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 966" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
diff --git a/_sources/contributor-guide/native_shuffle.md.txt b/_sources/contributor-guide/native_shuffle.md.txt
index e1cabfd6c6..b882acd25a 100644
--- a/_sources/contributor-guide/native_shuffle.md.txt
+++ b/_sources/contributor-guide/native_shuffle.md.txt
@@ -63,8 +63,8 @@ Native shuffle (`CometExchange`) is selected when all of the following condition
compares raw bytes. Scalar float and double are supported, including when
`spark.comet.exec.strictFloatingPoint` is enabled, because the native range partitioner
normalizes its comparison keys and its sampled boundary rows the same way the native sort
- does. Strict floating point only affects floating-point values nested in arrays, structs, or
- maps, which are rejected as range keys for being nested anyway.
+ does. The native sort normalizes floating-point values nested in arrays and structs as well,
+ but those keys are rejected as range keys for being nested.
- `HashPartitioning` keys must be primitive **by default**. Setting
`spark.comet.shuffle.native.partitioning.hash.nested.enabled` to `true` admits structs and
arrays as keys, checked recursively to their leaves, and maps on Spark 4.0 and later, where
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-3.4/misc.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-3.4/misc.md.txt
index b206e6aee3..6c5d228cea 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-3.4/misc.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-3.4/misc.md.txt
@@ -49,7 +49,7 @@ The following cases are not supported by Comet and always fall back to Spark, re
The following incompatibilities cause `SortOrder` to fall back to Spark by default. Set `spark.comet.expression.SortOrder.allowIncompatible=true` to enable Comet acceleration despite these differences.
-- Sorting on floating-point values nested in arrays, structs, or maps is not 100% compatible with Spark when `spark.comet.exec.strictFloatingPoint=true`
+- Sorting on floating-point values nested in an array or struct that can hold a null element or field is not 100% compatible with Spark when `spark.comet.exec.strictFloatingPoint=true`
## StaticInvoke
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-3.5/misc.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-3.5/misc.md.txt
index 876efc269d..aadd0a07ed 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-3.5/misc.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-3.5/misc.md.txt
@@ -49,7 +49,7 @@ The following cases are not supported by Comet and always fall back to Spark, re
The following incompatibilities cause `SortOrder` to fall back to Spark by default. Set `spark.comet.expression.SortOrder.allowIncompatible=true` to enable Comet acceleration despite these differences.
-- Sorting on floating-point values nested in arrays, structs, or maps is not 100% compatible with Spark when `spark.comet.exec.strictFloatingPoint=true`
+- Sorting on floating-point values nested in an array or struct that can hold a null element or field is not 100% compatible with Spark when `spark.comet.exec.strictFloatingPoint=true`
## StaticInvoke
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-4.0/misc.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-4.0/misc.md.txt
index 5b4c803c11..487b7416d3 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-4.0/misc.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-4.0/misc.md.txt
@@ -69,7 +69,7 @@ The following cases are not supported by Comet and always fall back to Spark, re
The following incompatibilities cause `SortOrder` to fall back to Spark by default. Set `spark.comet.expression.SortOrder.allowIncompatible=true` to enable Comet acceleration despite these differences.
-- Sorting on floating-point values nested in arrays, structs, or maps is not 100% compatible with Spark when `spark.comet.exec.strictFloatingPoint=true`
+- Sorting on floating-point values nested in an array or struct that can hold a null element or field is not 100% compatible with Spark when `spark.comet.exec.strictFloatingPoint=true`
## StaticInvoke
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-4.1/misc.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-4.1/misc.md.txt
index 5b4c803c11..487b7416d3 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-4.1/misc.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-4.1/misc.md.txt
@@ -69,7 +69,7 @@ The following cases are not supported by Comet and always fall back to Spark, re
The following incompatibilities cause `SortOrder` to fall back to Spark by default. Set `spark.comet.expression.SortOrder.allowIncompatible=true` to enable Comet acceleration despite these differences.
-- Sorting on floating-point values nested in arrays, structs, or maps is not 100% compatible with Spark when `spark.comet.exec.strictFloatingPoint=true`
+- Sorting on floating-point values nested in an array or struct that can hold a null element or field is not 100% compatible with Spark when `spark.comet.exec.strictFloatingPoint=true`
## StaticInvoke
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-4.2/misc.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-4.2/misc.md.txt
index 5b4c803c11..487b7416d3 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-4.2/misc.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-4.2/misc.md.txt
@@ -69,7 +69,7 @@ The following cases are not supported by Comet and always fall back to Spark, re
The following incompatibilities cause `SortOrder` to fall back to Spark by default. Set `spark.comet.expression.SortOrder.allowIncompatible=true` to enable Comet acceleration despite these differences.
-- Sorting on floating-point values nested in arrays, structs, or maps is not 100% compatible with Spark when `spark.comet.exec.strictFloatingPoint=true`
+- Sorting on floating-point values nested in an array or struct that can hold a null element or field is not 100% compatible with Spark when `spark.comet.exec.strictFloatingPoint=true`
## StaticInvoke
diff --git a/_sources/user-guide/latest/compatibility/floating-point.md.txt b/_sources/user-guide/latest/compatibility/floating-point.md.txt
index 7ed704bb54..0ffd7313ac 100644
--- a/_sources/user-guide/latest/compatibility/floating-point.md.txt
+++ b/_sources/user-guide/latest/compatibility/floating-point.md.txt
@@ -48,20 +48,25 @@ Spark's `ORDER BY`, `RANK`, `DENSE_RANK`, and window frame comparisons route thr
`SQLOrderingUtil.compareDoubles` / `compareFloats`, which equate all NaN representations and
define `-0.0 == 0.0`. NaN sorts above every non-NaN value.
-For scalar `FLOAT` and `DOUBLE` keys, Comet normalizes NaNs and signed zeros before native
-sorting, window peer comparisons, and `WindowGroupLimitExec` rank comparisons. Native range
-partitioning normalizes its keys and sampled boundaries in the same way. Only comparison keys
-are normalized; returned values retain their original NaN representations and zero signs.
-
-Native sorting of floating-point values nested in arrays or structs still uses Arrow's raw total
-ordering. Nested keys can therefore produce different ordering or rank results from Spark; see
-[#5507](https://github.com/apache/datafusion-comet/issues/5507).
-
-Because those scalar comparison keys match Spark, `spark.comet.exec.strictFloatingPoint=true` no
-longer forces a fallback for them: scalar `FLOAT` and `DOUBLE` sort keys, window and rank order
-keys, and range partitioning keys all stay native under strict mode. Floating-point values nested
-in arrays, structs, or maps still fall back under strict mode, because their ordering is the raw
-total ordering described above.
+For `FLOAT` and `DOUBLE` keys, and for keys that nest them in arrays and structs at any depth,
+Comet normalizes NaNs and signed zeros before native sorting, window peer comparisons, and
+`WindowGroupLimitExec` rank comparisons. Native range partitioning normalizes its keys and
+sampled boundaries in the same way; it only accepts scalar keys. Only comparison keys are
+normalized; returned values retain their original NaN representations and zero signs.
+
+Because those comparison keys match Spark, `spark.comet.exec.strictFloatingPoint=true` does not
+force a fallback for them: sort keys, window and rank order keys, and range partitioning keys all
+stay native under strict mode, whether the floats in them are scalar or nested.
+
+The exception is a key that nests floats in an array or struct whose type can hold a null element
+or field. Spark orders such a null below every other value, whatever the key's `NULLS FIRST` or
+`NULLS LAST`. The native sort places it by that null order, so `ASC NULLS LAST` and
+`DESC NULLS FIRST` can differ from Spark, and a `RANGE` window frame orders it above every other
+value, so a running aggregate can span the whole partition
+([#6476](https://github.com/apache/datafusion-comet/issues/6476),
+[#6477](https://github.com/apache/datafusion-comet/issues/6477)). Strict mode makes those keys fall
+back to Spark. A key whose type cannot hold a null, such as `array(coalesce(x, 0.0D))`, stays
+native.
`array_min` and `array_max` use Spark-compatible native comparisons in both strict and non-strict
floating-point modes. Signed zeros compare equal, and all NaN representations compare equal and
diff --git a/_sources/user-guide/latest/compatibility/operators.md.txt b/_sources/user-guide/latest/compatibility/operators.md.txt
index 53b290726a..657a32de53 100644
--- a/_sources/user-guide/latest/compatibility/operators.md.txt
+++ b/_sources/user-guide/latest/compatibility/operators.md.txt
@@ -88,6 +88,10 @@ incorrect result. When any single window expression in a `WindowExec` falls back
overflow instead of returning Spark's `NULL`.
- `RANGE` frame with an explicit offset when the `ORDER BY` column is `DATE` or `DECIMAL`
([#4834](https://github.com/apache/datafusion-comet/issues/4834)).
+- `RANGE` frame bounded by `CURRENT ROW` when an `ORDER BY` key is an array of arrays or structs, or a struct
+ holding an array, such as `array(named_struct('x', x))`. DataFusion cannot compare those values to find the
+ frame's bounds ([apache/datafusion#24937](https://github.com/apache/datafusion/issues/24937)). Ranking functions
+ and `ROWS` frames over the same keys run natively.
- `first_value` / `last_value` on a `RANGE` frame with a literal offset
([#4835](https://github.com/apache/datafusion-comet/issues/4835)).
- `lag` / `lead` with a non-literal default value ([#4268](https://github.com/apache/datafusion-comet/issues/4268)).
@@ -102,20 +106,10 @@ runs natively; it is controlled by `spark.comet.exec.windowGroupLimit.enabled` (
- Any `PARTITION BY` or `ORDER BY` key whose type carries a non-default `StringType` collation
(e.g. `UTF8_LCASE`). The native operator detects partitions and order-key peer groups by
comparing Arrow row-encoded keys for byte equality, which splits peers that Spark ties.
-- `RANK` and `DENSE_RANK` whose `ORDER BY` key has a `FLOAT` or `DOUBLE` nested in an array or
- struct. The same byte equality decides their ties, and nested floating-point values aren't
- normalized, so `-0.0` and `+0.0`, or two NaN representations, would get different ranks and the
- cutoff would drop rows that Spark keeps
- ([#5507](https://github.com/apache/datafusion-comet/issues/5507)).
-
-**Known incompatibilities:**
-
-- `ROW_NUMBER` over such a key still runs natively and follows the native sort, which compares
- nested floating-point values with Arrow's raw total ordering. Which of two rows that differ only
- in `-0.0` and `+0.0`, or in their NaN representation, gets the lower row number can therefore
- differ from Spark ([#5507](https://github.com/apache/datafusion-comet/issues/5507)). Scalar
- `FLOAT` and `DOUBLE` keys are normalized and match Spark; see
- [floating-point ordering](./floating-point.md).
+
+Floating-point `ORDER BY` keys, including floats nested in arrays and structs, are normalized
+and match Spark's ranks; see [floating-point ordering](./floating-point.md), which also covers
+strict floating-point mode.
## Round-Robin Partitioning
diff --git a/_sources/user-guide/latest/configs.md.txt b/_sources/user-guide/latest/configs.md.txt
index 2aef1757ca..68c56ed57c 100644
--- a/_sources/user-guide/latest/configs.md.txt
+++ b/_sources/user-guide/latest/configs.md.txt
@@ -68,7 +68,7 @@ Comet provides the following configuration settings.
| `spark.comet.exec.join.dynamicFilter.enabled` | Experimental opt-in: use a completed native hash join build's key domain to prune eligible native Parquet row groups and filter probe batches before the hash probe. Supports inner joins with one direct signed integer key and one native partition per input, including both Spark build sides. The probe filter remains active when reader pruning is unavailable. Unsupported joins retain their existing execution path. Filters do not cross Spark [...]
| `spark.comet.exec.pyarrowUDF.enabled` | Experimental: whether to enable optimized execution of PyArrow UDFs (mapInArrow/mapInPandas). When enabled, Comet passes Arrow columnar data directly to Python UDFs without the intermediate Arrow-to-Row-to-Arrow conversion that Spark normally performs. Disabled by default while the feature stabilizes. | false |
| `spark.comet.exec.scalaUDF.codegen.enabled` | Whether to route Spark `ScalaUDF` expressions through Comet's Arrow-direct codegen dispatcher. When enabled, a supported ScalaUDF is compiled into a per-batch kernel that reads and writes Arrow vectors directly from native execution. When disabled, plans containing a ScalaUDF fall back to Spark for the enclosing operator. The same dispatcher backs the regex family (`rlike`, `regexp_replace`, `split`, `regexp_extract`, `regexp_extract_all`, [...]
-| `spark.comet.exec.strictFloatingPoint` | When enabled, fall back to Spark for floating-point operations that may differ from Spark, such as comparing -0.0 and 0.0, or sorting floating-point values nested in arrays, structs, or maps. Scalar `ORDER BY`, window ordering and range partitioning keys are unaffected, because Comet normalizes those comparison keys to match Spark, and so is `sort_array`, which follows Spark's ordering. For more information, refer to the [Comet Compatibility Gui [...]
+| `spark.comet.exec.strictFloatingPoint` | When enabled, fall back to Spark for floating-point operations that may differ from Spark, such as comparing -0.0 and 0.0. `ORDER BY`, window ordering and range partitioning keys are unaffected, including floating-point values nested in arrays and structs, because Comet normalizes those comparison keys to match Spark. The exception is a nested key whose type can hold a null element or field, which falls back until the native sort and window fram [...]
| `spark.comet.exec.topK.dynamicFilter.enabled` | Experimental opt-in: use a local TopK heap's improving threshold to prune eligible native Parquet input. Requires spark.comet.exec.topK.fusion.enabled. Supports one direct signed integer sort key. Thresholds are local to each native execution and do not cross Spark exchanges or JVM/Arrow boundaries. Unsupported readers retain ordinary TopK execution. | false |
| `spark.comet.exec.topK.fusion.enabled` | Experimental opt-in: run an eligible local TopK in the same native execution as its Parquet scan. Supports one direct signed integer sort key. This changes the local execution pipeline and can reduce scan/TopK overlap, so it may be slower for some workloads. | false |
| `spark.comet.exec.transitionRevert.enabled` | When enabled, Comet reverts a query stage to Spark row-based execution if the number of columnar-to-row (C2R) transitions in the stage exceeds the configured threshold. This avoids the overhead of repeated format conversions in stages where many operators fall back to row-based execution. | false |
diff --git a/_sources/user-guide/latest/pyarrow-udfs.md.txt b/_sources/user-guide/latest/pyarrow-udfs.md.txt
index eb704deb6f..d3f405fc90 100644
--- a/_sources/user-guide/latest/pyarrow-udfs.md.txt
+++ b/_sources/user-guide/latest/pyarrow-udfs.md.txt
@@ -207,6 +207,17 @@ on the unoptimized path.
requested by the configuration. `EliminateRedundantTransitions` therefore skips the rewrite
and vanilla Spark handles the operation. Comet can read `large_string` and `large_binary`
columns returned by a Python worker; that output support does not widen the input vectors.
+- PySpark does not convert the batches a `mapInArrow` UDF returns to the declared output schema,
+ so Comet checks their Arrow schema before reading them. When a column cannot be read as its
+ declared type, Comet raises Spark's `ARROW_TYPE_MISMATCH` error (on Spark 4.0, a
+ `SparkException` with the same message). Column names, nullability, `large_string` and
+ `large_binary` offsets, the time zone of a timestamp, and an all-null column of Arrow type
+ `null` may differ from the declared schema. Vanilla Spark reads some mismatches that Comet
+ rejects, unless `spark.sql.execution.arrow.pyspark.validateSchema.enabled` is set: it rescales
+ a decimal to the declared precision and scale, reads a timestamp without a time zone as
+ `TimestampType`, reads other types of the same width (for example `int64` as a timestamp), and
+ ignores extra columns. Return the declared types, or set
+ `spark.comet.exec.pyarrowUDF.enabled=false` for such UDFs.
- Comet applies `spark.sql.execution.arrow.maxRecordsPerBatch` to every input batch, including
batches with only plain columns. Before decoding dictionary-encoded shuffle columns, Comet also
compares their estimated decoded size with `spark.sql.execution.arrow.maxBytesPerBatch`.
diff --git a/_sources/user-guide/latest/tuning/operators.md.txt b/_sources/user-guide/latest/tuning/operators.md.txt
index f36546c237..644f35483b 100644
--- a/_sources/user-guide/latest/tuning/operators.md.txt
+++ b/_sources/user-guide/latest/tuning/operators.md.txt
@@ -149,16 +149,17 @@ See [TopK metrics](../metrics.md#local-topk).
## Optimizing Sorting on Floating-Point Values
-Comet normalizes NaN payloads and signed zeros in scalar `FLOAT` and `DOUBLE` ordering keys, so `ORDER BY`, window
-ordering and range partitioning on them match Spark and stay native even with
-`spark.comet.exec.strictFloatingPoint=true`. Only the comparison key is normalized; returned values keep their original
-NaN representation and zero sign.
-
-Floating-point values nested in arrays, structs, or maps are compared with Arrow's raw total ordering instead, which can
-differ from Spark when the data contains both zero and negative zero, or more than one NaN representation. This is likely
-an edge case that is not of concern for many users. Setting `spark.comet.exec.strictFloatingPoint=true` makes those
-nested cases fall back to Spark, and they can be forced back onto the native path with
-`spark.comet.expression.SortOrder.allowIncompatible=true`.
+Comet normalizes NaN payloads and signed zeros in `FLOAT` and `DOUBLE` ordering keys, including floating-point values
+nested in arrays and structs, so `ORDER BY`, window ordering and range partitioning on them match Spark and stay native
+even with `spark.comet.exec.strictFloatingPoint=true`. Only the comparison key is normalized; returned values keep their
+original NaN representation and zero sign.
+
+The exception is a key that nests floating-point values in an array or struct whose type can hold a null element or
+field. Spark orders such a null below every other value, and the native sort and `RANGE` window frames do not
+([#6476](https://github.com/apache/datafusion-comet/issues/6476),
+[#6477](https://github.com/apache/datafusion-comet/issues/6477)), so
+`spark.comet.exec.strictFloatingPoint=true` makes those keys fall back to Spark. They can be forced back onto the
+native path with `spark.comet.expression.SortOrder.allowIncompatible=true`.
`sort_array` sorts array elements rather than ordering rows. It follows Spark's floating-point ordering as well, so it
also stays native with `spark.comet.exec.strictFloatingPoint=true`.
diff --git a/contributor-guide/native_shuffle.html b/contributor-guide/native_shuffle.html
index ba293f9a49..5d657a5a0c 100644
--- a/contributor-guide/native_shuffle.html
+++ b/contributor-guide/native_shuffle.html
@@ -592,8 +592,8 @@ nested type, because native cannot sort them, and rejects collated strings, beca
compares raw bytes. Scalar float and double are supported, including when
<code class="docutils literal notranslate"><span class="pre">spark.comet.exec.strictFloatingPoint</span></code> is enabled, because the native range partitioner
normalizes its comparison keys and its sampled boundary rows the same way the native sort
-does. Strict floating point only affects floating-point values nested in arrays, structs, or
-maps, which are rejected as range keys for being nested anyway.</p></li>
+does. The native sort normalizes floating-point values nested in arrays and structs as well,
+but those keys are rejected as range keys for being nested.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">HashPartitioning</span></code> keys must be primitive <strong>by default</strong>. Setting
<code class="docutils literal notranslate"><span class="pre">spark.comet.shuffle.native.partitioning.hash.nested.enabled</span></code> to <code class="docutils literal notranslate"><span class="pre">true</span></code> admits structs and
arrays as keys, checked recursively to their leaves, and maps on Spark 4.0 and later, where
diff --git a/searchindex.js b/searchindex.js
index 2e90249ec9..b27e610ab2 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[57, "id1"]], "%": [[55, "id1"]], "&": [[45, "id1"]], "*": [[55, "id2"]], "+": [[55, "id3"]], "-": [[55, "id4"]], "/": [[55, "id5"]], "1. Format Your Code": [[42, "format-your-code"]], "1. Install Comet": [[63, "install-comet"], [75, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[42, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[63, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[57, "id1"]], "%": [[55, "id1"]], "&": [[45, "id1"]], "*": [[55, "id2"]], "+": [[55, "id3"]], "-": [[55, "id4"]], "/": [[55, "id5"]], "1. Format Your Code": [[42, "format-your-code"]], "1. Install Comet": [[63, "install-comet"], [75, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[42, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[63, "clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/compatibility/expressions/spark-3.4/misc.html b/user-guide/latest/compatibility/expressions/spark-3.4/misc.html
index a03339d8af..2b06c57e91 100644
--- a/user-guide/latest/compatibility/expressions/spark-3.4/misc.html
+++ b/user-guide/latest/compatibility/expressions/spark-3.4/misc.html
@@ -839,7 +839,7 @@ under the License.
<h2>SortOrder<a class="headerlink" href="#sortorder" title="Link to this heading">#</a></h2>
<p>The following incompatibilities cause <code class="docutils literal notranslate"><span class="pre">SortOrder</span></code> to fall back to Spark by default. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.SortOrder.allowIncompatible=true</span></code> to enable Comet acceleration despite these differences.</p>
<ul class="simple">
-<li><p>Sorting on floating-point values nested in arrays, structs, or maps is not 100% compatible with Spark when <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.strictFloatingPoint=true</span></code></p></li>
+<li><p>Sorting on floating-point values nested in an array or struct that can hold a null element or field is not 100% compatible with Spark when <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.strictFloatingPoint=true</span></code></p></li>
</ul>
</section>
<section id="staticinvoke">
diff --git a/user-guide/latest/compatibility/expressions/spark-3.5/misc.html b/user-guide/latest/compatibility/expressions/spark-3.5/misc.html
index 17d874e98b..65522f152d 100644
--- a/user-guide/latest/compatibility/expressions/spark-3.5/misc.html
+++ b/user-guide/latest/compatibility/expressions/spark-3.5/misc.html
@@ -839,7 +839,7 @@ under the License.
<h2>SortOrder<a class="headerlink" href="#sortorder" title="Link to this heading">#</a></h2>
<p>The following incompatibilities cause <code class="docutils literal notranslate"><span class="pre">SortOrder</span></code> to fall back to Spark by default. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.SortOrder.allowIncompatible=true</span></code> to enable Comet acceleration despite these differences.</p>
<ul class="simple">
-<li><p>Sorting on floating-point values nested in arrays, structs, or maps is not 100% compatible with Spark when <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.strictFloatingPoint=true</span></code></p></li>
+<li><p>Sorting on floating-point values nested in an array or struct that can hold a null element or field is not 100% compatible with Spark when <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.strictFloatingPoint=true</span></code></p></li>
</ul>
</section>
<section id="staticinvoke">
diff --git a/user-guide/latest/compatibility/expressions/spark-4.0/misc.html b/user-guide/latest/compatibility/expressions/spark-4.0/misc.html
index 13cdc703fe..f24c688950 100644
--- a/user-guide/latest/compatibility/expressions/spark-4.0/misc.html
+++ b/user-guide/latest/compatibility/expressions/spark-4.0/misc.html
@@ -861,7 +861,7 @@ under the License.
<h2>SortOrder<a class="headerlink" href="#sortorder" title="Link to this heading">#</a></h2>
<p>The following incompatibilities cause <code class="docutils literal notranslate"><span class="pre">SortOrder</span></code> to fall back to Spark by default. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.SortOrder.allowIncompatible=true</span></code> to enable Comet acceleration despite these differences.</p>
<ul class="simple">
-<li><p>Sorting on floating-point values nested in arrays, structs, or maps is not 100% compatible with Spark when <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.strictFloatingPoint=true</span></code></p></li>
+<li><p>Sorting on floating-point values nested in an array or struct that can hold a null element or field is not 100% compatible with Spark when <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.strictFloatingPoint=true</span></code></p></li>
</ul>
</section>
<section id="staticinvoke">
diff --git a/user-guide/latest/compatibility/expressions/spark-4.1/misc.html b/user-guide/latest/compatibility/expressions/spark-4.1/misc.html
index 89f6948602..266a42ed9f 100644
--- a/user-guide/latest/compatibility/expressions/spark-4.1/misc.html
+++ b/user-guide/latest/compatibility/expressions/spark-4.1/misc.html
@@ -861,7 +861,7 @@ under the License.
<h2>SortOrder<a class="headerlink" href="#sortorder" title="Link to this heading">#</a></h2>
<p>The following incompatibilities cause <code class="docutils literal notranslate"><span class="pre">SortOrder</span></code> to fall back to Spark by default. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.SortOrder.allowIncompatible=true</span></code> to enable Comet acceleration despite these differences.</p>
<ul class="simple">
-<li><p>Sorting on floating-point values nested in arrays, structs, or maps is not 100% compatible with Spark when <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.strictFloatingPoint=true</span></code></p></li>
+<li><p>Sorting on floating-point values nested in an array or struct that can hold a null element or field is not 100% compatible with Spark when <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.strictFloatingPoint=true</span></code></p></li>
</ul>
</section>
<section id="staticinvoke">
diff --git a/user-guide/latest/compatibility/expressions/spark-4.2/misc.html b/user-guide/latest/compatibility/expressions/spark-4.2/misc.html
index 02699765dc..c06c4a5b8f 100644
--- a/user-guide/latest/compatibility/expressions/spark-4.2/misc.html
+++ b/user-guide/latest/compatibility/expressions/spark-4.2/misc.html
@@ -861,7 +861,7 @@ under the License.
<h2>SortOrder<a class="headerlink" href="#sortorder" title="Link to this heading">#</a></h2>
<p>The following incompatibilities cause <code class="docutils literal notranslate"><span class="pre">SortOrder</span></code> to fall back to Spark by default. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.SortOrder.allowIncompatible=true</span></code> to enable Comet acceleration despite these differences.</p>
<ul class="simple">
-<li><p>Sorting on floating-point values nested in arrays, structs, or maps is not 100% compatible with Spark when <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.strictFloatingPoint=true</span></code></p></li>
+<li><p>Sorting on floating-point values nested in an array or struct that can hold a null element or field is not 100% compatible with Spark when <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.strictFloatingPoint=true</span></code></p></li>
</ul>
</section>
<section id="staticinvoke">
diff --git a/user-guide/latest/compatibility/floating-point.html b/user-guide/latest/compatibility/floating-point.html
index 6202942c9e..e335a5a8b7 100644
--- a/user-guide/latest/compatibility/floating-point.html
+++ b/user-guide/latest/compatibility/floating-point.html
@@ -825,18 +825,23 @@ preserve SQL null semantics and do not change the values returned by projections
<p>Spark’s <code class="docutils literal notranslate"><span class="pre">ORDER</span> <span class="pre">BY</span></code>, <code class="docutils literal notranslate"><span class="pre">RANK</span></code>, <code class="docutils literal notranslate"><span class="pre">DENSE_RANK</span></code>, and window frame comparisons route through
<code class="docutils literal notranslate"><span class="pre">SQLOrderingUtil.compareDoubles</span></code> / <code class="docutils literal notranslate"><span class="pre">compareFloats</span></code>, which equate all NaN representations and
define <code class="docutils literal notranslate"><span class="pre">-0.0</span> <span class="pre">==</span> <span class="pre">0.0</span></code>. NaN sorts above every non-NaN value.</p>
-<p>For scalar <code class="docutils literal notranslate"><span class="pre">FLOAT</span></code> and <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> keys, Comet normalizes NaNs and signed zeros before native
-sorting, window peer comparisons, and <code class="docutils literal notranslate"><span class="pre">WindowGroupLimitExec</span></code> rank comparisons. Native range
-partitioning normalizes its keys and sampled boundaries in the same way. Only comparison keys
-are normalized; returned values retain their original NaN representations and zero signs.</p>
-<p>Native sorting of floating-point values nested in arrays or structs still uses Arrow’s raw total
-ordering. Nested keys can therefore produce different ordering or rank results from Spark; see
-<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/5507">#5507</a>.</p>
-<p>Because those scalar comparison keys match Spark, <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.strictFloatingPoint=true</span></code> no
-longer forces a fallback for them: scalar <code class="docutils literal notranslate"><span class="pre">FLOAT</span></code> and <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> sort keys, window and rank order
-keys, and range partitioning keys all stay native under strict mode. Floating-point values nested
-in arrays, structs, or maps still fall back under strict mode, because their ordering is the raw
-total ordering described above.</p>
+<p>For <code class="docutils literal notranslate"><span class="pre">FLOAT</span></code> and <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> keys, and for keys that nest them in arrays and structs at any depth,
+Comet normalizes NaNs and signed zeros before native sorting, window peer comparisons, and
+<code class="docutils literal notranslate"><span class="pre">WindowGroupLimitExec</span></code> rank comparisons. Native range partitioning normalizes its keys and
+sampled boundaries in the same way; it only accepts scalar keys. Only comparison keys are
+normalized; returned values retain their original NaN representations and zero signs.</p>
+<p>Because those comparison keys match Spark, <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.strictFloatingPoint=true</span></code> does not
+force a fallback for them: sort keys, window and rank order keys, and range partitioning keys all
+stay native under strict mode, whether the floats in them are scalar or nested.</p>
+<p>The exception is a key that nests floats in an array or struct whose type can hold a null element
+or field. Spark orders such a null below every other value, whatever the key’s <code class="docutils literal notranslate"><span class="pre">NULLS</span> <span class="pre">FIRST</span></code> or
+<code class="docutils literal notranslate"><span class="pre">NULLS</span> <span class="pre">LAST</span></code>. The native sort places it by that null order, so <code class="docutils literal notranslate"><span class="pre">ASC</span> <span class="pre">NULLS</span> <span class="pre">LAST</span></code> and
+<code class="docutils literal notranslate"><span class="pre">DESC</span> <span class="pre">NULLS</span> <span class="pre">FIRST</span></code> can differ from Spark, and a <code class="docutils literal notranslate"><span class="pre">RANGE</span></code> window frame orders it above every other
+value, so a running aggregate can span the whole partition
+(<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/6476">#6476</a>,
+<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/6477">#6477</a>). Strict mode makes those keys fall
+back to Spark. A key whose type cannot hold a null, such as <code class="docutils literal notranslate"><span class="pre">array(coalesce(x,</span> <span class="pre">0.0D))</span></code>, stays
+native.</p>
<p><code class="docutils literal notranslate"><span class="pre">array_min</span></code> and <code class="docutils literal notranslate"><span class="pre">array_max</span></code> use Spark-compatible native comparisons in both strict and non-strict
floating-point modes. Signed zeros compare equal, and all NaN representations compare equal and
greater than non-NaN values. The original first equal element is retained: for example,
diff --git a/user-guide/latest/compatibility/operators.html b/user-guide/latest/compatibility/operators.html
index 79eef37492..d81506a64e 100644
--- a/user-guide/latest/compatibility/operators.html
+++ b/user-guide/latest/compatibility/operators.html
@@ -864,6 +864,10 @@ support as the batch aggregates, so these fall back in both contexts.</p></li>
overflow instead of returning Spark’s <code class="docutils literal notranslate"><span class="pre">NULL</span></code>.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">RANGE</span></code> frame with an explicit offset when the <code class="docutils literal notranslate"><span class="pre">ORDER</span> <span class="pre">BY</span></code> column is <code class="docutils literal notranslate"><span class="pre">DATE</span></code> or <code class="docutils literal notranslate"><span class="pre">DECIMAL</span></code>
(<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4834">#4834</a>).</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">RANGE</span></code> frame bounded by <code class="docutils literal notranslate"><span class="pre">CURRENT</span> <span class="pre">ROW</span></code> when an <code class="docutils literal notranslate"><span class="pre">ORDER</span> <span class="pre">BY</span></code> key is an array of arrays or structs, or a struct
+holding an array, such as <code class="docutils literal notranslate"><span class="pre">array(named_struct('x',</span> <span class="pre">x))</span></code>. DataFusion cannot compare those values to find the
+frame’s bounds (<a class="reference external" href="https://github.com/apache/datafusion/issues/24937">apache/datafusion#24937</a>). Ranking functions
+and <code class="docutils literal notranslate"><span class="pre">ROWS</span></code> frames over the same keys run natively.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">first_value</span></code> / <code class="docutils literal notranslate"><span class="pre">last_value</span></code> on a <code class="docutils literal notranslate"><span class="pre">RANGE</span></code> frame with a literal offset
(<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4835">#4835</a>).</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">lag</span></code> / <code class="docutils literal notranslate"><span class="pre">lead</span></code> with a non-literal default value (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4268">#4268</a>).</p></li>
@@ -877,21 +881,10 @@ runs natively; it is controlled by <code class="docutils literal notranslate"><s
<li><p>Any <code class="docutils literal notranslate"><span class="pre">PARTITION</span> <span class="pre">BY</span></code> or <code class="docutils literal notranslate"><span class="pre">ORDER</span> <span class="pre">BY</span></code> key whose type carries a non-default <code class="docutils literal notranslate"><span class="pre">StringType</span></code> collation
(e.g. <code class="docutils literal notranslate"><span class="pre">UTF8_LCASE</span></code>). The native operator detects partitions and order-key peer groups by
comparing Arrow row-encoded keys for byte equality, which splits peers that Spark ties.</p></li>
-<li><p><code class="docutils literal notranslate"><span class="pre">RANK</span></code> and <code class="docutils literal notranslate"><span class="pre">DENSE_RANK</span></code> whose <code class="docutils literal notranslate"><span class="pre">ORDER</span> <span class="pre">BY</span></code> key has a <code class="docutils literal notranslate"><span class="pre">FLOAT</span></code> or <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> nested in an array or
-struct. The same byte equality decides their ties, and nested floating-point values aren’t
-normalized, so <code class="docutils literal notranslate"><span class="pre">-0.0</span></code> and <code class="docutils literal notranslate"><span class="pre">+0.0</span></code>, or two NaN representations, would get different ranks and the
-cutoff would drop rows that Spark keeps
-(<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/5507">#5507</a>).</p></li>
-</ul>
-<p><strong>Known incompatibilities:</strong></p>
-<ul class="simple">
-<li><p><code class="docutils literal notranslate"><span class="pre">ROW_NUMBER</span></code> over such a key still runs natively and follows the native sort, which compares
-nested floating-point values with Arrow’s raw total ordering. Which of two rows that differ only
-in <code class="docutils literal notranslate"><span class="pre">-0.0</span></code> and <code class="docutils literal notranslate"><span class="pre">+0.0</span></code>, or in their NaN representation, gets the lower row number can therefore
-differ from Spark (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/5507">#5507</a>). Scalar
-<code class="docutils literal notranslate"><span class="pre">FLOAT</span></code> and <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> keys are normalized and match Spark; see
-<a class="reference internal" href="floating-point.html"><span class="std std-doc">floating-point ordering</span></a>.</p></li>
</ul>
+<p>Floating-point <code class="docutils literal notranslate"><span class="pre">ORDER</span> <span class="pre">BY</span></code> keys, including floats nested in arrays and structs, are normalized
+and match Spark’s ranks; see <a class="reference internal" href="floating-point.html"><span class="std std-doc">floating-point ordering</span></a>, which also covers
+strict floating-point mode.</p>
</section>
<section id="round-robin-partitioning">
<h2>Round-Robin Partitioning<a class="headerlink" href="#round-robin-partitioning" title="Link to this heading">#</a></h2>
diff --git a/user-guide/latest/configs.html b/user-guide/latest/configs.html
index 04e170335b..037f3e1af7 100644
--- a/user-guide/latest/configs.html
+++ b/user-guide/latest/configs.html
@@ -942,7 +942,7 @@ under the License.
<td><p>true</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.strictFloatingPoint</span></code></p></td>
-<td><p>When enabled, fall back to Spark for floating-point operations that may differ from Spark, such as comparing -0.0 and 0.0, or sorting floating-point values nested in arrays, structs, or maps. Scalar <code class="docutils literal notranslate"><span class="pre">ORDER</span> <span class="pre">BY</span></code>, window ordering and range partitioning keys are unaffected, because Comet normalizes those comparison keys to match Spark, and so is <code class="docutils literal notranslate"> [...]
+<td><p>When enabled, fall back to Spark for floating-point operations that may differ from Spark, such as comparing -0.0 and 0.0. <code class="docutils literal notranslate"><span class="pre">ORDER</span> <span class="pre">BY</span></code>, window ordering and range partitioning keys are unaffected, including floating-point values nested in arrays and structs, because Comet normalizes those comparison keys to match Spark. The exception is a nested key whose type can hold a null element or [...]
<td><p>false</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.topK.dynamicFilter.enabled</span></code></p></td>
diff --git a/user-guide/latest/pyarrow-udfs.html b/user-guide/latest/pyarrow-udfs.html
index a025c3cf34..142f8cd076 100644
--- a/user-guide/latest/pyarrow-udfs.html
+++ b/user-guide/latest/pyarrow-udfs.html
@@ -995,6 +995,17 @@ and <code class="docutils literal notranslate"><span class="pre">binary</span></
requested by the configuration. <code class="docutils literal notranslate"><span class="pre">EliminateRedundantTransitions</span></code> therefore skips the rewrite
and vanilla Spark handles the operation. Comet can read <code class="docutils literal notranslate"><span class="pre">large_string</span></code> and <code class="docutils literal notranslate"><span class="pre">large_binary</span></code>
columns returned by a Python worker; that output support does not widen the input vectors.</p></li>
+<li><p>PySpark does not convert the batches a <code class="docutils literal notranslate"><span class="pre">mapInArrow</span></code> UDF returns to the declared output schema,
+so Comet checks their Arrow schema before reading them. When a column cannot be read as its
+declared type, Comet raises Spark’s <code class="docutils literal notranslate"><span class="pre">ARROW_TYPE_MISMATCH</span></code> error (on Spark 4.0, a
+<code class="docutils literal notranslate"><span class="pre">SparkException</span></code> with the same message). Column names, nullability, <code class="docutils literal notranslate"><span class="pre">large_string</span></code> and
+<code class="docutils literal notranslate"><span class="pre">large_binary</span></code> offsets, the time zone of a timestamp, and an all-null column of Arrow type
+<code class="docutils literal notranslate"><span class="pre">null</span></code> may differ from the declared schema. Vanilla Spark reads some mismatches that Comet
+rejects, unless <code class="docutils literal notranslate"><span class="pre">spark.sql.execution.arrow.pyspark.validateSchema.enabled</span></code> is set: it rescales
+a decimal to the declared precision and scale, reads a timestamp without a time zone as
+<code class="docutils literal notranslate"><span class="pre">TimestampType</span></code>, reads other types of the same width (for example <code class="docutils literal notranslate"><span class="pre">int64</span></code> as a timestamp), and
+ignores extra columns. Return the declared types, or set
+<code class="docutils literal notranslate"><span class="pre">spark.comet.exec.pyarrowUDF.enabled=false</span></code> for such UDFs.</p></li>
<li><p>Comet applies <code class="docutils literal notranslate"><span class="pre">spark.sql.execution.arrow.maxRecordsPerBatch</span></code> to every input batch, including
batches with only plain columns. Before decoding dictionary-encoded shuffle columns, Comet also
compares their estimated decoded size with <code class="docutils literal notranslate"><span class="pre">spark.sql.execution.arrow.maxBytesPerBatch</span></code>.
diff --git a/user-guide/latest/tuning/operators.html b/user-guide/latest/tuning/operators.html
index 47fa11f95a..2611b96dab 100644
--- a/user-guide/latest/tuning/operators.html
+++ b/user-guide/latest/tuning/operators.html
@@ -915,15 +915,16 @@ See <a class="reference internal" href="../metrics.html#local-topk"><span class=
</section>
<section id="optimizing-sorting-on-floating-point-values">
<h2>Optimizing Sorting on Floating-Point Values<a class="headerlink" href="#optimizing-sorting-on-floating-point-values" title="Link to this heading">#</a></h2>
-<p>Comet normalizes NaN payloads and signed zeros in scalar <code class="docutils literal notranslate"><span class="pre">FLOAT</span></code> and <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> ordering keys, so <code class="docutils literal notranslate"><span class="pre">ORDER</span> <span class="pre">BY</span></code>, window
-ordering and range partitioning on them match Spark and stay native even with
-<code class="docutils literal notranslate"><span class="pre">spark.comet.exec.strictFloatingPoint=true</span></code>. Only the comparison key is normalized; returned values keep their original
-NaN representation and zero sign.</p>
-<p>Floating-point values nested in arrays, structs, or maps are compared with Arrow’s raw total ordering instead, which can
-differ from Spark when the data contains both zero and negative zero, or more than one NaN representation. This is likely
-an edge case that is not of concern for many users. Setting <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.strictFloatingPoint=true</span></code> makes those
-nested cases fall back to Spark, and they can be forced back onto the native path with
-<code class="docutils literal notranslate"><span class="pre">spark.comet.expression.SortOrder.allowIncompatible=true</span></code>.</p>
+<p>Comet normalizes NaN payloads and signed zeros in <code class="docutils literal notranslate"><span class="pre">FLOAT</span></code> and <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> ordering keys, including floating-point values
+nested in arrays and structs, so <code class="docutils literal notranslate"><span class="pre">ORDER</span> <span class="pre">BY</span></code>, window ordering and range partitioning on them match Spark and stay native
+even with <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.strictFloatingPoint=true</span></code>. Only the comparison key is normalized; returned values keep their
+original NaN representation and zero sign.</p>
+<p>The exception is a key that nests floating-point values in an array or struct whose type can hold a null element or
+field. Spark orders such a null below every other value, and the native sort and <code class="docutils literal notranslate"><span class="pre">RANGE</span></code> window frames do not
+(<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/6476">#6476</a>,
+<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/6477">#6477</a>), so
+<code class="docutils literal notranslate"><span class="pre">spark.comet.exec.strictFloatingPoint=true</span></code> makes those keys fall back to Spark. They can be forced back onto the
+native path with <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.SortOrder.allowIncompatible=true</span></code>.</p>
<p><code class="docutils literal notranslate"><span class="pre">sort_array</span></code> sorts array elements rather than ordering rows. It follows Spark’s floating-point ordering as well, so it
also stays native with <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.strictFloatingPoint=true</span></code>.</p>
</section>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org