You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/16 22:51:33 UTC
(datafusion-comet) 01/01: Publish built docs triggered by 46b9c6afe20ccb8f43ca747d87b18f408486b364
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit e244a69a909fe9d3f32cb463d11dc2f3fd48d368
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Fri Oct 16 22:51:25 2026 +0000
Publish built docs triggered by 46b9c6afe20ccb8f43ca747d87b18f408486b364
---
.../mermaid-09341aca03757d9d3637a2eaff73472d472d0673.svg | 2 +-
_sources/contributor-guide/adding_a_new_operator.md.txt | 13 +++++++++++--
.../contributor-guide/expression-audits/map_funcs.md.txt | 4 +++-
_sources/contributor-guide/floating_point.md.txt | 9 +++++----
_sources/contributor-guide/iceberg-writes.md.txt | 1 +
.../latest/compatibility/expressions/spark-3.4/map.md.txt | 4 +++-
.../latest/compatibility/expressions/spark-3.5/map.md.txt | 4 +++-
.../latest/compatibility/expressions/spark-4.0/map.md.txt | 4 +++-
.../latest/compatibility/expressions/spark-4.1/map.md.txt | 4 +++-
.../latest/compatibility/expressions/spark-4.2/map.md.txt | 4 +++-
.../user-guide/latest/compatibility/floating-point.md.txt | 8 ++++++++
_sources/user-guide/latest/compatibility/operators.md.txt | 8 ++++++++
_sources/user-guide/latest/configs.md.txt | 2 +-
_sources/user-guide/latest/expressions.md.txt | 4 ++--
_sources/user-guide/latest/operators.md.txt | 5 +++--
contributor-guide/adding_a_new_operator.html | 13 +++++++++++--
contributor-guide/expression-audits/map_funcs.html | 4 +++-
contributor-guide/floating_point.html | 9 +++++----
contributor-guide/iceberg-writes.html | 7 +++++--
searchindex.js | 2 +-
.../latest/compatibility/expressions/spark-3.4/map.html | 4 +++-
.../latest/compatibility/expressions/spark-3.5/map.html | 4 +++-
.../latest/compatibility/expressions/spark-4.0/map.html | 4 +++-
.../latest/compatibility/expressions/spark-4.1/map.html | 4 +++-
.../latest/compatibility/expressions/spark-4.2/map.html | 4 +++-
user-guide/latest/compatibility/floating-point.html | 9 +++++++++
user-guide/latest/compatibility/operators.html | 9 +++++++++
user-guide/latest/configs.html | 2 +-
user-guide/latest/expressions.html | 6 +++---
user-guide/latest/operators.html | 5 +++--
30 files changed, 123 insertions(+), 39 deletions(-)
diff --git a/_images/mermaid-09341aca03757d9d3637a2eaff73472d472d0673.svg b/_images/mermaid-09341aca03757d9d3637a2eaff73472d472d0673.svg
index 6b9a1dda9f..e03d2e5944 100644
--- a/_images/mermaid-09341aca03757d9d3637a2eaff73472d472d0673.svg
+++ b/_images/mermaid-09341aca03757d9d3637a2eaff73472d472d0673.svg
@@ -1 +1 @@
-<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 1014" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-sv [...]
\ No newline at end of file
+<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 1014" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-sv [...]
\ No newline at end of file
diff --git a/_sources/contributor-guide/adding_a_new_operator.md.txt b/_sources/contributor-guide/adding_a_new_operator.md.txt
index 85946f44fd..caf4519833 100644
--- a/_sources/contributor-guide/adding_a_new_operator.md.txt
+++ b/_sources/contributor-guide/adding_a_new_operator.md.txt
@@ -383,8 +383,17 @@ The examples above follow the implementations in
rather than serialization state; this method does not define equality.
Some operators use a different convention. `CometNativeScanExec` compares `originalPlan` to retain
-scan identity, and its `doCanonicalize` canonicalizes that plan while removing unused dynamic
-pruning filters. `CometBroadcastExchangeExec` also compares `originalPlan` before canonicalization,
+scan identity. Its `doCanonicalize` passes `originalPlan` to
+`CometScanUtils.canonicalizeOriginalPlan`, which strips every dynamic pruning filter from that copy
+before canonicalizing it, because the copy is stale and no DPP rule rewrites it. The scan's DPP
+identity comes from its top-level `partitionFilters` instead, where
+`CometScanUtils.filterUnusedDynamicPruningExpressions` drops only
+`DynamicPruningExpression(TrueLiteral)`, as Spark's `FileSourceScanExec` does. A new scan with DPP
+support should call both helpers (the second when it keeps a `FileSourceScanExec` as
+`originalPlan`): a filter that still holds the adaptive broadcast placeholder must
+stay in the canonical form, because AQE canonicalizes a query stage as its exchange was before that
+placeholder was converted, and dropping it lets AQE reuse one stage for scans with different
+filters. `CometBroadcastExchangeExec` also compares `originalPlan` before canonicalization,
but its `doCanonicalize` clears that reference and retains the canonicalized child. Follow each
operator's equality and canonicalization together rather than copying an exclusion in isolation.
diff --git a/_sources/contributor-guide/expression-audits/map_funcs.md.txt b/_sources/contributor-guide/expression-audits/map_funcs.md.txt
index f50475a449..d37a29df1b 100644
--- a/_sources/contributor-guide/expression-audits/map_funcs.md.txt
+++ b/_sources/contributor-guide/expression-audits/map_funcs.md.txt
@@ -46,8 +46,9 @@
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. `MapFromArrays(left, right) extends BinaryExpression with NullIntolerant`; Spark uses `ArrayBasedMapBuilder` to detect duplicate keys (subject to `spark.sql.mapKeyDedupPolicy`) and rejects null keys with `RuntimeException("Cannot use null as map key")`. Comet `CometMapFromArrays` emits nested `CaseWhen(IsNotNull(left), CaseWhen(IsNotNull(right), map(left, right), null), null)` guards so NULL-array inputs return NULL rather than triggering the [...]
-- Spark 4.0.1 (audited 2026-05-27): semantics unchanged; `NullIntolerant` trait replaced by `nullIntolerant: Boolean`.
+- Spark 4.0.1 (audited 2026-05-27): `NullIntolerant` trait replaced by `nullIntolerant: Boolean`. `ArrayBasedMapBuilder` gains the `keyNormalizer` described below.
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+- Floating-point keys (2026-10-14, [#6549](https://github.com/apache/datafusion-comet/issues/6549)): `ArrayBasedMapBuilder` finds a duplicate `FLOAT` or `DOUBLE` key in a `HashMap` of boxed values (`Double.equals`, so NaNs with different bits are one key) and a duplicate struct or array key in a `TreeMap` in the SQL ordering (`-0.0` equals `0.0`). From Spark 4.0 its `keyNormalizer` also normalizes a `FLOAT` or `DOUBLE` key before the lookup, unless `spark.sql.legacy.disableMapKeyNormaliz [...]
## map_from_entries
@@ -56,6 +57,7 @@
- Spark 4.0.1 (audited 2026-05-27): semantics unchanged; trait refactor.
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
- Known limitation: input arrays where the struct's key or value type contains `BinaryType` are marked `Incompatible` and fall back unless `spark.comet.expression.MapFromEntries.allowIncompatible=true`.
+- Floating-point keys (2026-10-14, [#6549](https://github.com/apache/datafusion-comet/issues/6549)): `ArrayBasedMapBuilder` finds a duplicate `FLOAT` or `DOUBLE` key in a `HashMap` of boxed values (`Double.equals`, so NaNs with different bits are one key) and a duplicate struct or array key in a `TreeMap` in the SQL ordering (`-0.0` equals `0.0`). From Spark 4.0 its `keyNormalizer` also normalizes a `FLOAT` or `DOUBLE` key before the lookup, unless `spark.sql.legacy.disableMapKeyNormaliz [...]
## map_keys
diff --git a/_sources/contributor-guide/floating_point.md.txt b/_sources/contributor-guide/floating_point.md.txt
index 165b280c63..758350db8f 100644
--- a/_sources/contributor-guide/floating_point.md.txt
+++ b/_sources/contributor-guide/floating_point.md.txt
@@ -62,10 +62,11 @@ it runs against:
(before 3.5.2, unless probing merges them). Spark 4.2.0 normalizes their arguments in the plan
(SPARK-54918). From 4.0.5, 4.1.4 and 4.2.1 they normalize while they evaluate instead
(SPARK-59602).
-- Map construction (`ArrayBasedMapBuilder`) finds duplicate keys by `Double.equals` in Spark 3.4
- and 3.5. From Spark 4.0 it normalizes each key first, unless
- `spark.sql.legacy.disableMapKeyNormalization` is set
- ([#6549](https://github.com/apache/datafusion-comet/issues/6549)).
+- Map construction (`ArrayBasedMapBuilder`) finds a duplicate `FLOAT` or `DOUBLE` key in a
+ `HashMap` of boxed values, which follows `Double.equals`, and a duplicate struct or array key in a
+ `TreeMap` in the SQL ordering. From Spark 4.0 it normalizes a `FLOAT` or `DOUBLE` key before the
+ lookup, unless `spark.sql.legacy.disableMapKeyNormalization` is set, and `map_from_entries`
+ stores the normalized key ([#6549](https://github.com/apache/datafusion-comet/issues/6549)).
## How Arrow and DataFusion differ
diff --git a/_sources/contributor-guide/iceberg-writes.md.txt b/_sources/contributor-guide/iceberg-writes.md.txt
index e9e8e8cddc..1922122265 100644
--- a/_sources/contributor-guide/iceberg-writes.md.txt
+++ b/_sources/contributor-guide/iceberg-writes.md.txt
@@ -449,6 +449,7 @@ the writer: run the write suites and the Iceberg Spark tests. The pin policy is
| `IcebergWriteProtoTranslationSuite` | Translation of properties into `IcebergParquetWriteSettings` and the writer mode. |
| Rust tests in `iceberg_write.rs` and in `iceberg_partition_*.rs` | File rolling on the 1000-row grid, fanout order, clustered input checks, cleanup guard, manifest round trip, memory reservation, partition path rendering, partition values past `chrono`'s calendar. |
| Rust tests in `iceberg_dictionary.rs` | The per-column dictionary choice against answers recorded from parquet-mr, including columns either side of its cut-off. |
+| `iceberg_dictionary_chooser` benchmark in `native/core/benches` | The time one partition's dictionary choice takes, over primitive, struct, array, map and nested columns. |
| `CometIcebergWriteBenchmark` | Native versus iceberg-java for unpartitioned, clustered, fanout and copy-on-write delete writes. It checks each arm's plan before timing it. |
| `CometIcebergEndToEndBenchmark` | An Iceberg read, hash repartition and Iceberg write against stock Spark, over primitive, struct, array, map and nested columns. It checks each arm's plan and the rows it wrote before timing it. |
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-3.4/map.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-3.4/map.md.txt
index 3f2b190fa7..6ccb94e784 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-3.4/map.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-3.4/map.md.txt
@@ -38,8 +38,9 @@ The following differences from Spark are always present and do not require any a
- Spark rejects a `NULL` element inside the keys array with a `RuntimeException` (`Cannot use null as map key`); Comet's native `map_from_arrays` / `map_from_entries` does not detect a per-element `NULL` key and produces a map with a `NULL` key instead ([#4680](https://github.com/apache/datafusion-comet/issues/4680)).
-The following incompatibilities cause `MapFromArrays` to fall back to Spark by default. Set `spark.comet.expression.MapFromArrays.allowIncompatible=true` to enable Comet acceleration despite these differences.
+By default, `MapFromArrays` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.MapFromArrays.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark:
+- Spark finds duplicate map keys that hold a `FLOAT` or `DOUBLE` by value, so NaNs with different bits are one key, and so are `-0.0` and `0.0` inside a struct or array key. From Spark 4.0, a `-0.0` key is also normalized to `0.0`, unless `spark.sql.legacy.disableMapKeyNormalization` is set. Comet's native map construction compares the keys' bits ([#6549](https://github.com/apache/datafusion-comet/issues/6549)).
- `spark.sql.mapKeyDedupPolicy` is set to `LAST_WIN`; Comet's native map construction does not implement LAST_WIN dedup semantics.
## MapFromEntries
@@ -52,6 +53,7 @@ By default, `MapFromEntries` is evaluated in the JVM using Spark's own code-gene
- `BinaryType` is not supported as a map key in `map_from_entries`
- `BinaryType` is not supported as a map value in `map_from_entries`
+- Spark finds duplicate map keys that hold a `FLOAT` or `DOUBLE` by value, so NaNs with different bits are one key, and so are `-0.0` and `0.0` inside a struct or array key. From Spark 4.0, a `-0.0` key is also normalized to `0.0`, unless `spark.sql.legacy.disableMapKeyNormalization` is set. Comet's native map construction compares the keys' bits ([#6549](https://github.com/apache/datafusion-comet/issues/6549)).
- `spark.sql.mapKeyDedupPolicy` is set to `LAST_WIN`; Comet's native map construction does not implement LAST_WIN dedup semantics.
## StringToMap
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-3.5/map.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-3.5/map.md.txt
index 3f2b190fa7..6ccb94e784 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-3.5/map.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-3.5/map.md.txt
@@ -38,8 +38,9 @@ The following differences from Spark are always present and do not require any a
- Spark rejects a `NULL` element inside the keys array with a `RuntimeException` (`Cannot use null as map key`); Comet's native `map_from_arrays` / `map_from_entries` does not detect a per-element `NULL` key and produces a map with a `NULL` key instead ([#4680](https://github.com/apache/datafusion-comet/issues/4680)).
-The following incompatibilities cause `MapFromArrays` to fall back to Spark by default. Set `spark.comet.expression.MapFromArrays.allowIncompatible=true` to enable Comet acceleration despite these differences.
+By default, `MapFromArrays` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.MapFromArrays.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark:
+- Spark finds duplicate map keys that hold a `FLOAT` or `DOUBLE` by value, so NaNs with different bits are one key, and so are `-0.0` and `0.0` inside a struct or array key. From Spark 4.0, a `-0.0` key is also normalized to `0.0`, unless `spark.sql.legacy.disableMapKeyNormalization` is set. Comet's native map construction compares the keys' bits ([#6549](https://github.com/apache/datafusion-comet/issues/6549)).
- `spark.sql.mapKeyDedupPolicy` is set to `LAST_WIN`; Comet's native map construction does not implement LAST_WIN dedup semantics.
## MapFromEntries
@@ -52,6 +53,7 @@ By default, `MapFromEntries` is evaluated in the JVM using Spark's own code-gene
- `BinaryType` is not supported as a map key in `map_from_entries`
- `BinaryType` is not supported as a map value in `map_from_entries`
+- Spark finds duplicate map keys that hold a `FLOAT` or `DOUBLE` by value, so NaNs with different bits are one key, and so are `-0.0` and `0.0` inside a struct or array key. From Spark 4.0, a `-0.0` key is also normalized to `0.0`, unless `spark.sql.legacy.disableMapKeyNormalization` is set. Comet's native map construction compares the keys' bits ([#6549](https://github.com/apache/datafusion-comet/issues/6549)).
- `spark.sql.mapKeyDedupPolicy` is set to `LAST_WIN`; Comet's native map construction does not implement LAST_WIN dedup semantics.
## StringToMap
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-4.0/map.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-4.0/map.md.txt
index 98a501a42a..1fec37f806 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-4.0/map.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-4.0/map.md.txt
@@ -38,8 +38,9 @@ The following differences from Spark are always present and do not require any a
- Spark rejects a `NULL` element inside the keys array with a `RuntimeException` (`Cannot use null as map key`); Comet's native `map_from_arrays` / `map_from_entries` does not detect a per-element `NULL` key and produces a map with a `NULL` key instead ([#4680](https://github.com/apache/datafusion-comet/issues/4680)).
-The following incompatibilities cause `MapFromArrays` to fall back to Spark by default. Set `spark.comet.expression.MapFromArrays.allowIncompatible=true` to enable Comet acceleration despite these differences.
+By default, `MapFromArrays` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.MapFromArrays.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark:
+- Spark finds duplicate map keys that hold a `FLOAT` or `DOUBLE` by value, so NaNs with different bits are one key, and so are `-0.0` and `0.0` inside a struct or array key. From Spark 4.0, a `-0.0` key is also normalized to `0.0`, unless `spark.sql.legacy.disableMapKeyNormalization` is set. Comet's native map construction compares the keys' bits ([#6549](https://github.com/apache/datafusion-comet/issues/6549)).
- `spark.sql.mapKeyDedupPolicy` is set to `LAST_WIN`; Comet's native map construction does not implement LAST_WIN dedup semantics.
## MapFromEntries
@@ -52,6 +53,7 @@ By default, `MapFromEntries` is evaluated in the JVM using Spark's own code-gene
- `BinaryType` is not supported as a map key in `map_from_entries`
- `BinaryType` is not supported as a map value in `map_from_entries`
+- Spark finds duplicate map keys that hold a `FLOAT` or `DOUBLE` by value, so NaNs with different bits are one key, and so are `-0.0` and `0.0` inside a struct or array key. From Spark 4.0, a `-0.0` key is also normalized to `0.0`, unless `spark.sql.legacy.disableMapKeyNormalization` is set. Comet's native map construction compares the keys' bits ([#6549](https://github.com/apache/datafusion-comet/issues/6549)).
- `spark.sql.mapKeyDedupPolicy` is set to `LAST_WIN`; Comet's native map construction does not implement LAST_WIN dedup semantics.
## MapSort
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-4.1/map.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-4.1/map.md.txt
index 98a501a42a..1fec37f806 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-4.1/map.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-4.1/map.md.txt
@@ -38,8 +38,9 @@ The following differences from Spark are always present and do not require any a
- Spark rejects a `NULL` element inside the keys array with a `RuntimeException` (`Cannot use null as map key`); Comet's native `map_from_arrays` / `map_from_entries` does not detect a per-element `NULL` key and produces a map with a `NULL` key instead ([#4680](https://github.com/apache/datafusion-comet/issues/4680)).
-The following incompatibilities cause `MapFromArrays` to fall back to Spark by default. Set `spark.comet.expression.MapFromArrays.allowIncompatible=true` to enable Comet acceleration despite these differences.
+By default, `MapFromArrays` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.MapFromArrays.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark:
+- Spark finds duplicate map keys that hold a `FLOAT` or `DOUBLE` by value, so NaNs with different bits are one key, and so are `-0.0` and `0.0` inside a struct or array key. From Spark 4.0, a `-0.0` key is also normalized to `0.0`, unless `spark.sql.legacy.disableMapKeyNormalization` is set. Comet's native map construction compares the keys' bits ([#6549](https://github.com/apache/datafusion-comet/issues/6549)).
- `spark.sql.mapKeyDedupPolicy` is set to `LAST_WIN`; Comet's native map construction does not implement LAST_WIN dedup semantics.
## MapFromEntries
@@ -52,6 +53,7 @@ By default, `MapFromEntries` is evaluated in the JVM using Spark's own code-gene
- `BinaryType` is not supported as a map key in `map_from_entries`
- `BinaryType` is not supported as a map value in `map_from_entries`
+- Spark finds duplicate map keys that hold a `FLOAT` or `DOUBLE` by value, so NaNs with different bits are one key, and so are `-0.0` and `0.0` inside a struct or array key. From Spark 4.0, a `-0.0` key is also normalized to `0.0`, unless `spark.sql.legacy.disableMapKeyNormalization` is set. Comet's native map construction compares the keys' bits ([#6549](https://github.com/apache/datafusion-comet/issues/6549)).
- `spark.sql.mapKeyDedupPolicy` is set to `LAST_WIN`; Comet's native map construction does not implement LAST_WIN dedup semantics.
## MapSort
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-4.2/map.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-4.2/map.md.txt
index 98a501a42a..1fec37f806 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-4.2/map.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-4.2/map.md.txt
@@ -38,8 +38,9 @@ The following differences from Spark are always present and do not require any a
- Spark rejects a `NULL` element inside the keys array with a `RuntimeException` (`Cannot use null as map key`); Comet's native `map_from_arrays` / `map_from_entries` does not detect a per-element `NULL` key and produces a map with a `NULL` key instead ([#4680](https://github.com/apache/datafusion-comet/issues/4680)).
-The following incompatibilities cause `MapFromArrays` to fall back to Spark by default. Set `spark.comet.expression.MapFromArrays.allowIncompatible=true` to enable Comet acceleration despite these differences.
+By default, `MapFromArrays` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.MapFromArrays.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark:
+- Spark finds duplicate map keys that hold a `FLOAT` or `DOUBLE` by value, so NaNs with different bits are one key, and so are `-0.0` and `0.0` inside a struct or array key. From Spark 4.0, a `-0.0` key is also normalized to `0.0`, unless `spark.sql.legacy.disableMapKeyNormalization` is set. Comet's native map construction compares the keys' bits ([#6549](https://github.com/apache/datafusion-comet/issues/6549)).
- `spark.sql.mapKeyDedupPolicy` is set to `LAST_WIN`; Comet's native map construction does not implement LAST_WIN dedup semantics.
## MapFromEntries
@@ -52,6 +53,7 @@ By default, `MapFromEntries` is evaluated in the JVM using Spark's own code-gene
- `BinaryType` is not supported as a map key in `map_from_entries`
- `BinaryType` is not supported as a map value in `map_from_entries`
+- Spark finds duplicate map keys that hold a `FLOAT` or `DOUBLE` by value, so NaNs with different bits are one key, and so are `-0.0` and `0.0` inside a struct or array key. From Spark 4.0, a `-0.0` key is also normalized to `0.0`, unless `spark.sql.legacy.disableMapKeyNormalization` is set. Comet's native map construction compares the keys' bits ([#6549](https://github.com/apache/datafusion-comet/issues/6549)).
- `spark.sql.mapKeyDedupPolicy` is set to `LAST_WIN`; Comet's native map construction does not implement LAST_WIN dedup semantics.
## MapSort
diff --git a/_sources/user-guide/latest/compatibility/floating-point.md.txt b/_sources/user-guide/latest/compatibility/floating-point.md.txt
index f00ceceaf6..77d76f53c6 100644
--- a/_sources/user-guide/latest/compatibility/floating-point.md.txt
+++ b/_sources/user-guide/latest/compatibility/floating-point.md.txt
@@ -164,3 +164,11 @@ representations tie, and keeps equal elements in their original order. In an arr
can be null, such as one built from nullable columns, `-0.0` and `0.0` therefore tie. Sorting an
array whose elements cannot be null in ascending order, Spark's generated code puts `-0.0` before
`0.0`, and so does Comet. All three expressions run natively in strict floating-point mode.
+
+## Map construction
+
+`map_from_arrays` and `map_from_entries` don't build a map natively when its key type holds a
+`FLOAT` or `DOUBLE` at any depth. They run Spark's code through the codegen dispatcher instead, so
+duplicate keys, including NaNs and zeros of either sign, behave as in Spark. `FLOAT` and `DOUBLE`
+values don't affect either function. The expressions' entries in the
+[expression compatibility guide](expressions/map.md) give the reason.
diff --git a/_sources/user-guide/latest/compatibility/operators.md.txt b/_sources/user-guide/latest/compatibility/operators.md.txt
index 1b9a61bf7f..8312fb520a 100644
--- a/_sources/user-guide/latest/compatibility/operators.md.txt
+++ b/_sources/user-guide/latest/compatibility/operators.md.txt
@@ -109,6 +109,14 @@ rows by the grouping keys only, and Spark and Comet can leave rows with equal ke
orders, so a group with more than one candidate value can return a different value than Spark.
Both results are valid under Spark's semantics for these functions.
+## Hash Join
+
+Spark plans `NOT IN` as a null-aware anti join, and the native hash join builds it on Spark's
+streamed side without keeping that side's order. When Spark reports the streamed side as ordered,
+for example the output of a window, a sort-merge join, or a cached table sorted within its
+partitions, Comet falls back to Spark for the join, so that a sort Spark removed above it on the
+strength of that order is not lost.
+
## Window Functions
Comet runs `WindowExec` natively and it is enabled by default (`spark.comet.exec.window.enabled`). A broad set of
diff --git a/_sources/user-guide/latest/configs.md.txt b/_sources/user-guide/latest/configs.md.txt
index cf23f66dc7..4e0ecf2752 100644
--- a/_sources/user-guide/latest/configs.md.txt
+++ b/_sources/user-guide/latest/configs.md.txt
@@ -210,7 +210,7 @@ them like any other production setting.
| `spark.comet.exec.coalesce.enabled` | Whether to enable coalesce by default. | true |
| `spark.comet.exec.collectLimit.enabled` | Whether to enable collectLimit by default. | true |
| `spark.comet.exec.emptyRelation.enabled` | Whether to enable emptyRelation by default. | true |
-| `spark.comet.exec.existenceJoin.enabled` | Whether to enable native ExistenceJoin, produced when EXISTS / NOT EXISTS / IN is combined with another predicate via OR. Sort-merge joins, residual (non-equi) join conditions, computed (non-column) join keys, and NOT IN fall back to Spark. | true |
+| `spark.comet.exec.existenceJoin.enabled` | Whether to enable native ExistenceJoin, produced when EXISTS / NOT EXISTS / IN is combined with another predicate via OR. Sort-merge joins, residual (non-equi) join conditions, computed (non-column) join keys, and NOT IN combined with OR fall back to Spark. | true |
| `spark.comet.exec.expand.enabled` | Whether to enable expand by default. | true |
| `spark.comet.exec.explode.enabled` | Whether to enable explode by default. | true |
| `spark.comet.exec.filter.enabled` | Whether to enable filter by default. | true |
diff --git a/_sources/user-guide/latest/expressions.md.txt b/_sources/user-guide/latest/expressions.md.txt
index a0ce771b8e..7e59061fb9 100644
--- a/_sources/user-guide/latest/expressions.md.txt
+++ b/_sources/user-guide/latest/expressions.md.txt
@@ -408,8 +408,8 @@ to Spark ([#2837](https://github.com/apache/datafusion-comet/issues/2837)). Enab
| `map_concat` | ✅ | Codegen dispatch | |
| `map_contains_key` | ✅ | — | |
| `map_entries` | ✅ | Native | |
-| `map_from_arrays` | ✅ | Native | |
-| `map_from_entries` | ✅ | Hybrid | BinaryType keys/values and `spark.sql.mapKeyDedupPolicy=LAST_WIN` route through the JVM codegen dispatcher ([details](compatibility/expressions/map.md)) |
+| `map_from_arrays` | ✅ | Hybrid | `FLOAT` or `DOUBLE` keys and `spark.sql.mapKeyDedupPolicy=LAST_WIN` route through the JVM codegen dispatcher ([details](compatibility/expressions/map.md)) |
+| `map_from_entries` | ✅ | Hybrid | BinaryType keys/values, `FLOAT` or `DOUBLE` keys and `spark.sql.mapKeyDedupPolicy=LAST_WIN` route through the JVM codegen dispatcher ([details](compatibility/expressions/map.md)) |
| `map_keys` | ✅ | Native | |
| `map_values` | ✅ | Native | |
| `str_to_map` | ✅ | Hybrid | |
diff --git a/_sources/user-guide/latest/operators.md.txt b/_sources/user-guide/latest/operators.md.txt
index aa4fd58967..566ec058f1 100644
--- a/_sources/user-guide/latest/operators.md.txt
+++ b/_sources/user-guide/latest/operators.md.txt
@@ -98,14 +98,15 @@ omitted from the tables below and may be reconsidered based on demand:
| Operator | Status | Notes |
| ----------------------------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `BroadcastHashJoinExec` | ✅ | |
+| `BroadcastHashJoinExec` | ✅ | Falls back to Spark for a `NOT IN` (null-aware anti join) whose streamed side reports an ordering. |
| `ShuffledHashJoinExec` | ✅ | |
| `SortMergeJoinExec` | ✅ | Supports `BINARY` join keys. Nested-type (struct, array, map) and collated-string join keys fall back to Spark. |
| `BroadcastNestedLoopJoinExec` | ✅ | Falls back to Spark when the preserved side is broadcast (for example LEFT OUTER with BROADCAST on the left) ([#4429](https://github.com/apache/datafusion-comet/pull/4429)). |
`ExistenceJoin` (produced when `EXISTS` / `IN` is combined with another predicate via `OR`) runs
natively on `BroadcastHashJoinExec` and `ShuffledHashJoinExec`. Existence sort-merge joins, residual
-(non-equi) join conditions, computed (non-column) join keys, and `NOT IN` fall back to Spark.
+(non-equi) join conditions, and computed (non-column) join keys fall back to Spark. So does `NOT IN`
+combined with `OR`, which Spark plans as an existence `BroadcastNestedLoopJoinExec`.
## Exchanges
diff --git a/contributor-guide/adding_a_new_operator.html b/contributor-guide/adding_a_new_operator.html
index e89edcdc07..2ab8dd5646 100644
--- a/contributor-guide/adding_a_new_operator.html
+++ b/contributor-guide/adding_a_new_operator.html
@@ -897,8 +897,17 @@ plan. The bytes therefore do not distinguish canonicalized plans.</p></li>
rather than serialization state; this method does not define equality.</p></li>
</ul>
<p>Some operators use a different convention. <code class="docutils literal notranslate"><span class="pre">CometNativeScanExec</span></code> compares <code class="docutils literal notranslate"><span class="pre">originalPlan</span></code> to retain
-scan identity, and its <code class="docutils literal notranslate"><span class="pre">doCanonicalize</span></code> canonicalizes that plan while removing unused dynamic
-pruning filters. <code class="docutils literal notranslate"><span class="pre">CometBroadcastExchangeExec</span></code> also compares <code class="docutils literal notranslate"><span class="pre">originalPlan</span></code> before canonicalization,
+scan identity. Its <code class="docutils literal notranslate"><span class="pre">doCanonicalize</span></code> passes <code class="docutils literal notranslate"><span class="pre">originalPlan</span></code> to
+<code class="docutils literal notranslate"><span class="pre">CometScanUtils.canonicalizeOriginalPlan</span></code>, which strips every dynamic pruning filter from that copy
+before canonicalizing it, because the copy is stale and no DPP rule rewrites it. The scan’s DPP
+identity comes from its top-level <code class="docutils literal notranslate"><span class="pre">partitionFilters</span></code> instead, where
+<code class="docutils literal notranslate"><span class="pre">CometScanUtils.filterUnusedDynamicPruningExpressions</span></code> drops only
+<code class="docutils literal notranslate"><span class="pre">DynamicPruningExpression(TrueLiteral)</span></code>, as Spark’s <code class="docutils literal notranslate"><span class="pre">FileSourceScanExec</span></code> does. A new scan with DPP
+support should call both helpers (the second when it keeps a <code class="docutils literal notranslate"><span class="pre">FileSourceScanExec</span></code> as
+<code class="docutils literal notranslate"><span class="pre">originalPlan</span></code>): a filter that still holds the adaptive broadcast placeholder must
+stay in the canonical form, because AQE canonicalizes a query stage as its exchange was before that
+placeholder was converted, and dropping it lets AQE reuse one stage for scans with different
+filters. <code class="docutils literal notranslate"><span class="pre">CometBroadcastExchangeExec</span></code> also compares <code class="docutils literal notranslate"><span class="pre">originalPlan</span></code> before canonicalization,
but its <code class="docutils literal notranslate"><span class="pre">doCanonicalize</span></code> clears that reference and retains the canonicalized child. Follow each
operator’s equality and canonicalization together rather than copying an exclusion in isolation.</p>
</section>
diff --git a/contributor-guide/expression-audits/map_funcs.html b/contributor-guide/expression-audits/map_funcs.html
index ab5cb7c36a..1b12ca5ff8 100644
--- a/contributor-guide/expression-audits/map_funcs.html
+++ b/contributor-guide/expression-audits/map_funcs.html
@@ -590,8 +590,9 @@ under the License.
<ul class="simple">
<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">MapFromArrays(left,</span> <span class="pre">right)</span> <span class="pre">extends</span> <span class="pre">BinaryExpression</span> <span class="pre">with</span> <span class="pre">NullIntolerant</span></code>; Spark uses <code class="docutils literal notranslate"><span class="pre">ArrayBasedMapBuilder</span></code> to detect duplicate keys (subject to <code class="docutils li [...]
-<li><p>Spark 4.0.1 (audited 2026-05-27): semantics unchanged; <code class="docutils literal notranslate"><span class="pre">NullIntolerant</span></code> trait replaced by <code class="docutils literal notranslate"><span class="pre">nullIntolerant:</span> <span class="pre">Boolean</span></code>.</p></li>
+<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">NullIntolerant</span></code> trait replaced by <code class="docutils literal notranslate"><span class="pre">nullIntolerant:</span> <span class="pre">Boolean</span></code>. <code class="docutils literal notranslate"><span class="pre">ArrayBasedMapBuilder</span></code> gains the <code class="docutils literal notranslate"><span class="pre">keyNormalizer</span></code> described below.</p></li>
<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+<li><p>Floating-point keys (2026-10-14, <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/6549">#6549</a>): <code class="docutils literal notranslate"><span class="pre">ArrayBasedMapBuilder</span></code> finds a duplicate <code class="docutils literal notranslate"><span class="pre">FLOAT</span></code> or <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> key in a <code class="docutils literal notranslate"><span class="pre [...]
</ul>
</section>
<section id="map-from-entries">
@@ -602,6 +603,7 @@ under the License.
<li><p>Spark 4.0.1 (audited 2026-05-27): semantics unchanged; trait refactor.</p></li>
<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
<li><p>Known limitation: input arrays where the struct’s key or value type contains <code class="docutils literal notranslate"><span class="pre">BinaryType</span></code> are marked <code class="docutils literal notranslate"><span class="pre">Incompatible</span></code> and fall back unless <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.MapFromEntries.allowIncompatible=true</span></code>.</p></li>
+<li><p>Floating-point keys (2026-10-14, <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/6549">#6549</a>): <code class="docutils literal notranslate"><span class="pre">ArrayBasedMapBuilder</span></code> finds a duplicate <code class="docutils literal notranslate"><span class="pre">FLOAT</span></code> or <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> key in a <code class="docutils literal notranslate"><span class="pre [...]
</ul>
</section>
<section id="map-keys">
diff --git a/contributor-guide/floating_point.html b/contributor-guide/floating_point.html
index f50ec4e1d5..e3389474c8 100644
--- a/contributor-guide/floating_point.html
+++ b/contributor-guide/floating_point.html
@@ -626,10 +626,11 @@ into <code class="docutils literal notranslate"><span class="pre">0.0</span></co
(before 3.5.2, unless probing merges them). Spark 4.2.0 normalizes their arguments in the plan
(SPARK-54918). From 4.0.5, 4.1.4 and 4.2.1 they normalize while they evaluate instead
(SPARK-59602).</p></li>
-<li><p>Map construction (<code class="docutils literal notranslate"><span class="pre">ArrayBasedMapBuilder</span></code>) finds duplicate keys by <code class="docutils literal notranslate"><span class="pre">Double.equals</span></code> in Spark 3.4
-and 3.5. From Spark 4.0 it normalizes each key first, unless
-<code class="docutils literal notranslate"><span class="pre">spark.sql.legacy.disableMapKeyNormalization</span></code> is set
-(<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/6549">#6549</a>).</p></li>
+<li><p>Map construction (<code class="docutils literal notranslate"><span class="pre">ArrayBasedMapBuilder</span></code>) finds a duplicate <code class="docutils literal notranslate"><span class="pre">FLOAT</span></code> or <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> key in a
+<code class="docutils literal notranslate"><span class="pre">HashMap</span></code> of boxed values, which follows <code class="docutils literal notranslate"><span class="pre">Double.equals</span></code>, and a duplicate struct or array key in a
+<code class="docutils literal notranslate"><span class="pre">TreeMap</span></code> in the SQL ordering. From Spark 4.0 it normalizes a <code class="docutils literal notranslate"><span class="pre">FLOAT</span></code> or <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> key before the
+lookup, unless <code class="docutils literal notranslate"><span class="pre">spark.sql.legacy.disableMapKeyNormalization</span></code> is set, and <code class="docutils literal notranslate"><span class="pre">map_from_entries</span></code>
+stores the normalized key (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/6549">#6549</a>).</p></li>
</ul>
</section>
<section id="how-arrow-and-datafusion-differ">
diff --git a/contributor-guide/iceberg-writes.html b/contributor-guide/iceberg-writes.html
index 405318e453..833f62a946 100644
--- a/contributor-guide/iceberg-writes.html
+++ b/contributor-guide/iceberg-writes.html
@@ -1040,10 +1040,13 @@ the writer: run the write suites and the Iceberg Spark tests. The pin policy is
<tr class="row-even"><td><p>Rust tests in <code class="docutils literal notranslate"><span class="pre">iceberg_dictionary.rs</span></code></p></td>
<td><p>The per-column dictionary choice against answers recorded from parquet-mr, including columns either side of its cut-off.</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">CometIcebergWriteBenchmark</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">iceberg_dictionary_chooser</span></code> benchmark in <code class="docutils literal notranslate"><span class="pre">native/core/benches</span></code></p></td>
+<td><p>The time one partition’s dictionary choice takes, over primitive, struct, array, map and nested columns.</p></td>
+</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">CometIcebergWriteBenchmark</span></code></p></td>
<td><p>Native versus iceberg-java for unpartitioned, clustered, fanout and copy-on-write delete writes. It checks each arm’s plan before timing it.</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">CometIcebergEndToEndBenchmark</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">CometIcebergEndToEndBenchmark</span></code></p></td>
<td><p>An Iceberg read, hash repartition and Iceberg write against stock Spark, over primitive, struct, array, map and nested columns. It checks each arm’s plan and the rows it wrote before timing it.</p></td>
</tr>
</tbody>
diff --git a/searchindex.js b/searchindex.js
index 4bda261d07..9ee778a6fa 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[60, "id1"]], "%": [[58, "id1"]], "&": [[48, "id1"]], "*": [[58, "id2"]], "+": [[58, "id3"]], "-": [[58, "id4"]], "/": [[58, "id5"]], "1. Format Your Code": [[45, "format-your-code"]], "1. Install Comet": [[67, "install-comet"], [79, "install-comet"]], "1. Native Operators (nativeExecs map)": [[31, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[45, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[67, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[60, "id1"]], "%": [[58, "id1"]], "&": [[48, "id1"]], "*": [[58, "id2"]], "+": [[58, "id3"]], "-": [[58, "id4"]], "/": [[58, "id5"]], "1. Format Your Code": [[45, "format-your-code"]], "1. Install Comet": [[67, "install-comet"], [79, "install-comet"]], "1. Native Operators (nativeExecs map)": [[31, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[45, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[67, "clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/compatibility/expressions/spark-3.4/map.html b/user-guide/latest/compatibility/expressions/spark-3.4/map.html
index fa30cf2844..289d85daab 100644
--- a/user-guide/latest/compatibility/expressions/spark-3.4/map.html
+++ b/user-guide/latest/compatibility/expressions/spark-3.4/map.html
@@ -869,8 +869,9 @@ IEEE total ordering for floating-point, which differs from Spark’s <code class
<ul class="simple">
<li><p>Spark rejects a <code class="docutils literal notranslate"><span class="pre">NULL</span></code> element inside the keys array with a <code class="docutils literal notranslate"><span class="pre">RuntimeException</span></code> (<code class="docutils literal notranslate"><span class="pre">Cannot</span> <span class="pre">use</span> <span class="pre">null</span> <span class="pre">as</span> <span class="pre">map</span> <span class="pre">key</span></code>); Comet’s native <code class="do [...]
</ul>
-<p>The following incompatibilities cause <code class="docutils literal notranslate"><span class="pre">MapFromArrays</span></code> to fall back to Spark by default. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.MapFromArrays.allowIncompatible=true</span></code> to enable Comet acceleration despite these differences.</p>
+<p>By default, <code class="docutils literal notranslate"><span class="pre">MapFromArrays</span></code> is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.MapFromArrays.allowIncompatible=true</span></code> to opt into Comet’s native implementation instead, which has the following differences from Spark:</p>
<ul class="simple">
+<li><p>Spark finds duplicate map keys that hold a <code class="docutils literal notranslate"><span class="pre">FLOAT</span></code> or <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> by value, so NaNs with different bits are one key, and so are <code class="docutils literal notranslate"><span class="pre">-0.0</span></code> and <code class="docutils literal notranslate"><span class="pre">0.0</span></code> inside a struct or array key. From Spark 4.0, a <co [...]
<li><p><code class="docutils literal notranslate"><span class="pre">spark.sql.mapKeyDedupPolicy</span></code> is set to <code class="docutils literal notranslate"><span class="pre">LAST_WIN</span></code>; Comet’s native map construction does not implement LAST_WIN dedup semantics.</p></li>
</ul>
</section>
@@ -884,6 +885,7 @@ IEEE total ordering for floating-point, which differs from Spark’s <code class
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">BinaryType</span></code> is not supported as a map key in <code class="docutils literal notranslate"><span class="pre">map_from_entries</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">BinaryType</span></code> is not supported as a map value in <code class="docutils literal notranslate"><span class="pre">map_from_entries</span></code></p></li>
+<li><p>Spark finds duplicate map keys that hold a <code class="docutils literal notranslate"><span class="pre">FLOAT</span></code> or <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> by value, so NaNs with different bits are one key, and so are <code class="docutils literal notranslate"><span class="pre">-0.0</span></code> and <code class="docutils literal notranslate"><span class="pre">0.0</span></code> inside a struct or array key. From Spark 4.0, a <co [...]
<li><p><code class="docutils literal notranslate"><span class="pre">spark.sql.mapKeyDedupPolicy</span></code> is set to <code class="docutils literal notranslate"><span class="pre">LAST_WIN</span></code>; Comet’s native map construction does not implement LAST_WIN dedup semantics.</p></li>
</ul>
</section>
diff --git a/user-guide/latest/compatibility/expressions/spark-3.5/map.html b/user-guide/latest/compatibility/expressions/spark-3.5/map.html
index 0a193b366b..54d29724f1 100644
--- a/user-guide/latest/compatibility/expressions/spark-3.5/map.html
+++ b/user-guide/latest/compatibility/expressions/spark-3.5/map.html
@@ -869,8 +869,9 @@ IEEE total ordering for floating-point, which differs from Spark’s <code class
<ul class="simple">
<li><p>Spark rejects a <code class="docutils literal notranslate"><span class="pre">NULL</span></code> element inside the keys array with a <code class="docutils literal notranslate"><span class="pre">RuntimeException</span></code> (<code class="docutils literal notranslate"><span class="pre">Cannot</span> <span class="pre">use</span> <span class="pre">null</span> <span class="pre">as</span> <span class="pre">map</span> <span class="pre">key</span></code>); Comet’s native <code class="do [...]
</ul>
-<p>The following incompatibilities cause <code class="docutils literal notranslate"><span class="pre">MapFromArrays</span></code> to fall back to Spark by default. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.MapFromArrays.allowIncompatible=true</span></code> to enable Comet acceleration despite these differences.</p>
+<p>By default, <code class="docutils literal notranslate"><span class="pre">MapFromArrays</span></code> is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.MapFromArrays.allowIncompatible=true</span></code> to opt into Comet’s native implementation instead, which has the following differences from Spark:</p>
<ul class="simple">
+<li><p>Spark finds duplicate map keys that hold a <code class="docutils literal notranslate"><span class="pre">FLOAT</span></code> or <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> by value, so NaNs with different bits are one key, and so are <code class="docutils literal notranslate"><span class="pre">-0.0</span></code> and <code class="docutils literal notranslate"><span class="pre">0.0</span></code> inside a struct or array key. From Spark 4.0, a <co [...]
<li><p><code class="docutils literal notranslate"><span class="pre">spark.sql.mapKeyDedupPolicy</span></code> is set to <code class="docutils literal notranslate"><span class="pre">LAST_WIN</span></code>; Comet’s native map construction does not implement LAST_WIN dedup semantics.</p></li>
</ul>
</section>
@@ -884,6 +885,7 @@ IEEE total ordering for floating-point, which differs from Spark’s <code class
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">BinaryType</span></code> is not supported as a map key in <code class="docutils literal notranslate"><span class="pre">map_from_entries</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">BinaryType</span></code> is not supported as a map value in <code class="docutils literal notranslate"><span class="pre">map_from_entries</span></code></p></li>
+<li><p>Spark finds duplicate map keys that hold a <code class="docutils literal notranslate"><span class="pre">FLOAT</span></code> or <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> by value, so NaNs with different bits are one key, and so are <code class="docutils literal notranslate"><span class="pre">-0.0</span></code> and <code class="docutils literal notranslate"><span class="pre">0.0</span></code> inside a struct or array key. From Spark 4.0, a <co [...]
<li><p><code class="docutils literal notranslate"><span class="pre">spark.sql.mapKeyDedupPolicy</span></code> is set to <code class="docutils literal notranslate"><span class="pre">LAST_WIN</span></code>; Comet’s native map construction does not implement LAST_WIN dedup semantics.</p></li>
</ul>
</section>
diff --git a/user-guide/latest/compatibility/expressions/spark-4.0/map.html b/user-guide/latest/compatibility/expressions/spark-4.0/map.html
index 81e119c8e7..407040ed63 100644
--- a/user-guide/latest/compatibility/expressions/spark-4.0/map.html
+++ b/user-guide/latest/compatibility/expressions/spark-4.0/map.html
@@ -869,8 +869,9 @@ IEEE total ordering for floating-point, which differs from Spark’s <code class
<ul class="simple">
<li><p>Spark rejects a <code class="docutils literal notranslate"><span class="pre">NULL</span></code> element inside the keys array with a <code class="docutils literal notranslate"><span class="pre">RuntimeException</span></code> (<code class="docutils literal notranslate"><span class="pre">Cannot</span> <span class="pre">use</span> <span class="pre">null</span> <span class="pre">as</span> <span class="pre">map</span> <span class="pre">key</span></code>); Comet’s native <code class="do [...]
</ul>
-<p>The following incompatibilities cause <code class="docutils literal notranslate"><span class="pre">MapFromArrays</span></code> to fall back to Spark by default. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.MapFromArrays.allowIncompatible=true</span></code> to enable Comet acceleration despite these differences.</p>
+<p>By default, <code class="docutils literal notranslate"><span class="pre">MapFromArrays</span></code> is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.MapFromArrays.allowIncompatible=true</span></code> to opt into Comet’s native implementation instead, which has the following differences from Spark:</p>
<ul class="simple">
+<li><p>Spark finds duplicate map keys that hold a <code class="docutils literal notranslate"><span class="pre">FLOAT</span></code> or <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> by value, so NaNs with different bits are one key, and so are <code class="docutils literal notranslate"><span class="pre">-0.0</span></code> and <code class="docutils literal notranslate"><span class="pre">0.0</span></code> inside a struct or array key. From Spark 4.0, a <co [...]
<li><p><code class="docutils literal notranslate"><span class="pre">spark.sql.mapKeyDedupPolicy</span></code> is set to <code class="docutils literal notranslate"><span class="pre">LAST_WIN</span></code>; Comet’s native map construction does not implement LAST_WIN dedup semantics.</p></li>
</ul>
</section>
@@ -884,6 +885,7 @@ IEEE total ordering for floating-point, which differs from Spark’s <code class
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">BinaryType</span></code> is not supported as a map key in <code class="docutils literal notranslate"><span class="pre">map_from_entries</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">BinaryType</span></code> is not supported as a map value in <code class="docutils literal notranslate"><span class="pre">map_from_entries</span></code></p></li>
+<li><p>Spark finds duplicate map keys that hold a <code class="docutils literal notranslate"><span class="pre">FLOAT</span></code> or <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> by value, so NaNs with different bits are one key, and so are <code class="docutils literal notranslate"><span class="pre">-0.0</span></code> and <code class="docutils literal notranslate"><span class="pre">0.0</span></code> inside a struct or array key. From Spark 4.0, a <co [...]
<li><p><code class="docutils literal notranslate"><span class="pre">spark.sql.mapKeyDedupPolicy</span></code> is set to <code class="docutils literal notranslate"><span class="pre">LAST_WIN</span></code>; Comet’s native map construction does not implement LAST_WIN dedup semantics.</p></li>
</ul>
</section>
diff --git a/user-guide/latest/compatibility/expressions/spark-4.1/map.html b/user-guide/latest/compatibility/expressions/spark-4.1/map.html
index c052cede0a..ffcf787795 100644
--- a/user-guide/latest/compatibility/expressions/spark-4.1/map.html
+++ b/user-guide/latest/compatibility/expressions/spark-4.1/map.html
@@ -869,8 +869,9 @@ IEEE total ordering for floating-point, which differs from Spark’s <code class
<ul class="simple">
<li><p>Spark rejects a <code class="docutils literal notranslate"><span class="pre">NULL</span></code> element inside the keys array with a <code class="docutils literal notranslate"><span class="pre">RuntimeException</span></code> (<code class="docutils literal notranslate"><span class="pre">Cannot</span> <span class="pre">use</span> <span class="pre">null</span> <span class="pre">as</span> <span class="pre">map</span> <span class="pre">key</span></code>); Comet’s native <code class="do [...]
</ul>
-<p>The following incompatibilities cause <code class="docutils literal notranslate"><span class="pre">MapFromArrays</span></code> to fall back to Spark by default. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.MapFromArrays.allowIncompatible=true</span></code> to enable Comet acceleration despite these differences.</p>
+<p>By default, <code class="docutils literal notranslate"><span class="pre">MapFromArrays</span></code> is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.MapFromArrays.allowIncompatible=true</span></code> to opt into Comet’s native implementation instead, which has the following differences from Spark:</p>
<ul class="simple">
+<li><p>Spark finds duplicate map keys that hold a <code class="docutils literal notranslate"><span class="pre">FLOAT</span></code> or <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> by value, so NaNs with different bits are one key, and so are <code class="docutils literal notranslate"><span class="pre">-0.0</span></code> and <code class="docutils literal notranslate"><span class="pre">0.0</span></code> inside a struct or array key. From Spark 4.0, a <co [...]
<li><p><code class="docutils literal notranslate"><span class="pre">spark.sql.mapKeyDedupPolicy</span></code> is set to <code class="docutils literal notranslate"><span class="pre">LAST_WIN</span></code>; Comet’s native map construction does not implement LAST_WIN dedup semantics.</p></li>
</ul>
</section>
@@ -884,6 +885,7 @@ IEEE total ordering for floating-point, which differs from Spark’s <code class
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">BinaryType</span></code> is not supported as a map key in <code class="docutils literal notranslate"><span class="pre">map_from_entries</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">BinaryType</span></code> is not supported as a map value in <code class="docutils literal notranslate"><span class="pre">map_from_entries</span></code></p></li>
+<li><p>Spark finds duplicate map keys that hold a <code class="docutils literal notranslate"><span class="pre">FLOAT</span></code> or <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> by value, so NaNs with different bits are one key, and so are <code class="docutils literal notranslate"><span class="pre">-0.0</span></code> and <code class="docutils literal notranslate"><span class="pre">0.0</span></code> inside a struct or array key. From Spark 4.0, a <co [...]
<li><p><code class="docutils literal notranslate"><span class="pre">spark.sql.mapKeyDedupPolicy</span></code> is set to <code class="docutils literal notranslate"><span class="pre">LAST_WIN</span></code>; Comet’s native map construction does not implement LAST_WIN dedup semantics.</p></li>
</ul>
</section>
diff --git a/user-guide/latest/compatibility/expressions/spark-4.2/map.html b/user-guide/latest/compatibility/expressions/spark-4.2/map.html
index 3564854b39..acf2a1f5f7 100644
--- a/user-guide/latest/compatibility/expressions/spark-4.2/map.html
+++ b/user-guide/latest/compatibility/expressions/spark-4.2/map.html
@@ -869,8 +869,9 @@ IEEE total ordering for floating-point, which differs from Spark’s <code class
<ul class="simple">
<li><p>Spark rejects a <code class="docutils literal notranslate"><span class="pre">NULL</span></code> element inside the keys array with a <code class="docutils literal notranslate"><span class="pre">RuntimeException</span></code> (<code class="docutils literal notranslate"><span class="pre">Cannot</span> <span class="pre">use</span> <span class="pre">null</span> <span class="pre">as</span> <span class="pre">map</span> <span class="pre">key</span></code>); Comet’s native <code class="do [...]
</ul>
-<p>The following incompatibilities cause <code class="docutils literal notranslate"><span class="pre">MapFromArrays</span></code> to fall back to Spark by default. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.MapFromArrays.allowIncompatible=true</span></code> to enable Comet acceleration despite these differences.</p>
+<p>By default, <code class="docutils literal notranslate"><span class="pre">MapFromArrays</span></code> is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.MapFromArrays.allowIncompatible=true</span></code> to opt into Comet’s native implementation instead, which has the following differences from Spark:</p>
<ul class="simple">
+<li><p>Spark finds duplicate map keys that hold a <code class="docutils literal notranslate"><span class="pre">FLOAT</span></code> or <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> by value, so NaNs with different bits are one key, and so are <code class="docutils literal notranslate"><span class="pre">-0.0</span></code> and <code class="docutils literal notranslate"><span class="pre">0.0</span></code> inside a struct or array key. From Spark 4.0, a <co [...]
<li><p><code class="docutils literal notranslate"><span class="pre">spark.sql.mapKeyDedupPolicy</span></code> is set to <code class="docutils literal notranslate"><span class="pre">LAST_WIN</span></code>; Comet’s native map construction does not implement LAST_WIN dedup semantics.</p></li>
</ul>
</section>
@@ -884,6 +885,7 @@ IEEE total ordering for floating-point, which differs from Spark’s <code class
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">BinaryType</span></code> is not supported as a map key in <code class="docutils literal notranslate"><span class="pre">map_from_entries</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">BinaryType</span></code> is not supported as a map value in <code class="docutils literal notranslate"><span class="pre">map_from_entries</span></code></p></li>
+<li><p>Spark finds duplicate map keys that hold a <code class="docutils literal notranslate"><span class="pre">FLOAT</span></code> or <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> by value, so NaNs with different bits are one key, and so are <code class="docutils literal notranslate"><span class="pre">-0.0</span></code> and <code class="docutils literal notranslate"><span class="pre">0.0</span></code> inside a struct or array key. From Spark 4.0, a <co [...]
<li><p><code class="docutils literal notranslate"><span class="pre">spark.sql.mapKeyDedupPolicy</span></code> is set to <code class="docutils literal notranslate"><span class="pre">LAST_WIN</span></code>; Comet’s native map construction does not implement LAST_WIN dedup semantics.</p></li>
</ul>
</section>
diff --git a/user-guide/latest/compatibility/floating-point.html b/user-guide/latest/compatibility/floating-point.html
index 49bf90f25f..03b658890e 100644
--- a/user-guide/latest/compatibility/floating-point.html
+++ b/user-guide/latest/compatibility/floating-point.html
@@ -974,6 +974,14 @@ can be null, such as one built from nullable columns, <code class="docutils lite
array whose elements cannot be null in ascending order, Spark’s generated code puts <code class="docutils literal notranslate"><span class="pre">-0.0</span></code> before
<code class="docutils literal notranslate"><span class="pre">0.0</span></code>, and so does Comet. All three expressions run natively in strict floating-point mode.</p>
</section>
+<section id="map-construction">
+<h2>Map construction<a class="headerlink" href="#map-construction" title="Link to this heading">#</a></h2>
+<p><code class="docutils literal notranslate"><span class="pre">map_from_arrays</span></code> and <code class="docutils literal notranslate"><span class="pre">map_from_entries</span></code> don’t build a map natively when its key type holds a
+<code class="docutils literal notranslate"><span class="pre">FLOAT</span></code> or <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> at any depth. They run Spark’s code through the codegen dispatcher instead, so
+duplicate keys, including NaNs and zeros of either sign, behave as in Spark. <code class="docutils literal notranslate"><span class="pre">FLOAT</span></code> and <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code>
+values don’t affect either function. The expressions’ entries in the
+<a class="reference internal" href="#expressions/map.md"><span class="xref myst">expression compatibility guide</span></a> give the reason.</p>
+</section>
</section>
@@ -1028,6 +1036,7 @@ array whose elements cannot be null in ascending order, Spark’s generated code
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#min-max-greatest-and-least"><code class="docutils literal notranslate"><span class="pre">min</span></code>, <code class="docutils literal notranslate"><span class="pre">max</span></code>, <code class="docutils literal notranslate"><span class="pre">greatest</span></code>, and <code class="docutils literal notranslate"><span class="pre">least</span></code></a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#array-distinct-and-union">Array distinct and union</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#array-contains-array-remove-and-sort-array"><code class="docutils literal notranslate"><span class="pre">array_contains</span></code>, <code class="docutils literal notranslate"><span class="pre">array_remove</span></code> and <code class="docutils literal notranslate"><span class="pre">sort_array</span></code></a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#map-construction">Map construction</a></li>
</ul>
</nav></div>
diff --git a/user-guide/latest/compatibility/operators.html b/user-guide/latest/compatibility/operators.html
index 5d146160f6..658b200dcc 100644
--- a/user-guide/latest/compatibility/operators.html
+++ b/user-guide/latest/compatibility/operators.html
@@ -928,6 +928,14 @@ rows by the grouping keys only, and Spark and Comet can leave rows with equal ke
orders, so a group with more than one candidate value can return a different value than Spark.
Both results are valid under Spark’s semantics for these functions.</p>
</section>
+<section id="hash-join">
+<h2>Hash Join<a class="headerlink" href="#hash-join" title="Link to this heading">#</a></h2>
+<p>Spark plans <code class="docutils literal notranslate"><span class="pre">NOT</span> <span class="pre">IN</span></code> as a null-aware anti join, and the native hash join builds it on Spark’s
+streamed side without keeping that side’s order. When Spark reports the streamed side as ordered,
+for example the output of a window, a sort-merge join, or a cached table sorted within its
+partitions, Comet falls back to Spark for the join, so that a sort Spark removed above it on the
+strength of that order is not lost.</p>
+</section>
<section id="window-functions">
<h2>Window Functions<a class="headerlink" href="#window-functions" title="Link to this heading">#</a></h2>
<p>Comet runs <code class="docutils literal notranslate"><span class="pre">WindowExec</span></code> natively and it is enabled by default (<code class="docutils literal notranslate"><span class="pre">spark.comet.exec.window.enabled</span></code>). A broad set of
@@ -1113,6 +1121,7 @@ they will be identical to Spark. Unsorted results may have different row orderin
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#sort">Sort</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#hash-aggregation">Hash Aggregation</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#sort-aggregation">Sort Aggregation</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#hash-join">Hash Join</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#window-functions">Window Functions</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#merge-into-mergerowsexec">MERGE INTO (MergeRowsExec)</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#round-robin-partitioning">Round-Robin Partitioning</a></li>
diff --git a/user-guide/latest/configs.html b/user-guide/latest/configs.html
index 3ee479baae..4c66def35f 100644
--- a/user-guide/latest/configs.html
+++ b/user-guide/latest/configs.html
@@ -1402,7 +1402,7 @@ them like any other production setting.</p>
<td><p>true</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.existenceJoin.enabled</span></code></p></td>
-<td><p>Whether to enable native ExistenceJoin, produced when EXISTS / NOT EXISTS / IN is combined with another predicate via OR. Sort-merge joins, residual (non-equi) join conditions, computed (non-column) join keys, and NOT IN fall back to Spark.</p></td>
+<td><p>Whether to enable native ExistenceJoin, produced when EXISTS / NOT EXISTS / IN is combined with another predicate via OR. Sort-merge joins, residual (non-equi) join conditions, computed (non-column) join keys, and NOT IN combined with OR fall back to Spark.</p></td>
<td><p>true</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.expand.enabled</span></code></p></td>
diff --git a/user-guide/latest/expressions.html b/user-guide/latest/expressions.html
index 9b36a8c4b1..fab08627df 100644
--- a/user-guide/latest/expressions.html
+++ b/user-guide/latest/expressions.html
@@ -2305,13 +2305,13 @@ to Spark (<a class="reference external" href="https://github.com/apache/datafusi
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">map_from_arrays</span></code></p></td>
<td><p>✅</p></td>
-<td><p>Native</p></td>
-<td><p></p></td>
+<td><p>Hybrid</p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">FLOAT</span></code> or <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> keys and <code class="docutils literal notranslate"><span class="pre">spark.sql.mapKeyDedupPolicy=LAST_WIN</span></code> route through the JVM codegen dispatcher (<a class="reference internal" href="#compatibility/expressions/map.md"><span class="xref myst">details</span></a>)</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">map_from_entries</span></code></p></td>
<td><p>✅</p></td>
<td><p>Hybrid</p></td>
-<td><p>BinaryType keys/values and <code class="docutils literal notranslate"><span class="pre">spark.sql.mapKeyDedupPolicy=LAST_WIN</span></code> route through the JVM codegen dispatcher (<a class="reference internal" href="#compatibility/expressions/map.md"><span class="xref myst">details</span></a>)</p></td>
+<td><p>BinaryType keys/values, <code class="docutils literal notranslate"><span class="pre">FLOAT</span></code> or <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> keys and <code class="docutils literal notranslate"><span class="pre">spark.sql.mapKeyDedupPolicy=LAST_WIN</span></code> route through the JVM codegen dispatcher (<a class="reference internal" href="#compatibility/expressions/map.md"><span class="xref myst">details</span></a>)</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">map_keys</span></code></p></td>
<td><p>✅</p></td>
diff --git a/user-guide/latest/operators.html b/user-guide/latest/operators.html
index c1f05215f0..090855cefa 100644
--- a/user-guide/latest/operators.html
+++ b/user-guide/latest/operators.html
@@ -1033,7 +1033,7 @@ omitted from the tables below and may be reconsidered based on demand:</p>
<tbody>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">BroadcastHashJoinExec</span></code></p></td>
<td><p>✅</p></td>
-<td><p></p></td>
+<td><p>Falls back to Spark for a <code class="docutils literal notranslate"><span class="pre">NOT</span> <span class="pre">IN</span></code> (null-aware anti join) whose streamed side reports an ordering.</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">ShuffledHashJoinExec</span></code></p></td>
<td><p>✅</p></td>
@@ -1052,7 +1052,8 @@ omitted from the tables below and may be reconsidered based on demand:</p>
</div>
<p><code class="docutils literal notranslate"><span class="pre">ExistenceJoin</span></code> (produced when <code class="docutils literal notranslate"><span class="pre">EXISTS</span></code> / <code class="docutils literal notranslate"><span class="pre">IN</span></code> is combined with another predicate via <code class="docutils literal notranslate"><span class="pre">OR</span></code>) runs
natively on <code class="docutils literal notranslate"><span class="pre">BroadcastHashJoinExec</span></code> and <code class="docutils literal notranslate"><span class="pre">ShuffledHashJoinExec</span></code>. Existence sort-merge joins, residual
-(non-equi) join conditions, computed (non-column) join keys, and <code class="docutils literal notranslate"><span class="pre">NOT</span> <span class="pre">IN</span></code> fall back to Spark.</p>
+(non-equi) join conditions, and computed (non-column) join keys fall back to Spark. So does <code class="docutils literal notranslate"><span class="pre">NOT</span> <span class="pre">IN</span></code>
+combined with <code class="docutils literal notranslate"><span class="pre">OR</span></code>, which Spark plans as an existence <code class="docutils literal notranslate"><span class="pre">BroadcastNestedLoopJoinExec</span></code>.</p>
</section>
<section id="exchanges">
<h2>Exchanges<a class="headerlink" href="#exchanges" title="Link to this heading">#</a></h2>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org