You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/16 21:02:43 UTC
(datafusion-comet) 01/01: fix: stop building maps natively when the key type holds a FLOAT or DOUBLE (#6992)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-6992-c4ca20e8b6070e1b7482edda5cf6a9629de04fbb
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit 46b9c6afe20ccb8f43ca747d87b18f408486b364
Author: Andy Grove <ag...@apache.org>
AuthorDate: Fri Oct 16 20:50:42 2026 +0000
fix: stop building maps natively when the key type holds a FLOAT or DOUBLE (#6992)
* fix: stop building maps natively when the key type holds a FLOAT or DOUBLE
The native map_from_arrays and map_from_entries tell keys apart by
their bits, and they report Compatible for every key type. Spark's
ArrayBasedMapBuilder finds a duplicate FLOAT or DOUBLE key in a HashMap
of boxed values, where NaNs with different bits are one key, and a
duplicate struct or array key in a TreeMap in its SQL ordering, where
-0.0 equals 0.0 too. From Spark 4.0 it also normalizes a top-level key
before the lookup, unless spark.sql.legacy.disableMapKeyNormalization is
set, so the two zeros are one key and map_from_entries stores 0.0.
Comet built maps that Spark rejects with DUPLICATED_MAP_KEY and kept
-0.0 keys that Spark stores as 0.0.
Until the builders follow these rules natively, report any key type
that holds a float at any depth as Incompatible, on every Spark version.
map_from_entries then runs through the codegen dispatcher and
map_from_arrays falls back to Spark, as each already does for LAST_WIN.
Closes #6549.
* fix: route map_from_arrays float keys through the codegen dispatcher
Address review feedback on the float map key gate:
- Mix CodegenDispatchFallback into CometMapFromArrays so FLOAT or
DOUBLE keys and LAST_WIN stay in the Comet pipeline, as they do for
map_from_entries.
- Use expect_error_class for the dispatched DUPLICATED_MAP_KEY queries,
and fix the array-key zeros query to read the row that holds zeros.
- Rename the fixtures to map_builders_float_key_gate* so they do not
collide with the ones in #6682.
- Correct the MapKeySupport reason, which described the lookup as
Spark normalizing keys, and trim the repeated explanation of the
builder rules down to MapBuilderKeySupport.floatingPointReason.
* test: cover the dispatched map builders under LAST_WIN with a repeated key
The dedup-policy fixtures turn the codegen dispatcher off, so nothing ran the
default LAST_WIN path with a key that really repeats. Add
map_builders_float_key_last_win.sql, which runs map_from_arrays and
map_from_entries through the dispatcher with a repeated string key, NaNs that
differ only in the sign bit, and 0.0 / -0.0, comparing the kept value and the
stored key with Spark's on every version.
Also drop the claim that Spark normalizes map keys before a lookup from the
MapKeySupport scaladoc and the lookup fixtures. Normalization is a builder
behavior from Spark 4.0, and the lookup compares with nanSafeCompareDoubles on
every version.
---
.../expression-audits/map_funcs.md | 4 +-
docs/source/contributor-guide/floating_point.md | 9 +-
.../latest/compatibility/floating-point.md | 8 ++
docs/source/user-guide/latest/expressions.md | 4 +-
.../main/scala/org/apache/comet/serde/maps.scala | 50 ++++++++---
.../sql-tests/expressions/map/element_at_map.sql | 10 +--
.../sql-tests/expressions/map/get_map_value.sql | 6 +-
.../map/map_builders_float_key_gate.sql | 99 ++++++++++++++++++++++
.../map/map_builders_float_key_gate_legacy.sql | 41 +++++++++
.../map/map_builders_float_key_gate_spark_4_0.sql | 42 +++++++++
.../map/map_builders_float_key_last_win.sql | 93 ++++++++++++++++++++
.../map/map_from_arrays_dedup_policy.sql | 9 +-
.../map/map_from_entries_dedup_policy.sql | 5 +-
.../org/apache/comet/CometMapExpressionSuite.scala | 6 +-
14 files changed, 353 insertions(+), 33 deletions(-)
diff --git a/docs/source/contributor-guide/expression-audits/map_funcs.md b/docs/source/contributor-guide/expression-audits/map_funcs.md
index f50475a449..d37a29df1b 100644
--- a/docs/source/contributor-guide/expression-audits/map_funcs.md
+++ b/docs/source/contributor-guide/expression-audits/map_funcs.md
@@ -46,8 +46,9 @@
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. `MapFromArrays(left, right) extends BinaryExpression with NullIntolerant`; Spark uses `ArrayBasedMapBuilder` to detect duplicate keys (subject to `spark.sql.mapKeyDedupPolicy`) and rejects null keys with `RuntimeException("Cannot use null as map key")`. Comet `CometMapFromArrays` emits nested `CaseWhen(IsNotNull(left), CaseWhen(IsNotNull(right), map(left, right), null), null)` guards so NULL-array inputs return NULL rather than triggering the [...]
-- Spark 4.0.1 (audited 2026-05-27): semantics unchanged; `NullIntolerant` trait replaced by `nullIntolerant: Boolean`.
+- Spark 4.0.1 (audited 2026-05-27): `NullIntolerant` trait replaced by `nullIntolerant: Boolean`. `ArrayBasedMapBuilder` gains the `keyNormalizer` described below.
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+- Floating-point keys (2026-10-14, [#6549](https://github.com/apache/datafusion-comet/issues/6549)): `ArrayBasedMapBuilder` finds a duplicate `FLOAT` or `DOUBLE` key in a `HashMap` of boxed values (`Double.equals`, so NaNs with different bits are one key) and a duplicate struct or array key in a `TreeMap` in the SQL ordering (`-0.0` equals `0.0`). From Spark 4.0 its `keyNormalizer` also normalizes a `FLOAT` or `DOUBLE` key before the lookup, unless `spark.sql.legacy.disableMapKeyNormaliz [...]
## map_from_entries
@@ -56,6 +57,7 @@
- Spark 4.0.1 (audited 2026-05-27): semantics unchanged; trait refactor.
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
- Known limitation: input arrays where the struct's key or value type contains `BinaryType` are marked `Incompatible` and fall back unless `spark.comet.expression.MapFromEntries.allowIncompatible=true`.
+- Floating-point keys (2026-10-14, [#6549](https://github.com/apache/datafusion-comet/issues/6549)): `ArrayBasedMapBuilder` finds a duplicate `FLOAT` or `DOUBLE` key in a `HashMap` of boxed values (`Double.equals`, so NaNs with different bits are one key) and a duplicate struct or array key in a `TreeMap` in the SQL ordering (`-0.0` equals `0.0`). From Spark 4.0 its `keyNormalizer` also normalizes a `FLOAT` or `DOUBLE` key before the lookup, unless `spark.sql.legacy.disableMapKeyNormaliz [...]
## map_keys
diff --git a/docs/source/contributor-guide/floating_point.md b/docs/source/contributor-guide/floating_point.md
index 165b280c63..758350db8f 100644
--- a/docs/source/contributor-guide/floating_point.md
+++ b/docs/source/contributor-guide/floating_point.md
@@ -62,10 +62,11 @@ it runs against:
(before 3.5.2, unless probing merges them). Spark 4.2.0 normalizes their arguments in the plan
(SPARK-54918). From 4.0.5, 4.1.4 and 4.2.1 they normalize while they evaluate instead
(SPARK-59602).
-- Map construction (`ArrayBasedMapBuilder`) finds duplicate keys by `Double.equals` in Spark 3.4
- and 3.5. From Spark 4.0 it normalizes each key first, unless
- `spark.sql.legacy.disableMapKeyNormalization` is set
- ([#6549](https://github.com/apache/datafusion-comet/issues/6549)).
+- Map construction (`ArrayBasedMapBuilder`) finds a duplicate `FLOAT` or `DOUBLE` key in a
+ `HashMap` of boxed values, which follows `Double.equals`, and a duplicate struct or array key in a
+ `TreeMap` in the SQL ordering. From Spark 4.0 it normalizes a `FLOAT` or `DOUBLE` key before the
+ lookup, unless `spark.sql.legacy.disableMapKeyNormalization` is set, and `map_from_entries`
+ stores the normalized key ([#6549](https://github.com/apache/datafusion-comet/issues/6549)).
## How Arrow and DataFusion differ
diff --git a/docs/source/user-guide/latest/compatibility/floating-point.md b/docs/source/user-guide/latest/compatibility/floating-point.md
index f00ceceaf6..77d76f53c6 100644
--- a/docs/source/user-guide/latest/compatibility/floating-point.md
+++ b/docs/source/user-guide/latest/compatibility/floating-point.md
@@ -164,3 +164,11 @@ representations tie, and keeps equal elements in their original order. In an arr
can be null, such as one built from nullable columns, `-0.0` and `0.0` therefore tie. Sorting an
array whose elements cannot be null in ascending order, Spark's generated code puts `-0.0` before
`0.0`, and so does Comet. All three expressions run natively in strict floating-point mode.
+
+## Map construction
+
+`map_from_arrays` and `map_from_entries` don't build a map natively when its key type holds a
+`FLOAT` or `DOUBLE` at any depth. They run Spark's code through the codegen dispatcher instead, so
+duplicate keys, including NaNs and zeros of either sign, behave as in Spark. `FLOAT` and `DOUBLE`
+values don't affect either function. The expressions' entries in the
+[expression compatibility guide](expressions/map.md) give the reason.
diff --git a/docs/source/user-guide/latest/expressions.md b/docs/source/user-guide/latest/expressions.md
index b0918c292e..bb2c80a5ce 100644
--- a/docs/source/user-guide/latest/expressions.md
+++ b/docs/source/user-guide/latest/expressions.md
@@ -408,8 +408,8 @@ to Spark ([#2837](https://github.com/apache/datafusion-comet/issues/2837)). Enab
| `map_concat` | ✅ | Codegen dispatch | |
| `map_contains_key` | ✅ | — | |
| `map_entries` | ✅ | Native | |
-| `map_from_arrays` | ✅ | Native | |
-| `map_from_entries` | ✅ | Hybrid | BinaryType keys/values and `spark.sql.mapKeyDedupPolicy=LAST_WIN` route through the JVM codegen dispatcher ([details](compatibility/expressions/map.md)) |
+| `map_from_arrays` | ✅ | Hybrid | `FLOAT` or `DOUBLE` keys and `spark.sql.mapKeyDedupPolicy=LAST_WIN` route through the JVM codegen dispatcher ([details](compatibility/expressions/map.md)) |
+| `map_from_entries` | ✅ | Hybrid | BinaryType keys/values, `FLOAT` or `DOUBLE` keys and `spark.sql.mapKeyDedupPolicy=LAST_WIN` route through the JVM codegen dispatcher ([details](compatibility/expressions/map.md)) |
| `map_keys` | ✅ | Native | |
| `map_values` | ✅ | Native | |
| `str_to_map` | ✅ | Hybrid | |
diff --git a/spark/src/main/scala/org/apache/comet/serde/maps.scala b/spark/src/main/scala/org/apache/comet/serde/maps.scala
index ae32794d7f..86376ccfca 100644
--- a/spark/src/main/scala/org/apache/comet/serde/maps.scala
+++ b/spark/src/main/scala/org/apache/comet/serde/maps.scala
@@ -34,7 +34,7 @@ import org.apache.comet.shims.CometTypeShim
private[serde] object MapKeySupport {
private val floatingPointReason: String =
- "Spark normalizes floating-point map keys, so `-0.0` matches a `+0.0` key and all `NaN`s " +
+ "Spark looks a key up in its SQL ordering, so `-0.0` matches a `+0.0` key and all `NaN`s " +
"match each other; Comet's native map lookup compares the raw Arrow values."
private val collationReason: String =
@@ -48,12 +48,13 @@ private[serde] object MapKeySupport {
/**
* The `SupportLevel` for a map-consuming expression whose stored-key type is `keyType`. Spark
- * finds a key with `TypeUtils.getInterpretedOrdering` over the keys `ArrayBasedMapBuilder`
- * stored, having first normalized them, while native `map_extract` compares the Arrow values as
- * they are, so decline the key types where those disagree:
- * - `ArrayBasedMapBuilder` rewrites a `-0.0` key to `+0.0` and canonicalises `NaN`, and
- * `nanSafeCompareDoubles` treats `-0.0` and `+0.0` as equal, so Spark answers a `-0.0`
- * lookup from a `+0.0` key where native finds nothing.
+ * finds a key with `TypeUtils.getInterpretedOrdering` over the map's stored keys, while native
+ * `map_extract` compares the Arrow values as they are, so decline the key types where those
+ * disagree:
+ * - `nanSafeCompareDoubles` treats `-0.0` and `+0.0` as equal and all `NaN`s as equal, so
+ * Spark answers a `-0.0` lookup from a `+0.0` key where native finds nothing, on every
+ * version. (How `ArrayBasedMapBuilder` treats float keys when it builds a map differs by
+ * version; see `MapBuilderKeySupport`.)
* - a non-default collation compares under rules native applies as `UTF8_BINARY`.
* - for a complex key type, `map_extract`'s `coerce_types` returns the map's exact key field
* type, so Comet's planner casts the lookup key to it. That cannot reproduce Spark's
@@ -132,6 +133,23 @@ object CometMapExtract extends CometExpressionSerde[GetMapValue] {
}
}
+/**
+ * Shared gate for the native map builders, `map_from_arrays` and `map_from_entries`, which tell
+ * keys apart by their bits. A key type that holds a float at any depth is `Incompatible` on every
+ * Spark version; `floatingPointReason` explains why.
+ */
+private object MapBuilderKeySupport {
+ val floatingPointReason: String =
+ "Spark finds duplicate map keys that hold a `FLOAT` or `DOUBLE` by value, so NaNs with " +
+ "different bits are one key, and so are `-0.0` and `0.0` inside a struct or array key. " +
+ "From Spark 4.0, a `-0.0` key is also normalized to `0.0`, unless " +
+ "`spark.sql.legacy.disableMapKeyNormalization` is set. Comet's native map construction " +
+ "compares the keys' bits ([#6549](https://github.com/apache/datafusion-comet/issues/6549))."
+
+ def hasFloatingPointKey(dataType: MapType): Boolean =
+ SupportLevel.containsType(dataType.keyType, classOf[FloatType], classOf[DoubleType])
+}
+
private object MapKeyDedupPolicySupport {
val incompatibleReason: String =
s"`${SQLConf.MAP_KEY_DEDUP_POLICY.key}` is set to " +
@@ -151,16 +169,20 @@ private object MapKeyDedupPolicySupport {
.equalsIgnoreCase(SQLConf.MapKeyDedupPolicy.LAST_WIN.toString)
}
-object CometMapFromArrays extends CometExpressionSerde[MapFromArrays] {
+object CometMapFromArrays
+ extends CometExpressionSerde[MapFromArrays]
+ with CodegenDispatchFallback {
override def getIncompatibleReasons(): Seq[String] =
- Seq(MapKeyDedupPolicySupport.incompatibleReason)
+ Seq(MapBuilderKeySupport.floatingPointReason, MapKeyDedupPolicySupport.incompatibleReason)
override def getCompatibleNotes(): Seq[String] =
Seq(MapKeyDedupPolicySupport.nullKeyReason)
override def getSupportLevel(expr: MapFromArrays): SupportLevel = {
- if (MapKeyDedupPolicySupport.isLastWin) {
+ if (MapBuilderKeySupport.hasFloatingPointKey(expr.dataType)) {
+ Incompatible(Some(MapBuilderKeySupport.floatingPointReason))
+ } else if (MapKeyDedupPolicySupport.isLastWin) {
Incompatible(Some(MapKeyDedupPolicySupport.incompatibleReason))
} else {
Compatible(None)
@@ -214,7 +236,11 @@ object CometMapFromEntries
"`BinaryType` is not supported as a map value in `map_from_entries`"
override def getIncompatibleReasons(): Seq[String] =
- Seq(keyUnsupportedReason, valueUnsupportedReason, MapKeyDedupPolicySupport.incompatibleReason)
+ Seq(
+ keyUnsupportedReason,
+ valueUnsupportedReason,
+ MapBuilderKeySupport.floatingPointReason,
+ MapKeyDedupPolicySupport.incompatibleReason)
override def getCompatibleNotes(): Seq[String] =
Seq(MapKeyDedupPolicySupport.nullKeyReason)
@@ -224,6 +250,8 @@ object CometMapFromEntries
Incompatible(Some(keyUnsupportedReason))
} else if (SupportLevel.containsType(expr.dataType.valueType, classOf[BinaryType])) {
Incompatible(Some(valueUnsupportedReason))
+ } else if (MapBuilderKeySupport.hasFloatingPointKey(expr.dataType)) {
+ Incompatible(Some(MapBuilderKeySupport.floatingPointReason))
} else if (MapKeyDedupPolicySupport.isLastWin) {
Incompatible(Some(MapKeyDedupPolicySupport.incompatibleReason))
} else {
diff --git a/spark/src/test/resources/sql-tests/expressions/map/element_at_map.sql b/spark/src/test/resources/sql-tests/expressions/map/element_at_map.sql
index 95450b8159..b98854320f 100644
--- a/spark/src/test/resources/sql-tests/expressions/map/element_at_map.sql
+++ b/spark/src/test/resources/sql-tests/expressions/map/element_at_map.sql
@@ -56,17 +56,17 @@ SELECT element_at(map('a', 1, 'b', 2), 'a'), element_at(map('a', 1, 'b', 2), 'mi
-- Spark. These stay on the constructor path here because the SQL harness excludes
-- `ConstantFolding`; `CometMapExpressionSuite` covers the folded-literal form of each.
--- Spark stores `-0.0` map keys as `+0.0` and compares with `nanSafeCompareDoubles`, so a `-0.0`
--- lookup finds the `+0.0` key. Native lookup compares the raw Arrow values.
-query expect_fallback(Spark normalizes floating-point map keys)
+-- Spark compares with `nanSafeCompareDoubles`, so a `-0.0` lookup finds the `+0.0` key. Native
+-- lookup compares the raw Arrow values.
+query expect_fallback(Spark looks a key up in its SQL ordering)
SELECT element_at(map(CAST(0 AS DOUBLE), 7), CAST(-0.0 AS DOUBLE))
-query expect_fallback(Spark normalizes floating-point map keys)
+query expect_fallback(Spark looks a key up in its SQL ordering)
SELECT element_at(map(CAST(0 AS FLOAT), 7), CAST(-0.0 AS FLOAT))
-- The floating-point decline walks every nesting level of the key type, so an array-of-double key
-- falls back for the same reason.
-query expect_fallback(Spark normalizes floating-point map keys)
+query expect_fallback(Spark looks a key up in its SQL ordering)
SELECT element_at(map(array(CAST(0 AS DOUBLE)), 7), array(CAST(-0.0 AS DOUBLE)))
-- A complex key type: `map_extract` casts the lookup key to the map's exact Arrow key type, so a
diff --git a/spark/src/test/resources/sql-tests/expressions/map/get_map_value.sql b/spark/src/test/resources/sql-tests/expressions/map/get_map_value.sql
index 2c6c23846e..b693cf5ae1 100644
--- a/spark/src/test/resources/sql-tests/expressions/map/get_map_value.sql
+++ b/spark/src/test/resources/sql-tests/expressions/map/get_map_value.sql
@@ -36,13 +36,13 @@ SELECT map('a', 1, 'b', 2)['a'], map('a', 1, 'b', 2)['missing'], map('a', 1, 'b'
-- `element_at_map.sql` exercises the same guard on `CometElementAt`). A `map<double,int>` column
-- is used rather than a `map(...)` literal because Spark's `SimplifyExtractValueOps` rewrites
-- `map(...)[key]` over a literal map into a `CASE` before it can reach the native lookup. Spark
--- stores a `-0.0` key as `+0.0` and finds it with `nanSafeCompareDoubles`, so it returns `7` for a
--- `-0.0` lookup; native lookup compares the raw Arrow values.
+-- compares with `nanSafeCompareDoubles`, so a `-0.0` lookup finds the `+0.0` key and returns `7`;
+-- native lookup compares the raw Arrow values.
statement
CREATE TABLE test_map_double(m map<double, int>) USING parquet
statement
INSERT INTO test_map_double VALUES (map(CAST(0 AS DOUBLE), 7)), (map(CAST(1 AS DOUBLE), 8)), (NULL)
-query expect_fallback(Spark normalizes floating-point map keys)
+query expect_fallback(Spark looks a key up in its SQL ordering)
SELECT m[CAST(-0.0 AS DOUBLE)] FROM test_map_double
diff --git a/spark/src/test/resources/sql-tests/expressions/map/map_builders_float_key_gate.sql b/spark/src/test/resources/sql-tests/expressions/map/map_builders_float_key_gate.sql
new file mode 100644
index 0000000000..6708e62e3c
--- /dev/null
+++ b/spark/src/test/resources/sql-tests/expressions/map/map_builders_float_key_gate.sql
@@ -0,0 +1,99 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- Map keys that hold a FLOAT or DOUBLE (#6549) are declined by the native builders. See
+-- `MapBuilderKeySupport.floatingPointReason` for how Spark treats them. map_from_entries runs
+-- through the codegen dispatcher, as does map_from_arrays. Top-level zeros, which only Spark 4.0+
+-- merges, are in map_builders_float_key_gate_spark_4_0.sql.
+
+-- Parquet canonicalizes NaN, so negating a stored NaN gives one with the sign bit set.
+statement
+CREATE TABLE float_map_keys(id int, d double, f float) USING parquet
+
+statement
+INSERT INTO float_map_keys VALUES
+ (1, 0.0, 0.0),
+ (2, double('NaN'), float('NaN')),
+ (3, 1.5, 1.5)
+
+-- A lone -0.0 key: Spark 4.0+ stores 0.0, earlier releases keep -0.0.
+query expect_dispatch(map_from_entries)
+SELECT id,
+ map_keys(map_from_entries(array(named_struct('k', -d, 'v', 1), named_struct('k', 2.5D, 'v', 2)))),
+ map_keys(map_from_entries(array(named_struct('k', -f, 'v', 1), named_struct('k', 2.5F, 'v', 2))))
+FROM float_map_keys
+
+query expect_dispatch(map_from_arrays)
+SELECT id,
+ map_keys(map_from_arrays(array(-d, 2.5D), array(1, 2))),
+ map_keys(map_from_arrays(array(-f, 2.5F), array(1, 2)))
+FROM float_map_keys
+
+-- NaNs that differ only in the sign bit are one key on every version.
+query expect_error_class(DUPLICATED_MAP_KEY)
+SELECT map_from_entries(array(named_struct('k', d, 'v', 1), named_struct('k', -d, 'v', 2)))
+FROM float_map_keys WHERE id = 2
+
+query expect_error_class(DUPLICATED_MAP_KEY)
+SELECT map_from_entries(array(named_struct('k', f, 'v', 1), named_struct('k', -f, 'v', 2)))
+FROM float_map_keys WHERE id = 2
+
+query expect_error_class(DUPLICATED_MAP_KEY)
+SELECT map_from_arrays(array(d, -d), array(1, 2)) FROM float_map_keys WHERE id = 2
+
+query expect_error_class(DUPLICATED_MAP_KEY)
+SELECT map_from_arrays(array(f, -f), array(1, 2)) FROM float_map_keys WHERE id = 2
+
+-- Struct and array keys compare in Spark's SQL ordering, where -0.0 equals 0.0, on every version.
+query expect_error_class(DUPLICATED_MAP_KEY)
+SELECT map_from_entries(array(
+ named_struct('k', named_struct('x', d), 'v', 1),
+ named_struct('k', named_struct('x', -d), 'v', 2)))
+FROM float_map_keys WHERE id = 1
+
+query expect_error_class(DUPLICATED_MAP_KEY)
+SELECT map_from_entries(array(
+ named_struct('k', array(f), 'v', 1),
+ named_struct('k', array(-f), 'v', 2)))
+FROM float_map_keys WHERE id = 1
+
+query expect_error_class(DUPLICATED_MAP_KEY)
+SELECT map_from_arrays(array(named_struct('x', d), named_struct('x', -d)), array(1, 2))
+FROM float_map_keys WHERE id = 1
+
+query expect_error_class(DUPLICATED_MAP_KEY)
+SELECT map_from_arrays(array(array(d), array(-d)), array(1, 2)) FROM float_map_keys WHERE id = 1
+
+-- NaNs that differ only in the sign bit are one key inside an array key too.
+query expect_error_class(DUPLICATED_MAP_KEY)
+SELECT map_from_arrays(array(array(d), array(-d)), array(1, 2)) FROM float_map_keys WHERE id = 2
+
+-- Distinct keys still build a map.
+query expect_dispatch(map_from_entries)
+SELECT id, map_from_entries(array(named_struct('k', d, 'v', 1), named_struct('k', d + 1, 'v', 2)))
+FROM float_map_keys WHERE id IN (1, 3)
+
+query expect_dispatch(map_from_arrays)
+SELECT id, map_from_arrays(array(d, d + 1), array(1, 2)) FROM float_map_keys WHERE id IN (1, 3)
+
+-- Only the key type matters: a FLOAT or DOUBLE value stays native.
+query expect_native(map_from_entries)
+SELECT id, map_from_entries(array(named_struct('k', id, 'v', d), named_struct('k', id + 10, 'v', -d)))
+FROM float_map_keys
+
+query
+SELECT id, map_from_arrays(array(id, id + 10), array(d, -d)) FROM float_map_keys
diff --git a/spark/src/test/resources/sql-tests/expressions/map/map_builders_float_key_gate_legacy.sql b/spark/src/test/resources/sql-tests/expressions/map/map_builders_float_key_gate_legacy.sql
new file mode 100644
index 0000000000..c1cbfab335
--- /dev/null
+++ b/spark/src/test/resources/sql-tests/expressions/map/map_builders_float_key_gate_legacy.sql
@@ -0,0 +1,41 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- With spark.sql.legacy.disableMapKeyNormalization, Spark 4.0+ leaves FLOAT and DOUBLE map keys as
+-- they are, as Spark 3.4 and 3.5 always do (they don't have the flag). So 0.0 and -0.0 are two
+-- keys and a -0.0 key stays -0.0. map_from_entries and map_from_arrays run Spark's code in the
+-- codegen dispatcher, which has to see the session's flag.
+
+-- Config: spark.sql.legacy.disableMapKeyNormalization=true
+
+statement
+CREATE TABLE float_map_keys_legacy(d double, f float) USING parquet
+
+statement
+INSERT INTO float_map_keys_legacy VALUES (0.0, 0.0)
+
+query expect_dispatch(map_from_entries)
+SELECT
+ map_keys(map_from_entries(array(named_struct('k', d, 'v', 1), named_struct('k', -d, 'v', 2)))),
+ map_keys(map_from_entries(array(named_struct('k', -f, 'v', 1), named_struct('k', 2.5F, 'v', 2))))
+FROM float_map_keys_legacy
+
+query expect_dispatch(map_from_arrays)
+SELECT
+ map_keys(map_from_arrays(array(d, -d), array(1, 2))),
+ map_keys(map_from_arrays(array(f, -f), array(1, 2)))
+FROM float_map_keys_legacy
diff --git a/spark/src/test/resources/sql-tests/expressions/map/map_builders_float_key_gate_spark_4_0.sql b/spark/src/test/resources/sql-tests/expressions/map/map_builders_float_key_gate_spark_4_0.sql
new file mode 100644
index 0000000000..7813c6598c
--- /dev/null
+++ b/spark/src/test/resources/sql-tests/expressions/map/map_builders_float_key_gate_spark_4_0.sql
@@ -0,0 +1,42 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- From Spark 4.0, ArrayBasedMapBuilder normalizes a top-level FLOAT or DOUBLE key before it looks
+-- for a duplicate, so 0.0 and -0.0 are one key (#6549). Earlier releases keep them apart, and so
+-- does spark.sql.legacy.disableMapKeyNormalization (map_builders_float_key_gate_legacy.sql).
+
+-- MinSparkVersion: 4.0
+
+statement
+CREATE TABLE float_map_keys_zeros(d double, f float) USING parquet
+
+statement
+INSERT INTO float_map_keys_zeros VALUES (0.0, 0.0)
+
+query expect_error_class(DUPLICATED_MAP_KEY)
+SELECT map_from_entries(array(named_struct('k', d, 'v', 1), named_struct('k', -d, 'v', 2)))
+FROM float_map_keys_zeros
+
+query expect_error_class(DUPLICATED_MAP_KEY)
+SELECT map_from_entries(array(named_struct('k', f, 'v', 1), named_struct('k', -f, 'v', 2)))
+FROM float_map_keys_zeros
+
+query expect_error_class(DUPLICATED_MAP_KEY)
+SELECT map_from_arrays(array(d, -d), array(1, 2)) FROM float_map_keys_zeros
+
+query expect_error_class(DUPLICATED_MAP_KEY)
+SELECT map_from_arrays(array(f, -f), array(1, 2)) FROM float_map_keys_zeros
diff --git a/spark/src/test/resources/sql-tests/expressions/map/map_builders_float_key_last_win.sql b/spark/src/test/resources/sql-tests/expressions/map/map_builders_float_key_last_win.sql
new file mode 100644
index 0000000000..b3c210dab1
--- /dev/null
+++ b/spark/src/test/resources/sql-tests/expressions/map/map_builders_float_key_last_win.sql
@@ -0,0 +1,93 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+-- Under `spark.sql.mapKeyDedupPolicy=LAST_WIN` a repeated map key is not an error: the builder keeps
+-- the last value. `CometMapFromArrays` and `CometMapFromEntries` mix in `CodegenDispatchFallback`,
+-- so with the dispatcher on (the default) both run Spark's own `ArrayBasedMapBuilder` in the JVM
+-- codegen dispatcher. The dedup-policy fixtures turn the dispatcher off to cover the fallback, so
+-- these queries cover the dispatched path with a key that really repeats: `from` then goes through
+-- `build()`, and both the kept value and the stored key are compared with Spark's.
+--
+-- A `FLOAT` or `DOUBLE` key is declined natively on every version (#6549), so those dispatch too.
+-- NaNs that differ only in the sign bit are one key on every version. `-0.0` and `0.0` are one key
+-- from Spark 4.0, unless `spark.sql.legacy.disableMapKeyNormalization` is set, and two keys on
+-- earlier releases; either way the dispatcher has to agree with Spark.
+
+-- Config: spark.sql.mapKeyDedupPolicy=LAST_WIN
+-- Config: spark.comet.exec.scalaUDF.codegen.enabled=true
+
+-- Parquet canonicalizes NaN, so negating a stored NaN gives one with the sign bit set.
+statement
+CREATE TABLE float_map_keys_last_win(id int, d double, f float, k array<string>, v array<int>)
+USING parquet
+
+statement
+INSERT INTO float_map_keys_last_win VALUES
+ (1, 0.0, 0.0, array('a', 'a', 'b'), array(1, 2, 3)),
+ (2, double('NaN'), float('NaN'), array('x', 'x'), array(10, 20)),
+ (3, 1.5, 1.5, array('a', 'b', 'c'), array(1, 2, 3))
+
+-- A repeated string key: the last value wins.
+query expect_dispatch(map_from_arrays)
+SELECT id, map_keys(map_from_arrays(k, v)), map_values(map_from_arrays(k, v))
+FROM float_map_keys_last_win
+
+query expect_dispatch(map_from_entries)
+SELECT id,
+ map_keys(map_from_entries(arrays_zip(k, v))),
+ map_values(map_from_entries(arrays_zip(k, v)))
+FROM float_map_keys_last_win
+
+-- NaNs with different sign bits are one DOUBLE or FLOAT key: one entry, holding the last value.
+query expect_dispatch(map_from_arrays)
+SELECT map_keys(map_from_arrays(array(d, -d), array(1, 2))),
+ map_values(map_from_arrays(array(d, -d), array(1, 2))),
+ map_keys(map_from_arrays(array(f, -f), array(1, 2))),
+ map_values(map_from_arrays(array(f, -f), array(1, 2)))
+FROM float_map_keys_last_win WHERE id = 2
+
+query expect_dispatch(map_from_entries)
+SELECT map_keys(map_from_entries(array(named_struct('k', d, 'v', 1), named_struct('k', -d, 'v', 2)))),
+ map_values(map_from_entries(array(named_struct('k', d, 'v', 1), named_struct('k', -d, 'v', 2)))),
+ map_keys(map_from_entries(array(named_struct('k', f, 'v', 1), named_struct('k', -f, 'v', 2)))),
+ map_values(map_from_entries(array(named_struct('k', f, 'v', 1), named_struct('k', -f, 'v', 2))))
+FROM float_map_keys_last_win WHERE id = 2
+
+-- 0.0 and -0.0: one key holding the last value from Spark 4.0, two keys before that.
+query expect_dispatch(map_from_arrays)
+SELECT map_keys(map_from_arrays(array(d, -d), array(1, 2))),
+ map_values(map_from_arrays(array(d, -d), array(1, 2))),
+ map_keys(map_from_arrays(array(f, -f), array(1, 2))),
+ map_values(map_from_arrays(array(f, -f), array(1, 2)))
+FROM float_map_keys_last_win WHERE id = 1
+
+query expect_dispatch(map_from_entries)
+SELECT map_keys(map_from_entries(array(named_struct('k', d, 'v', 1), named_struct('k', -d, 'v', 2)))),
+ map_values(map_from_entries(array(named_struct('k', d, 'v', 1), named_struct('k', -d, 'v', 2)))),
+ map_keys(map_from_entries(array(named_struct('k', f, 'v', 1), named_struct('k', -f, 'v', 2)))),
+ map_values(map_from_entries(array(named_struct('k', f, 'v', 1), named_struct('k', -f, 'v', 2))))
+FROM float_map_keys_last_win WHERE id = 1
+
+-- The stored key, when -0.0 is the first key: 0.0 from Spark 4.0 (normalized), -0.0 before that.
+query expect_dispatch(map_from_arrays)
+SELECT map_keys(map_from_arrays(array(-d, d), array(1, 2))),
+ map_values(map_from_arrays(array(-d, d), array(1, 2)))
+FROM float_map_keys_last_win WHERE id = 1
+
+query expect_dispatch(map_from_entries)
+SELECT map_keys(map_from_entries(array(named_struct('k', -d, 'v', 1), named_struct('k', d, 'v', 2)))),
+ map_values(map_from_entries(array(named_struct('k', -d, 'v', 1), named_struct('k', d, 'v', 2))))
+FROM float_map_keys_last_win WHERE id = 1
diff --git a/spark/src/test/resources/sql-tests/expressions/map/map_from_arrays_dedup_policy.sql b/spark/src/test/resources/sql-tests/expressions/map/map_from_arrays_dedup_policy.sql
index fffaf5f9a9..f1d4995246 100644
--- a/spark/src/test/resources/sql-tests/expressions/map/map_from_arrays_dedup_policy.sql
+++ b/spark/src/test/resources/sql-tests/expressions/map/map_from_arrays_dedup_policy.sql
@@ -17,10 +17,15 @@
-- Verifies that `map_from_arrays` falls back to Spark when `spark.sql.mapKeyDedupPolicy` is set
-- to `LAST_WIN`. Spark's ArrayBasedMapBuilder keeps the last occurrence of each duplicate key;
--- Comet's native `map` scalar has no LAST_WIN path, so it must fall back. The default `EXCEPTION`
--- mode agrees with Comet and is covered by `map_from_arrays.sql`.
+-- Comet's native `map` scalar has no LAST_WIN path. `CometMapFromArrays` mixes in
+-- `CodegenDispatchFallback`, so its native `Incompatible` normally routes through the JVM codegen
+-- dispatcher; we disable the dispatcher here so the incompat branch surfaces as a genuine Spark
+-- fallback rather than in-pipeline codegen. The dispatched path, with a key that really repeats, is
+-- covered by `map_builders_float_key_last_win.sql`. The default `EXCEPTION` mode agrees with Comet
+-- and is covered by `map_from_arrays.sql`.
-- Config: spark.sql.mapKeyDedupPolicy=LAST_WIN
+-- Config: spark.comet.exec.scalaUDF.codegen.enabled=false
statement
CREATE TABLE test_map_from_arrays_dedup(k array<string>, v array<int>) USING parquet
diff --git a/spark/src/test/resources/sql-tests/expressions/map/map_from_entries_dedup_policy.sql b/spark/src/test/resources/sql-tests/expressions/map/map_from_entries_dedup_policy.sql
index feba795193..73706cb1f4 100644
--- a/spark/src/test/resources/sql-tests/expressions/map/map_from_entries_dedup_policy.sql
+++ b/spark/src/test/resources/sql-tests/expressions/map/map_from_entries_dedup_policy.sql
@@ -19,8 +19,9 @@
-- to `LAST_WIN`. `CometMapFromEntries` mixes in `CodegenDispatchFallback`, so its native
-- `Incompatible` normally routes through the JVM codegen dispatcher; we disable the dispatcher
-- here so the incompat branch surfaces as a genuine Spark fallback rather than in-pipeline
--- codegen. The default `EXCEPTION` mode agrees with Comet and is covered by
--- `map_from_entries.sql`.
+-- codegen. The dispatched path, with a key that really repeats, is covered by
+-- `map_builders_float_key_last_win.sql`. The default `EXCEPTION` mode agrees with Comet and is
+-- covered by `map_from_entries.sql`.
-- Config: spark.sql.mapKeyDedupPolicy=LAST_WIN
-- Config: spark.comet.exec.scalaUDF.codegen.enabled=false
diff --git a/spark/src/test/scala/org/apache/comet/CometMapExpressionSuite.scala b/spark/src/test/scala/org/apache/comet/CometMapExpressionSuite.scala
index da07c05f67..6b8202d240 100644
--- a/spark/src/test/scala/org/apache/comet/CometMapExpressionSuite.scala
+++ b/spark/src/test/scala/org/apache/comet/CometMapExpressionSuite.scala
@@ -399,7 +399,7 @@ class CometMapExpressionSuite extends CometTestBase {
checkSparkAnswerAndFallbackReason(
"SELECT _1 AS id, element_at(element_at(map(1, map(CAST(0 AS DOUBLE), 7)), _1), " +
s"$negZero) AS v FROM tbl",
- "Spark normalizes floating-point map keys")
+ "Spark looks a key up in its SQL ordering")
}
}
@@ -489,7 +489,7 @@ class CometMapExpressionSuite extends CometTestBase {
}
// Direct single-level folded map with floating-point keys: `map(CAST(0 AS DOUBLE), 7)` folds to a
- // literal and `element_at` with a dynamic `-0.0` lookup must match Spark's `+0.0`-normalized key.
+ // literal and `element_at` with a dynamic `-0.0` lookup must match Spark's `+0.0` key.
// Native `map_extract` compares raw Arrow values, so `MapKeySupport` declines it at `element_at`.
// Spark returns 7, NULL, NULL. (`element_at_map.sql` covers the folding-off constructor form.)
test("folded map literal with floating-point keys in element_at falls back (multirow)") {
@@ -497,7 +497,7 @@ class CometMapExpressionSuite extends CometTestBase {
val lookup = "CAST(concat('-', CAST(_1 - 1 AS STRING), '.0') AS DOUBLE)"
checkSparkAnswerAndFallbackReason(
s"SELECT _1 AS id, element_at(map(CAST(0 AS DOUBLE), 7), $lookup) AS v FROM tbl",
- "Spark normalizes floating-point map keys")
+ "Spark looks a key up in its SQL ordering")
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org