You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/07/15 15:40:00 UTC

(datafusion) 01/01: fix: support type coercion for MAP literals with NULL values in VALUES lists (#23521)

This is an automated email from the ASF dual-hosted git repository.

github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-23521-0099876a3c0c6f0b17a97efbee6c790d0f2f10b3
in repository https://gitbox.apache.org/repos/asf/datafusion.git

commit cf139585f0db506c50d6020e361bb392ed515480
Author: Prateek Ganigi <91...@users.noreply.github.com>
AuthorDate: Wed Jul 15 08:39:36 2026 -0700

    fix: support type coercion for MAP literals with NULL values in VALUES lists (#23521)
    
    ## Which issue does this PR close?
    
    - Closes #23474.
    
    ## Rationale for this change
    
    A bare `MAP {'k1': NULL}` literal infers a `Null` value type, producing
    `Map("entries": Struct("key": Utf8, "value": Null))`. Placing it in a
    VALUES list alongside concretely-typed map rows failed with:
    
    Error during planning: Inconsistent data type across values list at row
    1 column 0.
    
    This is because `type_union_resolution_coercion` (the pairwise worker
    behind `type_union_resolution`, which the VALUES builder uses to widen
    row types) had recursion arms for `Dictionary`, `RunEndEncoded`,
    `Struct`, and `List` (via `list_coercion`), but none for `Map`. Scalar
    NULL inference across a VALUES list (`VALUES (1), (NULL)`) already
    worked; the same inference just never reached into a map's value type.
    This gap was noted in #23406, where the tests had to use a `CAST(NULL AS
    BIGINT)` workaround.
    
    ## What changes are included in this PR?
    
    - A one-line fix in
    `datafusion/expr-common/src/type_coercion/binary.rs`: wire the existing
    `map_coercion` helper (already used by `comparison_coercion` and
    `type_union_coercion`) into `type_union_resolution_coercion`'s fallback
    chain. The recursion into the map's key/value types then applies the
    exact same rules as scalar VALUES coercion (`null_coercion` for NULL
    inference, `binary_numeric_coercion` for numeric widening),
    symmetrically in either row order.
    - A unit test (`test_type_union_resolution_map`) covering Null + Int64
    value types in both orders, Int64 + Null + Float64 widening, and Map vs.
    Struct still refusing to unify.
    - A new "map NULL value coercion in VALUES" section in
    `datafusion/sqllogictest/test_files/map.slt` covering: the issue
    reproducer, the reversed row order, all-NULL values (resulting type
    `Map("entries": Struct("key": Utf8, "value": Null))`, verified via
    `arrow_typeof`), multi-key rows with one NULL value, NULL scattered
    across three rows, numeric widening to Float64, two-level nested maps,
    NULL round-tripping as NULL through `SELECT`, and `INSERT INTO ...
    VALUES`.
    - Incompatible concrete value types still error: `(MAP {'k': 1}), (MAP
    {'k': 'hello'})` follows the scalar VALUES rule (coerce to the numeric
    type, then fail with `Cast error: Cannot cast string 'hello' to value of
    Int64 type`, which is the same behavior as `VALUES (1), ('hello')`).
    - A note above the #23406 tests documenting that the `CAST(NULL AS
    BIGINT)` workaround is no longer required (those tests are left
    unchanged).
    
    ## Are these changes tested?
    
    Yes. New unit test in `datafusion-expr-common` and new sqllogictests in
    `map.slt` as described above. The full sqllogictest suite (492 files)
    passes locally with zero regressions, as do `cargo test -p
    datafusion-expr -p datafusion-expr-common`, `cargo fmt`, and `cargo
    clippy --workspace --all-targets -- -D warnings`.
    
    ## Are there any user-facing changes?
    
    `VALUES` lists (and other contexts that use `type_union_resolution`,
    such as `CASE`/`COALESCE`/array literals) now unify Map types whose
    key/value types are coercible, including bare NULL map values. No API
    changes; previously failing queries now succeed, and incompatible-type
    errors are preserved.
    
    ---------
    
    Co-authored-by: Jeffrey Vo <je...@gmail.com>
---
 datafusion/expr-common/src/type_coercion/binary.rs |   1 +
 .../src/type_coercion/binary/tests/comparison.rs   |  54 ++++++++
 datafusion/sqllogictest/test_files/map.slt         | 145 +++++++++++++++++++++
 3 files changed, 200 insertions(+)

diff --git a/datafusion/expr-common/src/type_coercion/binary.rs b/datafusion/expr-common/src/type_coercion/binary.rs
index c7a73a7c6c..23ccf7f815 100644
--- a/datafusion/expr-common/src/type_coercion/binary.rs
+++ b/datafusion/expr-common/src/type_coercion/binary.rs
@@ -768,6 +768,7 @@ fn type_union_resolution_coercion(
         }
         _ => binary_numeric_coercion(lhs_type, rhs_type)
             .or_else(|| list_coercion(lhs_type, rhs_type, type_union_resolution_coercion))
+            .or_else(|| map_coercion(lhs_type, rhs_type, type_union_resolution_coercion))
             .or_else(|| temporal_coercion_nonstrict_timezone(lhs_type, rhs_type))
             .or_else(|| string_coercion(lhs_type, rhs_type))
             .or_else(|| null_coercion(lhs_type, rhs_type))
diff --git a/datafusion/expr-common/src/type_coercion/binary/tests/comparison.rs b/datafusion/expr-common/src/type_coercion/binary/tests/comparison.rs
index 5871f24e7f..cfa3bbe189 100644
--- a/datafusion/expr-common/src/type_coercion/binary/tests/comparison.rs
+++ b/datafusion/expr-common/src/type_coercion/binary/tests/comparison.rs
@@ -908,6 +908,60 @@ fn test_type_union_coercion_prefers_finer_timestamp_unit() {
     );
 }
 
+/// Tests that `type_union_resolution` unifies Map types by recursing into the
+/// key/value types, so a Map whose value type is Null (e.g. `MAP {'k': NULL}`)
+/// unifies with a concretely-typed Map in a VALUES list.
+/// See <https://github.com/apache/datafusion/issues/23474>.
+#[test]
+fn test_type_union_resolution_map() {
+    fn map_type(value_type: DataType) -> DataType {
+        DataType::Map(
+            Arc::new(Field::new(
+                "entries",
+                DataType::Struct(Fields::from(vec![
+                    Field::new("key", DataType::Utf8, false),
+                    Field::new("value", value_type, true),
+                ])),
+                false,
+            )),
+            false,
+        )
+    }
+
+    // Null value type unifies with a concrete value type, in both orders
+    assert_eq!(
+        type_union_resolution(&[map_type(DataType::Int64), map_type(DataType::Null)]),
+        Some(map_type(DataType::Int64))
+    );
+    assert_eq!(
+        type_union_resolution(&[map_type(DataType::Null), map_type(DataType::Int64)]),
+        Some(map_type(DataType::Int64))
+    );
+
+    // Numeric value types widen following the scalar rules
+    assert_eq!(
+        type_union_resolution(&[
+            map_type(DataType::Int64),
+            map_type(DataType::Null),
+            map_type(DataType::Float64),
+        ]),
+        Some(map_type(DataType::Float64))
+    );
+
+    // Map cannot unify with a non-Map composite type
+    assert_eq!(
+        type_union_resolution(&[
+            map_type(DataType::Int64),
+            DataType::Struct(Fields::from(vec![Field::new(
+                "key",
+                DataType::Utf8,
+                false
+            )])),
+        ]),
+        None
+    );
+}
+
 /// Tests that comparison operators coerce to numeric when comparing
 /// numeric and string types.
 #[test]
diff --git a/datafusion/sqllogictest/test_files/map.slt b/datafusion/sqllogictest/test_files/map.slt
index f8cbb395cb..486a50f960 100644
--- a/datafusion/sqllogictest/test_files/map.slt
+++ b/datafusion/sqllogictest/test_files/map.slt
@@ -920,6 +920,10 @@ SELECT map([column1, column1 * 10], ['x','y']) FROM (VALUES (1), (2), (3)) t;
 # tests for DISTINCT / GROUP BY / aggregation on map columns
 # https://github.com/apache/datafusion/issues/15428
 
+# NOTE: the CAST(NULL AS BIGINT) in the VALUES list below predates the fix for
+# https://github.com/apache/datafusion/issues/23474 and is no longer required.
+# It is kept as-is to document the historical workaround; the un-cast form is
+# exercised in the "map NULL value coercion in VALUES" section further below.
 statement ok
 CREATE TABLE map_distinct_table AS VALUES
   (MAP {'k1': 1, 'k2': 2}, 'a', 1),
@@ -1048,3 +1052,144 @@ PUT 25
 
 statement ok
 DROP TABLE map_data;
+
+# map NULL value coercion in VALUES
+# https://github.com/apache/datafusion/issues/23474
+# A bare NULL map value used to fail type unification across a VALUES list
+# ("Inconsistent data type across values list") and required an explicit
+# CAST(NULL AS <type>). The map value type now unifies with concrete value
+# types following the same rules as scalar VALUES coercion.
+
+# concrete-typed row first, NULL-valued row second (the issue reproducer)
+statement ok
+CREATE TABLE map_null_concrete_first AS VALUES
+  (MAP {'k1': 1, 'k2': 2}),
+  (MAP {'k1': NULL});
+
+# NULL must round-trip as NULL after coercion, not a default value
+query ? rowsort
+SELECT * FROM map_null_concrete_first;
+----
+{k1: 1, k2: 2}
+{k1: NULL}
+
+# the NULL value type is coerced to the concrete value type (Int64)
+query T
+SELECT arrow_typeof(column1) FROM map_null_concrete_first LIMIT 1;
+----
+Map("entries": non-null Struct("key": non-null Utf8, "value": Int64), unsorted)
+
+statement ok
+DROP TABLE map_null_concrete_first;
+
+# NULL-valued row first, concrete-typed row second (coercion is symmetric)
+statement ok
+CREATE TABLE map_null_first AS VALUES
+  (MAP {'k1': NULL}),
+  (MAP {'k1': 1, 'k2': 2});
+
+query ? rowsort
+SELECT * FROM map_null_first;
+----
+{k1: 1, k2: 2}
+{k1: NULL}
+
+statement ok
+DROP TABLE map_null_first;
+
+# every row has a NULL value: succeeds and the value type stays Null
+statement ok
+CREATE TABLE map_all_null_values AS VALUES
+  (MAP {'k': NULL}),
+  (MAP {'k': NULL});
+
+query ? rowsort
+SELECT * FROM map_all_null_values;
+----
+{k: NULL}
+{k: NULL}
+
+query T
+SELECT arrow_typeof(column1) FROM map_all_null_values LIMIT 1;
+----
+Map("entries": non-null Struct("key": non-null Utf8, "value": Null), unsorted)
+
+statement ok
+DROP TABLE map_all_null_values;
+
+# multiple keys where only one value is NULL
+query ? rowsort
+SELECT * FROM (VALUES
+  (MAP {'a': 1, 'b': NULL}),
+  (MAP {'a': 2, 'b': 3})) t(column1);
+----
+{a: 1, b: NULL}
+{a: 2, b: 3}
+
+# three rows with the NULL-valued row in the middle
+query ? rowsort
+SELECT * FROM (VALUES
+  (MAP {'k': 1}),
+  (MAP {'k': NULL}),
+  (MAP {'k': 2})) t(column1);
+----
+{k: 1}
+{k: 2}
+{k: NULL}
+
+# numeric widening across a NULL-valued row follows the scalar rule
+# (Int64 + Float64 -> Float64)
+statement ok
+CREATE TABLE map_null_widening AS VALUES
+  (MAP {'k': 1}),
+  (MAP {'k': NULL}),
+  (MAP {'k': 1.5});
+
+query ? rowsort
+SELECT * FROM map_null_widening;
+----
+{k: 1.0}
+{k: 1.5}
+{k: NULL}
+
+query T
+SELECT arrow_typeof(column1) FROM map_null_widening LIMIT 1;
+----
+Map("entries": non-null Struct("key": non-null Utf8, "value": Float64), unsorted)
+
+statement ok
+DROP TABLE map_null_widening;
+
+# incompatible concrete value types with a NULL-valued row in between still
+# error; Int64/Utf8 follows the scalar VALUES rule (coerce to the numeric
+# type, then fail to cast the non-numeric string)
+query error Cast error: Cannot cast string 'hello' to value of Int64 type
+SELECT * FROM (VALUES
+  (MAP {'k': 1}),
+  (MAP {'k': NULL}),
+  (MAP {'k': 'hello'})) t(column1);
+
+# NULL value type unification recurses into nested maps
+query ? rowsort
+SELECT * FROM (VALUES
+  (MAP {'outer': MAP {'inner': 1}}),
+  (MAP {'outer': MAP {'inner': NULL}})) t(column1);
+----
+{outer: {inner: 1}}
+{outer: {inner: NULL}}
+
+# INSERT INTO ... VALUES also accepts a NULL map value without a cast
+statement ok
+CREATE TABLE map_null_insert AS VALUES (MAP {'k1': 1, 'k2': 2});
+
+statement ok
+INSERT INTO map_null_insert VALUES (MAP {'k1': NULL});
+
+query ? rowsort
+SELECT * FROM map_null_insert;
+----
+{k1: 1, k2: 2}
+{k1: NULL}
+
+statement ok
+DROP TABLE map_null_insert;


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org