You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/07/15 15:40:00 UTC
(datafusion) 01/01: fix: support type coercion for MAP literals with NULL values in VALUES lists (#23521)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-23521-0099876a3c0c6f0b17a97efbee6c790d0f2f10b3
in repository https://gitbox.apache.org/repos/asf/datafusion.git
commit cf139585f0db506c50d6020e361bb392ed515480
Author: Prateek Ganigi <91...@users.noreply.github.com>
AuthorDate: Wed Jul 15 08:39:36 2026 -0700
fix: support type coercion for MAP literals with NULL values in VALUES lists (#23521)
## Which issue does this PR close?
- Closes #23474.
## Rationale for this change
A bare `MAP {'k1': NULL}` literal infers a `Null` value type, producing
`Map("entries": Struct("key": Utf8, "value": Null))`. Placing it in a
VALUES list alongside concretely-typed map rows failed with:
Error during planning: Inconsistent data type across values list at row
1 column 0.
This is because `type_union_resolution_coercion` (the pairwise worker
behind `type_union_resolution`, which the VALUES builder uses to widen
row types) had recursion arms for `Dictionary`, `RunEndEncoded`,
`Struct`, and `List` (via `list_coercion`), but none for `Map`. Scalar
NULL inference across a VALUES list (`VALUES (1), (NULL)`) already
worked; the same inference just never reached into a map's value type.
This gap was noted in #23406, where the tests had to use a `CAST(NULL AS
BIGINT)` workaround.
## What changes are included in this PR?
- A one-line fix in
`datafusion/expr-common/src/type_coercion/binary.rs`: wire the existing
`map_coercion` helper (already used by `comparison_coercion` and
`type_union_coercion`) into `type_union_resolution_coercion`'s fallback
chain. The recursion into the map's key/value types then applies the
exact same rules as scalar VALUES coercion (`null_coercion` for NULL
inference, `binary_numeric_coercion` for numeric widening),
symmetrically in either row order.
- A unit test (`test_type_union_resolution_map`) covering Null + Int64
value types in both orders, Int64 + Null + Float64 widening, and Map vs.
Struct still refusing to unify.
- A new "map NULL value coercion in VALUES" section in
`datafusion/sqllogictest/test_files/map.slt` covering: the issue
reproducer, the reversed row order, all-NULL values (resulting type
`Map("entries": Struct("key": Utf8, "value": Null))`, verified via
`arrow_typeof`), multi-key rows with one NULL value, NULL scattered
across three rows, numeric widening to Float64, two-level nested maps,
NULL round-tripping as NULL through `SELECT`, and `INSERT INTO ...
VALUES`.
- Incompatible concrete value types still error: `(MAP {'k': 1}), (MAP
{'k': 'hello'})` follows the scalar VALUES rule (coerce to the numeric
type, then fail with `Cast error: Cannot cast string 'hello' to value of
Int64 type`, which is the same behavior as `VALUES (1), ('hello')`).
- A note above the #23406 tests documenting that the `CAST(NULL AS
BIGINT)` workaround is no longer required (those tests are left
unchanged).
## Are these changes tested?
Yes. New unit test in `datafusion-expr-common` and new sqllogictests in
`map.slt` as described above. The full sqllogictest suite (492 files)
passes locally with zero regressions, as do `cargo test -p
datafusion-expr -p datafusion-expr-common`, `cargo fmt`, and `cargo
clippy --workspace --all-targets -- -D warnings`.
## Are there any user-facing changes?
`VALUES` lists (and other contexts that use `type_union_resolution`,
such as `CASE`/`COALESCE`/array literals) now unify Map types whose
key/value types are coercible, including bare NULL map values. No API
changes; previously failing queries now succeed, and incompatible-type
errors are preserved.
---------
Co-authored-by: Jeffrey Vo <je...@gmail.com>
---
datafusion/expr-common/src/type_coercion/binary.rs | 1 +
.../src/type_coercion/binary/tests/comparison.rs | 54 ++++++++
datafusion/sqllogictest/test_files/map.slt | 145 +++++++++++++++++++++
3 files changed, 200 insertions(+)
diff --git a/datafusion/expr-common/src/type_coercion/binary.rs b/datafusion/expr-common/src/type_coercion/binary.rs
index c7a73a7c6c..23ccf7f815 100644
--- a/datafusion/expr-common/src/type_coercion/binary.rs
+++ b/datafusion/expr-common/src/type_coercion/binary.rs
@@ -768,6 +768,7 @@ fn type_union_resolution_coercion(
}
_ => binary_numeric_coercion(lhs_type, rhs_type)
.or_else(|| list_coercion(lhs_type, rhs_type, type_union_resolution_coercion))
+ .or_else(|| map_coercion(lhs_type, rhs_type, type_union_resolution_coercion))
.or_else(|| temporal_coercion_nonstrict_timezone(lhs_type, rhs_type))
.or_else(|| string_coercion(lhs_type, rhs_type))
.or_else(|| null_coercion(lhs_type, rhs_type))
diff --git a/datafusion/expr-common/src/type_coercion/binary/tests/comparison.rs b/datafusion/expr-common/src/type_coercion/binary/tests/comparison.rs
index 5871f24e7f..cfa3bbe189 100644
--- a/datafusion/expr-common/src/type_coercion/binary/tests/comparison.rs
+++ b/datafusion/expr-common/src/type_coercion/binary/tests/comparison.rs
@@ -908,6 +908,60 @@ fn test_type_union_coercion_prefers_finer_timestamp_unit() {
);
}
+/// Tests that `type_union_resolution` unifies Map types by recursing into the
+/// key/value types, so a Map whose value type is Null (e.g. `MAP {'k': NULL}`)
+/// unifies with a concretely-typed Map in a VALUES list.
+/// See <https://github.com/apache/datafusion/issues/23474>.
+#[test]
+fn test_type_union_resolution_map() {
+ fn map_type(value_type: DataType) -> DataType {
+ DataType::Map(
+ Arc::new(Field::new(
+ "entries",
+ DataType::Struct(Fields::from(vec![
+ Field::new("key", DataType::Utf8, false),
+ Field::new("value", value_type, true),
+ ])),
+ false,
+ )),
+ false,
+ )
+ }
+
+ // Null value type unifies with a concrete value type, in both orders
+ assert_eq!(
+ type_union_resolution(&[map_type(DataType::Int64), map_type(DataType::Null)]),
+ Some(map_type(DataType::Int64))
+ );
+ assert_eq!(
+ type_union_resolution(&[map_type(DataType::Null), map_type(DataType::Int64)]),
+ Some(map_type(DataType::Int64))
+ );
+
+ // Numeric value types widen following the scalar rules
+ assert_eq!(
+ type_union_resolution(&[
+ map_type(DataType::Int64),
+ map_type(DataType::Null),
+ map_type(DataType::Float64),
+ ]),
+ Some(map_type(DataType::Float64))
+ );
+
+ // Map cannot unify with a non-Map composite type
+ assert_eq!(
+ type_union_resolution(&[
+ map_type(DataType::Int64),
+ DataType::Struct(Fields::from(vec![Field::new(
+ "key",
+ DataType::Utf8,
+ false
+ )])),
+ ]),
+ None
+ );
+}
+
/// Tests that comparison operators coerce to numeric when comparing
/// numeric and string types.
#[test]
diff --git a/datafusion/sqllogictest/test_files/map.slt b/datafusion/sqllogictest/test_files/map.slt
index f8cbb395cb..486a50f960 100644
--- a/datafusion/sqllogictest/test_files/map.slt
+++ b/datafusion/sqllogictest/test_files/map.slt
@@ -920,6 +920,10 @@ SELECT map([column1, column1 * 10], ['x','y']) FROM (VALUES (1), (2), (3)) t;
# tests for DISTINCT / GROUP BY / aggregation on map columns
# https://github.com/apache/datafusion/issues/15428
+# NOTE: the CAST(NULL AS BIGINT) in the VALUES list below predates the fix for
+# https://github.com/apache/datafusion/issues/23474 and is no longer required.
+# It is kept as-is to document the historical workaround; the un-cast form is
+# exercised in the "map NULL value coercion in VALUES" section further below.
statement ok
CREATE TABLE map_distinct_table AS VALUES
(MAP {'k1': 1, 'k2': 2}, 'a', 1),
@@ -1048,3 +1052,144 @@ PUT 25
statement ok
DROP TABLE map_data;
+
+# map NULL value coercion in VALUES
+# https://github.com/apache/datafusion/issues/23474
+# A bare NULL map value used to fail type unification across a VALUES list
+# ("Inconsistent data type across values list") and required an explicit
+# CAST(NULL AS <type>). The map value type now unifies with concrete value
+# types following the same rules as scalar VALUES coercion.
+
+# concrete-typed row first, NULL-valued row second (the issue reproducer)
+statement ok
+CREATE TABLE map_null_concrete_first AS VALUES
+ (MAP {'k1': 1, 'k2': 2}),
+ (MAP {'k1': NULL});
+
+# NULL must round-trip as NULL after coercion, not a default value
+query ? rowsort
+SELECT * FROM map_null_concrete_first;
+----
+{k1: 1, k2: 2}
+{k1: NULL}
+
+# the NULL value type is coerced to the concrete value type (Int64)
+query T
+SELECT arrow_typeof(column1) FROM map_null_concrete_first LIMIT 1;
+----
+Map("entries": non-null Struct("key": non-null Utf8, "value": Int64), unsorted)
+
+statement ok
+DROP TABLE map_null_concrete_first;
+
+# NULL-valued row first, concrete-typed row second (coercion is symmetric)
+statement ok
+CREATE TABLE map_null_first AS VALUES
+ (MAP {'k1': NULL}),
+ (MAP {'k1': 1, 'k2': 2});
+
+query ? rowsort
+SELECT * FROM map_null_first;
+----
+{k1: 1, k2: 2}
+{k1: NULL}
+
+statement ok
+DROP TABLE map_null_first;
+
+# every row has a NULL value: succeeds and the value type stays Null
+statement ok
+CREATE TABLE map_all_null_values AS VALUES
+ (MAP {'k': NULL}),
+ (MAP {'k': NULL});
+
+query ? rowsort
+SELECT * FROM map_all_null_values;
+----
+{k: NULL}
+{k: NULL}
+
+query T
+SELECT arrow_typeof(column1) FROM map_all_null_values LIMIT 1;
+----
+Map("entries": non-null Struct("key": non-null Utf8, "value": Null), unsorted)
+
+statement ok
+DROP TABLE map_all_null_values;
+
+# multiple keys where only one value is NULL
+query ? rowsort
+SELECT * FROM (VALUES
+ (MAP {'a': 1, 'b': NULL}),
+ (MAP {'a': 2, 'b': 3})) t(column1);
+----
+{a: 1, b: NULL}
+{a: 2, b: 3}
+
+# three rows with the NULL-valued row in the middle
+query ? rowsort
+SELECT * FROM (VALUES
+ (MAP {'k': 1}),
+ (MAP {'k': NULL}),
+ (MAP {'k': 2})) t(column1);
+----
+{k: 1}
+{k: 2}
+{k: NULL}
+
+# numeric widening across a NULL-valued row follows the scalar rule
+# (Int64 + Float64 -> Float64)
+statement ok
+CREATE TABLE map_null_widening AS VALUES
+ (MAP {'k': 1}),
+ (MAP {'k': NULL}),
+ (MAP {'k': 1.5});
+
+query ? rowsort
+SELECT * FROM map_null_widening;
+----
+{k: 1.0}
+{k: 1.5}
+{k: NULL}
+
+query T
+SELECT arrow_typeof(column1) FROM map_null_widening LIMIT 1;
+----
+Map("entries": non-null Struct("key": non-null Utf8, "value": Float64), unsorted)
+
+statement ok
+DROP TABLE map_null_widening;
+
+# incompatible concrete value types with a NULL-valued row in between still
+# error; Int64/Utf8 follows the scalar VALUES rule (coerce to the numeric
+# type, then fail to cast the non-numeric string)
+query error Cast error: Cannot cast string 'hello' to value of Int64 type
+SELECT * FROM (VALUES
+ (MAP {'k': 1}),
+ (MAP {'k': NULL}),
+ (MAP {'k': 'hello'})) t(column1);
+
+# NULL value type unification recurses into nested maps
+query ? rowsort
+SELECT * FROM (VALUES
+ (MAP {'outer': MAP {'inner': 1}}),
+ (MAP {'outer': MAP {'inner': NULL}})) t(column1);
+----
+{outer: {inner: 1}}
+{outer: {inner: NULL}}
+
+# INSERT INTO ... VALUES also accepts a NULL map value without a cast
+statement ok
+CREATE TABLE map_null_insert AS VALUES (MAP {'k1': 1, 'k2': 2});
+
+statement ok
+INSERT INTO map_null_insert VALUES (MAP {'k1': NULL});
+
+query ? rowsort
+SELECT * FROM map_null_insert;
+----
+{k1: 1, k2: 2}
+{k1: NULL}
+
+statement ok
+DROP TABLE map_null_insert;
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org