You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/04/21 05:36:10 UTC
(datafusion) branch gh-readonly-queue/main/pr-21704-c470bb19340f4f3fa00751c2ba6313425bc5edd7 created (now e524f49643)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-21704-c470bb19340f4f3fa00751c2ba6313425bc5edd7
in repository https://gitbox.apache.org/repos/asf/datafusion.git
at e524f49643 fix: array_concat widens container variant for mixed List/LargeList inputs (#21704)
This branch includes the following new commits:
new e524f49643 fix: array_concat widens container variant for mixed List/LargeList inputs (#21704)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion) 01/01: fix: array_concat widens container variant for mixed List/LargeList inputs (#21704)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-21704-c470bb19340f4f3fa00751c2ba6313425bc5edd7
in repository https://gitbox.apache.org/repos/asf/datafusion.git
commit e524f4964344e949e232e08d6e51710685d50871
Author: Harrison Crosse <ha...@crosse.dev>
AuthorDate: Tue Apr 21 01:35:47 2026 -0400
fix: array_concat widens container variant for mixed List/LargeList inputs (#21704)
## Which issue does this PR close?
- Closes #21702.
## Rationale for this change
`array_concat` hit an internal cast error when given a mix of `List` and
`LargeList` (or `FixedSizeList` and `LargeList`) arguments:
```sql
> select array_concat(make_array(1, 2), arrow_cast([3, 4], 'LargeList(Int64)'));
DataFusion error: Internal error: could not cast array of type List(Int64) to arrow_array::array::list_array::GenericListArray<i64>.
```
`ArrayConcat::coerce_types` was coercing only the base element type,
leaving the outer container alone. When the resolved return type is
`LargeList`, `array_concat_inner` later tries to downcast each arg to
`GenericListArray<i64>`, which fails for any `List` argument that
slipped through.
## What changes are included in this PR?
In `ArrayConcat::coerce_types`, after coercing the base type, also
promote each input's outermost `List` to `LargeList` when the return
type is a `LargeList`. `FixedSizeList` inputs already go through
`FixedSizedListToList` first and then get promoted too. Per-arg
dimensionality is preserved, so nested cases keep working with
`align_array_dimensions`.
## Are these changes tested?
Yes, added sqllogictests in `array_concat.slt` covering:
- `List` + `LargeList`
- `LargeList` + `List`
- `FixedSizeList` + `LargeList`
- Three-way mix `List`, `LargeList`, `List`
Each one also asserts `arrow_typeof(...) = LargeList(Int64)`.
## Are there any user-facing changes?
Queries that previously returned an internal cast error now return the
concatenated `LargeList` as expected. No API changes.
---
datafusion/functions-nested/src/concat.rs | 17 ++++++++++--
.../sqllogictest/test_files/array/array_concat.slt | 32 ++++++++++++++++++++++
2 files changed, 47 insertions(+), 2 deletions(-)
diff --git a/datafusion/functions-nested/src/concat.rs b/datafusion/functions-nested/src/concat.rs
index c3dc4c67cf..8d06140889 100644
--- a/datafusion/functions-nested/src/concat.rs
+++ b/datafusion/functions-nested/src/concat.rs
@@ -317,10 +317,23 @@ impl ScalarUDFImpl for ArrayConcat {
}
fn coerce_types(&self, arg_types: &[DataType]) -> Result<Vec<DataType>> {
- let base_type = base_type(&self.return_type(arg_types)?);
+ let return_type = self.return_type(arg_types)?;
+ let base_type = base_type(&return_type);
let coercion = Some(&ListCoercion::FixedSizedListToList);
+ // When the return type is a `LargeList`, the outer container of every
+ // input must be widened to `LargeList` as well. Otherwise
+ // `array_concat_inner` would later try to downcast a `List` argument
+ // to `GenericListArray<i64>` and fail.
+ let promote_to_large_list = matches!(return_type, DataType::LargeList(_));
let arg_types = arg_types.iter().map(|arg_type| {
- coerced_type_with_base_type_only(arg_type, &base_type, coercion)
+ let coerced =
+ coerced_type_with_base_type_only(arg_type, &base_type, coercion);
+ match coerced {
+ DataType::List(field) if promote_to_large_list => {
+ DataType::LargeList(field)
+ }
+ other => other,
+ }
});
Ok(arg_types.collect())
diff --git a/datafusion/sqllogictest/test_files/array/array_concat.slt b/datafusion/sqllogictest/test_files/array/array_concat.slt
index 0f84781161..168b307a1e 100644
--- a/datafusion/sqllogictest/test_files/array/array_concat.slt
+++ b/datafusion/sqllogictest/test_files/array/array_concat.slt
@@ -121,6 +121,38 @@ select
----
[1, 2, 3] List(Utf8View)
+# Concatenating mixed list and large list — return type widens to LargeList
+query ?T
+select
+ array_concat(make_array(1, 2), arrow_cast([3, 4], 'LargeList(Int64)')),
+ arrow_typeof(array_concat(make_array(1, 2), arrow_cast([3, 4], 'LargeList(Int64)')));
+----
+[1, 2, 3, 4] LargeList(Int64)
+
+# Reverse argument order: LargeList first, plain list second
+query ?T
+select
+ array_concat(arrow_cast([1, 2], 'LargeList(Int64)'), make_array(3, 4)),
+ arrow_typeof(array_concat(arrow_cast([1, 2], 'LargeList(Int64)'), make_array(3, 4)));
+----
+[1, 2, 3, 4] LargeList(Int64)
+
+# FixedSizeList combined with LargeList — also widens to LargeList
+query ?T
+select
+ array_concat(arrow_cast([1, 2], 'FixedSizeList(2, Int64)'), arrow_cast([3, 4], 'LargeList(Int64)')),
+ arrow_typeof(array_concat(arrow_cast([1, 2], 'FixedSizeList(2, Int64)'), arrow_cast([3, 4], 'LargeList(Int64)')));
+----
+[1, 2, 3, 4] LargeList(Int64)
+
+# Three-way mix: List, LargeList, List
+query ?T
+select
+ array_concat(make_array(1, 2), arrow_cast([3], 'LargeList(Int64)'), make_array(4, 5)),
+ arrow_typeof(array_concat(make_array(1, 2), arrow_cast([3], 'LargeList(Int64)'), make_array(4, 5)));
+----
+[1, 2, 3, 4, 5] LargeList(Int64)
+
# array_concat with NULL elements inside arrays
query ?
select array_concat([1, NULL, 3], [NULL, 5]);
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org