You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Jefffrey (via GitHub)" <gi...@apache.org> on 2025/09/19 14:00:33 UTC
[I] Support array `flatten()` on `List(LargeList(_))` types [datafusion]
Jefffrey opened a new issue, #17670:
URL: https://github.com/apache/datafusion/issues/17670
### Is your feature request related to a problem or challenge?
Currently cannot flatten a `List` (or `FixedSizeList`) that contains a `LargeList` as it's inner element.
We should be able to support this (at least for queries we expect to succeed). For example, expect something like this to succeed in `array.slt`:
```sql
query ???
select flatten(arrow_cast(make_array([1], [2, 3], [null], make_array(4, null, 5)), 'FixedSizeList(4, LargeList(Int64))')),
flatten(arrow_cast(make_array([[1.1], [2.2]], [[3.3], [4.4]]), 'List(LargeList(FixedSizeList(1, Float64)))'));
----
[1, 2, 1, 3, 2] [1, 2, 3, NULL, 4, NULL, 5] [[1.1], [2.2], [3.3], [4.4]]
```
### Describe the solution you'd like
Need to consider return type, see how `LargeList` is missing here for the inner `field` match:
https://github.com/apache/datafusion/blob/35c1cfdd151b14bbba87dbdf03d49afd9f4718a6/datafusion/functions-nested/src/flatten.rs#L107-L110
This is where current error is happening:
https://github.com/apache/datafusion/blob/35c1cfdd151b14bbba87dbdf03d49afd9f4718a6/datafusion/functions-nested/src/flatten.rs#L166-L168
- We just throw error without trying to see if it's possible
Perhaps we can try some sort of "best effort" where we try to downcast the `LargeList` child to a `List` and if that succeeds (i.e. all offsets of `LargeList` can fit inside a `List`) we can flatten it to the parent `List`, otherwise error; alternatively just upcast the parent `List` to a `LargeList`, though this might be tricky considering `return_type()` wouldn't know this until execution and I don't think we want to blindly upcast all parent `List`s to `LargeList`.
Open to any other suggestions.
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support array `flatten()` on `List(LargeList(_))` types [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #17670: Support array `flatten()` on `List(LargeList(_))` types
URL: https://github.com/apache/datafusion/issues/17670
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support array `flatten()` on `List(LargeList(_))` types [datafusion]
Posted by "sdf-jkl (via GitHub)" <gi...@apache.org>.
sdf-jkl commented on issue #17670:
URL: https://github.com/apache/datafusion/issues/17670#issuecomment-3452657889
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support array `flatten()` on `List(LargeList(_))` types [datafusion]
Posted by "sdf-jkl (via GitHub)" <gi...@apache.org>.
sdf-jkl commented on issue #17670:
URL: https://github.com/apache/datafusion/issues/17670#issuecomment-3463655811
Hey @Jefffrey, I came up with an incomplete solution that covers the testcase you provided earlier, but it does not account for offsets bigger than `i32::MAX`, that cannot be converted to `i32`.
The `return_type` also currently only accommodates the given testcase.
Once we come up with a way to add the needed test coverage, I will proceed to implementing the `return_type`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org