You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "universalmind303 (via GitHub)" <gi...@apache.org> on 2023/11/07 19:09:23 UTC
[I] array functions don't work with `FixedSizeList` [arrow-datafusion]
universalmind303 opened a new issue, #8084:
URL: https://github.com/apache/arrow-datafusion/issues/8084
### Describe the bug
none of the scalar `array_` functions work on `FixedSizeList` dtype.
### To Reproduce
given a table of
```
+-----------------+--------+
| fixed_size_list | list |
+-----------------+--------+
| [0, 1] | [0, 1] |
| [0, 2] | [0, 2] |
+-----------------+--------+
```
I should be able to use `array_*` functions on both `list` and `fixed_size_list` fields
```sql
SELECT array_length(list) FROM my_table
+-----------------------------+
| array_length(my_table.list) |
+-----------------------------+
| 2 |
| 2 |
+-----------------------------+
```
Full reproducible example here: https://gist.github.com/universalmind303/e2e59cd6d84656f5f20e78bd6e9c6c74
### Expected behavior
```sql
SELECT array_length(fixed_size_list) FROM my_table
+----------------------------------------+
| array_length(my_table.fixed_size_list) |
+----------------------------------------+
| 2 |
| 2 |
+----------------------------------------+
```
### Additional context
I don't think **all** of them should work. Such as ones that mutate the array (`repeat`, `remove`, ...). But I'd expect many of the immutable functions to work such as `has`, `length`, `element`, and so on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] array functions don't work with `FixedSizeList` [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8084:
URL: https://github.com/apache/arrow-datafusion/issues/8084#issuecomment-1802788231
> Yes, I believe the array function now only accepts List. I am happy to take this ticket.
Thanks @Weijun-H
In order to make this project tractable, I suggest picking one array function (ideally a simple one) and figuring out a good pattern for:
1. Testing
2. Implementing
In particular, I think this may require writing some generic code (over OffsetSize) trait
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] array functions don't work with `FixedSizeList` [datafusion]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on issue #8084:
URL: https://github.com/apache/datafusion/issues/8084#issuecomment-3310682832
I plan to go through the array functions in datafusion-functions-nested and check this support as well as tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] array functions don't work with `FixedSizeList` [arrow-datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on issue #8084:
URL: https://github.com/apache/arrow-datafusion/issues/8084#issuecomment-1802890110
For LargeList, you can take https://github.com/apache/arrow-datafusion/pull/7897 for reference
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] array functions don't work with `FixedSizeList` [datafusion]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey closed issue #8084: array functions don't work with `FixedSizeList`
URL: https://github.com/apache/datafusion/issues/8084
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] array functions don't work with `FixedSizeList` [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8084:
URL: https://github.com/apache/arrow-datafusion/issues/8084#issuecomment-1800250248
The same thing probably goes for `LargeList` as well (it isn't supported)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] array functions don't work with `FixedSizeList` [arrow-datafusion]
Posted by "Weijun-H (via GitHub)" <gi...@apache.org>.
Weijun-H commented on issue #8084:
URL: https://github.com/apache/arrow-datafusion/issues/8084#issuecomment-1802382671
> The same thing probably goes for `LargeList` as well (it isn't supported)
Yes, I believe the array function now only accepts `List`. I am happy to take this ticket.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org