You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "jonmmease (via GitHub)" <gi...@apache.org> on 2023/09/04 11:57:09 UTC
[GitHub] [arrow-datafusion] jonmmease opened a new issue, #7473: Unsupported data type in hasher: List
jonmmease opened a new issue, #7473:
URL: https://github.com/apache/arrow-datafusion/issues/7473
### Describe the bug
DataFusion raises an error when a `List` column is used as the GROUP BY column
### To Reproduce
```sql
SELECT COUNT(*) FROM (VALUES (['a', 'b'], 1), (['a', 'b'], 2)) as values0 GROUP BY column1;
```
```
External error: Internal error: Unsupported data type in hasher: List(Field { name: "item", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }). This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker
```
### Expected behavior
Successful GROUP BY
### Additional context
Apologies if this is tracked elsewhere, but I didn't find it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] xudong963 closed issue #7473: Unsupported data type in hasher: List
Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 closed issue #7473: Unsupported data type in hasher: List
URL: https://github.com/apache/arrow-datafusion/issues/7473
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] tustvold commented on issue #7473: Unsupported data type in hasher: List
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #7473:
URL: https://github.com/apache/arrow-datafusion/issues/7473#issuecomment-1707086143
The trick will be to first compute the hashes for all the values in the list, by recursing, and then compute the hashes of these hashes using the list offsets.
This is not too dissimilar to what is done for dictionaries - https://github.com/apache/arrow-datafusion/blob/3ea870cf0995a86b5297a92be5d5432577aefabe/datafusion/physical-expr/src/hash_utils.rs#L190-L264
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jonmmease commented on issue #7473: Unsupported data type in hasher: List
Posted by "jonmmease (via GitHub)" <gi...@apache.org>.
jonmmease commented on issue #7473:
URL: https://github.com/apache/arrow-datafusion/issues/7473#issuecomment-1705217788
I started looking into this a bit, but I ran into trouble adding support to `create_hashes`, because the elements of a `ListArray` are `ArrayRef`s, which are not hashable.
https://github.com/apache/arrow-datafusion/blob/3ea870cf0995a86b5297a92be5d5432577aefabe/datafusion/physical-expr/src/hash_utils.rs#L264
Let me know if anyone has thoughts on how we should proceed. Does it make sense for ArrayRef be hashable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org