You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2023/10/26 15:52:48 UTC
[I] Panic with queries with multiple `COUNT DISTINCT` aggregates on dictionary values, and a group by [arrow-datafusion]
alamb opened a new issue, #7938:
URL: https://github.com/apache/arrow-datafusion/issues/7938
### Describe the bug
DataFusion panic's when runnning a query like
```
select count(distinct column1), count(distinct column2) from test group by column1;
```
Where `column1` and `column2` are Dictionary
### To Reproduce
```
❯ create table test as values (1, arrow_cast('foo', 'Dictionary(Int32, Utf8)')), (2, arrow_cast('bar', 'Dictionary(Int32, Utf8)'));
0 rows in set. Query took 0.002 seconds.
❯ select * from test;
+---------+---------+
| column1 | column2 |
+---------+---------+
| 1 | foo |
| 2 | bar |
+---------+---------+
2 rows in set. Query took 0.001 seconds.
❯ select count(distinct column1), count(distinct column2) from test group by column1;
thread 'tokio-runtime-worker' panicked at /Users/alamb/Software/arrow-datafusion/datafusion/common/src/scalar.rs:1846:18:
Unsupported data type Dictionary(Int32, Utf8) for ScalarValue::list_to_array
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
External error: Join Error
caused by
External error: task 66 panicked
```
### Expected behavior
The test should produce the same as when the values are cast to string:
```
❯ select count(distinct column1::varchar), count(distinct column2::varchar) from test group by column1;
+------------------------------+------------------------------+
| COUNT(DISTINCT test.column1) | COUNT(DISTINCT test.column2) |
+------------------------------+------------------------------+
| 1 | 1 |
| 1 | 1 |
+------------------------------+------------------------------+
```
### Additional context
I believe this is a regression introduced in https://github.com/apache/arrow-datafusion/pull/7629 (not yet released)
We found this downstream in IOx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Panic with queries with multiple `COUNT DISTINCT` aggregates on dictionary values, and a group by [arrow-datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on issue #7938:
URL: https://github.com/apache/arrow-datafusion/issues/7938#issuecomment-1782083281
Other than Dictionary, there are also many types I did not support yet, since I am not sure whether we need them all, so I only supported the type that is used (aka pass all test).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Panic with queries with multiple `COUNT DISTINCT` aggregates on dictionary values, and a group by [arrow-datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on issue #7938:
URL: https://github.com/apache/arrow-datafusion/issues/7938#issuecomment-1782096338
btw I forgot to fix this "Unsupported data type {:?} for ScalarValue::list_to_array" to "Unsupported data type {:?} for ScalarValue::new_list"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Panic with queries with multiple `COUNT DISTINCT` aggregates on dictionary values, and a group by [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #7938:
URL: https://github.com/apache/arrow-datafusion/issues/7938#issuecomment-1786004627
Here is proposed fix: https://github.com/apache/arrow-datafusion/pull/7989
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Panic with queries with multiple `COUNT DISTINCT` aggregates on dictionary values, and a group by [arrow-datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on issue #7938:
URL: https://github.com/apache/arrow-datafusion/issues/7938#issuecomment-1782112411
In this case we may need to change to logic of state() https://github.com/apache/arrow-datafusion/blob/a9d66e2b492843c2fb335a7dfe27fed073629b09/datafusion/physical-expr/src/aggregate/count_distinct.rs#L145-L149
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Panic with queries with multiple `COUNT DISTINCT` aggregates on dictionary values, and a group by [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #7938:
URL: https://github.com/apache/arrow-datafusion/issues/7938#issuecomment-1782615292
>
Other than Dictionary, there are also many types I did not support yet, since I am not sure whether we need them all, so I only supported the type that is used (aka pass all test).
Yeah, there is clearly a test coverage gap -- I'll take a look into what is going on
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Panic with queries with multiple `COUNT DISTINCT` aggregates on dictionary values, and a group by [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #7938: Panic with queries with multiple `COUNT DISTINCT` aggregates on dictionary values, and a group by
URL: https://github.com/apache/arrow-datafusion/issues/7938
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org