You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Andy Grove (Jira)" <ji...@apache.org> on 2020/10/12 21:27:00 UTC
[jira] [Commented] (ARROW-10275) [Rust] [Datafusion] GROUP BY with
a high cardinality doesn't seem to finish
[ https://issues.apache.org/jira/browse/ARROW-10275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17212682#comment-17212682 ]
Andy Grove commented on ARROW-10275:
------------------------------------
I have seen the same behavior. We have mostly been testing hash aggregates with queries that produce low cardinality results and will need to spend time testing for high cardinality results and see how we can optimize this.
> [Rust] [Datafusion] GROUP BY with a high cardinality doesn't seem to finish
> ---------------------------------------------------------------------------
>
> Key: ARROW-10275
> URL: https://issues.apache.org/jira/browse/ARROW-10275
> Project: Apache Arrow
> Issue Type: Bug
> Components: Rust - DataFusion
> Affects Versions: 2.0.0
> Environment: Ubuntu 20.04
> Reporter: Josh Taylor
> Priority: Minor
>
> Group by with a high cardinality (columns with lots of unique values) don't seem to finish.
> I've tried with both datafusion-cli and this:
> [https://github.com/joshuataylor/parquet-group-by/blob/main/src/main.rs]
> When doing O_ORDERKEY there are ~15 000 000 unique records, so it seems to stall. I've tried with limit but it doesn't work either.
> My parquet file: [https://drive.google.com/file/d/1aCW7SW2rUVioSePduhgo_91F5-xDMyjp/view?usp=sharing]
> datafusion-cli:
> {code:java}
> CREATE EXTERNAL TABLE something STORED AS PARQUET LOCATION 'demo.parquet';
> select O_ORDERKEY from something group by O_ORDERKEY;
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)