[jira] [Created] (DRILL-5604) Possible performance degradation with hash aggregate when number of distinct keys increase

["Rahul Challapalli (JIRA)" <ji...@apache.org>]

Rahul Challapalli created DRILL-5604:
----------------------------------------

             Summary: Possible performance degradation with hash aggregate when number of distinct keys increase
                 Key: DRILL-5604
                 URL: https://issues.apache.org/jira/browse/DRILL-5604
             Project: Apache Drill
          Issue Type: Bug
          Components: Execution - Relational Operators
    Affects Versions: 1.11.0
            Reporter: Rahul Challapalli


git.commit.id.abbrev=90f43bf

I tried to track the runtime as we gradually increase the no of distinct keys without increasing the total no of records. Below is one such test on top of tpcds sf1000 dataset

{code}
0: jdbc:drill:zk=10.10.100.190:5181> select count(distinct ss_list_price) from store_sales;
+---------+
| EXPR$0  |
+---------+
| 19736   |
+---------+
1 row selected (163.345 seconds)
0: jdbc:drill:zk=10.10.100.190:5181> select count(distinct ss_net_profit) from store_sales;
+----------+
|  EXPR$0  |
+----------+
| 1525675  |
+----------+
1 row selected (2094.962 seconds)
{code}

In both the above queries, the hash agg code processed 2879987999 records. So the time difference is due to overheads like hash table resizing etc. The second query took ~30 mins more than the first raising doubts whether there is an issue somewhere.

The dataset is too large to attach to a jira and so are the logs



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)