[jira] [Created] (HIVE-28428) Map hash aggregation performance degradation

["Ryu Kobayashi (Jira)" <ji...@apache.org>]

Ryu Kobayashi created HIVE-28428:
------------------------------------

             Summary:  Map hash aggregation performance degradation
                 Key: HIVE-28428
                 URL: https://issues.apache.org/jira/browse/HIVE-28428
             Project: Hive
          Issue Type: Improvement
            Reporter: Ryu Kobayashi
         Attachments: 2024-08-02 14.35.46.png, image-2024-08-02-14-37-01-824.png, image-2024-08-02-14-38-45-459.png

The following ticket has been fixed to enable map hash aggregation, but performance degradation than when it is disabled.
https://issues.apache.org/jira/browse/HIVE-23356

I found a few reasons for this. If there are a large number of keys, the following log will be output in large volume, affecting performance. And, this can also cause an OOM.
{code:java}
2024-08-02 05:21:53,675 [INFO] [TezChild] |exec.GroupByOperator|: Hash Tbl flush: #hash table = 171000
2024-08-02 05:21:53,713 [INFO] [TezChild] |exec.GroupByOperator|: Hash Table flushed: new size = 153900
{code}
By fixing this, we can improve performance as follows.
Before:

!image-2024-08-02-14-37-01-824.png!

After:

!2024-08-02 14.35.46.png!

And, currently the flush size is fixed, but performance can be improved by changing it depending on the data:

!image-2024-08-02-14-38-45-459.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)