You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Ayush Saxena (Jira)" <ji...@apache.org> on 2023/12/07 12:16:00 UTC
[jira] [Updated] (HIVE-27520) Out of Memory when set hive.hashtable.initialCapacity to a large number
[ https://issues.apache.org/jira/browse/HIVE-27520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ayush Saxena updated HIVE-27520:
--------------------------------
Priority: Major (was: Critical)
> Out of Memory when set hive.hashtable.initialCapacity to a large number
> -----------------------------------------------------------------------
>
> Key: HIVE-27520
> URL: https://issues.apache.org/jira/browse/HIVE-27520
> Project: Hive
> Issue Type: Bug
> Reporter: ConfX
> Priority: Major
> Attachments: reproduce.sh
>
>
> h2. What happened:
> When setting {{hive.hashtable.initialCapacity}} to a large number, Hive throws an out-of-memory exception due to inappropriate checking and handling.
> *There is no checking or error-handling logic for this parameter at all.*
> h2. StackTrace:
> {noformat}
> java.lang.IllegalArgumentException: Illegal initial capacity: -923553553
> at java.base/java.util.HashMap.<init>(HashMap.java:445)
> at org.apache.hadoop.hive.ql.exec.persistence.HashMapWrapper.<init>(HashMapWrapper.java:89)
> at org.apache.hadoop.hive.ql.exec.persistence.HashMapWrapper.<init>(HashMapWrapper.java:81)
> at org.apache.hadoop.hive.ql.exec.vector.mapjoin.MapJoinTestConfig.createMapJoin(MapJoinTestConfig.java:837)
> at org.apache.hadoop.hive.ql.exec.vector.mapjoin.MapJoinTestConfig.createMapJoinImplementation(MapJoinTestConfig.java:988)
> at org.apache.hadoop.hive.ql.exec.vector.mapjoin.MapJoinTestConfig.createMapJoinImplementation(MapJoinTestConfig.java:971)
> {noformat}
> h2. Reproduce:
> (1) Set {{hive.hashtable.initialCapacity}} to a large value, e.g., 1223962862
> (2) Run a simple test that exercises this parameter, e.g. {{org.apache.hadoop.hive.ql.exec.vector.mapjoin.TestMapJoinOperator#testMultiKey3}}
>
> For an easy reproduction, run the reproduce.sh in the attachment.
> We are happy to provide a patch if this issue is confirmed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)