You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Gopal Vijayaraghavan (Jira)" <ji...@apache.org> on 2019/12/18 16:52:00 UTC
[jira] [Commented] (HIVE-22662)
java.lang.NegativeArraySizeException when trying to get HashPartition
[ https://issues.apache.org/jira/browse/HIVE-22662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16999340#comment-16999340 ]
Gopal Vijayaraghavan commented on HIVE-22662:
---------------------------------------------
HybridHashTableContainer was disabled by default in HiveConf
{code}
HIVEUSEHYBRIDGRACEHASHJOIN("hive.mapjoin.hybridgrace.hashtable", false, "Whether to use hybrid" +
"grace hash join as the join method for mapjoin. Tez only."),
{code}
bq. this parameter is set hive.auto.convert.join=false the issue disappears
Can you confirm what happens when the hybridgrace is turned off to match the main branch features?
> java.lang.NegativeArraySizeException when trying to get HashPartition
> ---------------------------------------------------------------------
>
> Key: HIVE-22662
> URL: https://issues.apache.org/jira/browse/HIVE-22662
> Project: Hive
> Issue Type: Bug
> Components: Hive
> Affects Versions: 3.1.0
> Environment: Using hdp 3.1.4.0-315
> Reporter: Diego Patricio Moncayo
> Priority: Minor
>
> Running a query between a big table and a small one, this error appeared:
> {code:java}
> Caused by: java.lang.RuntimeException: Map operator initialization failed
> at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:361)
> at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
> ... 16 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Async Initialization failed. abortRequested=false
> at org.apache.hadoop.hive.ql.exec.Operator.completeInitialization(Operator.java:465)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:399)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:572)
> at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:524)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
> at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:342)
> ... 17 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NegativeArraySizeException
> at org.apache.hadoop.hive.ql.exec.tez.ObjectCache.retrieve(ObjectCache.java:103)
> at org.apache.hadoop.hive.ql.exec.tez.ObjectCache$1.call(ObjectCache.java:113)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NegativeArraySizeException
> at org.apache.hadoop.hive.ql.exec.tez.HashTableLoader.load(HashTableLoader.java:271)
> at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTableInternal(MapJoinOperator.java:344)
> at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:413)
> at org.apache.hadoop.hive.ql.exec.MapJoinOperator.lambda$initializeOp$0(MapJoinOperator.java:215)
> at org.apache.hadoop.hive.ql.exec.tez.ObjectCache.retrieve(ObjectCache.java:96)
> ... 5 more
> Caused by: java.lang.NegativeArraySizeException
> at org.apache.hadoop.hive.ql.exec.persistence.HybridHashTableContainer.<init>(HybridHashTableContainer.java:362)
> at org.apache.hadoop.hive.ql.exec.persistence.HybridHashTableContainer.<init>(HybridHashTableContainer.java:277)
> at org.apache.hadoop.hive.ql.exec.tez.HashTableLoader.load(HashTableLoader.java:231)
> ... 9 more
> {code}
> When looking into the stacktrace, this line appeared.
> {code:java}
> hashPartitions = new HashPartition[numPartitions];
> {code}
> Apparently the int variable numPartition overflows
> When this parameter is set hive.auto.convert.join=false the issue disappears
> Similar to this Issue https://issues.apache.org/jira/browse/HIVE-12547 but not the same
--
This message was sent by Atlassian Jira
(v8.3.4#803005)