You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "yx91490 (JIRA)" <ji...@apache.org> on 2019/01/27 07:40:00 UTC
[jira] [Comment Edited] (HIVE-21153) task run in parallel cause
ArrayIndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/HIVE-21153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16753315#comment-16753315 ]
yx91490 edited comment on HIVE-21153 at 1/27/19 7:39 AM:
---------------------------------------------------------
yes, I use hive.exec.parallel=true, the full hive confs are:
set hive.map.aggr=true
set hive.exec.parallel=true
set hive.exec.reducers.max=100
set hive.exec.parallel.thread.number=18
set mapred.reduce.tasks=68
set hive.exec.dynamic.partition.mode=nonstrict;
disable it may help to avoid this problem, but our users want to run job faster.
the "latest" means the master branch in github.com/apache/hive.
and why do you think this wouldn't happen in Hive-3.x ?
was (Author: yx91490):
yes, I use hive.exec.parallel=true, the full hive confs are:
set hive.map.aggr=true
set hive.exec.parallel=true
set hive.exec.reducers.max=100
set hive.exec.parallel.thread.number=18
set mapred.reduce.tasks=68
set hive.exec.dynamic.partition.mode=nonstrict;
the "latest" means the master branch in github.com/apache/hive.
and why do you think this wouldn't happen in Hive-3.x ?
> task run in parallel cause ArrayIndexOutOfBoundsException
> ---------------------------------------------------------
>
> Key: HIVE-21153
> URL: https://issues.apache.org/jira/browse/HIVE-21153
> Project: Hive
> Issue Type: Bug
> Reporter: yx91490
> Priority: Major
>
> It happens occasionally when we run sql in parallel and the job is blocked.the reason is HiveConf class is not thread safe, and SessionState.createSessionDirs() is not Synchronized.
> the error message:
> java.lang.ArrayIndexOutOfBoundsException: 22
> at java.util.ArrayList.add(ArrayList.java:459)
> at org.apache.hadoop.hive.conf.HiveConf.setupRestrictList(HiveConf.java:3216)
> at org.apache.hadoop.hive.conf.HiveConf.addToRestrictList(HiveConf.java:3191)
> at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:651)
> at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:547)
> at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:78)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)