You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Jim Green <op...@gmail.com> on 2015/07/01 22:43:52 UTC
Hive 1.0 on Tez 0.5.3 reproduced HIVE-8204
Hi Team,
Is HIVE-8204 really fixed or not?
Below test can reproduce exactly the same stacktrace when dynamic partition
pruning is enabled.
*1. Create test tables*
create table h1_fact(id int, id2 string) PARTITIONED BY (yyyymmdd string)
row format delimited fields terminated by ',';
create table h1_dim1(id int, id2 string) PARTITIONED BY (yyyymmdd string)
row format delimited fields terminated by ',’;
*2. Load some data.*
*3. Below SQL will fail:*
set hive.execution.engine=tez;
select h1_fact.id2 from h1_fact join h1_dim1
on h1_fact.yyyymmdd=h1_dim1.yyyymmdd
where h1_dim1.id2 <> 'adfsdfd’;
2015-07-01 20:37:51,210 ERROR [main]: ql.Driver
(SessionState.java:printError(833)) - FAILED: IndexOutOfBoundsException
Index: 0, Size: 0
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at java.util.ArrayList.rangeCheck(ArrayList.java:653)
at java.util.ArrayList.get(ArrayList.java:429)
at
org.apache.hadoop.hive.ql.optimizer.RemoveDynamicPruningBySize.process(RemoveDynamicPruningBySize.java:61)
at
org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
at org.apache.hadoop.hive.ql.lib.ForwardWalker.walk(ForwardWalker.java:79)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
at
org.apache.hadoop.hive.ql.parse.TezCompiler.runStatsDependentOptimizations(TezCompiler.java:281)
at
org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:123)
at
org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:99)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10250)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:221)
at
org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:221)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:429)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:315)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1082)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1144)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1019)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1009)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:201)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:153)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:364)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:712)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:631)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:570)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
--
Thanks,
www.openkb.info
(Open KnowledgeBase for Hadoop/Database/OS/Network/Tool)
Re: Hive 1.0 on Tez 0.5.3 reproduced HIVE-8204
Posted by Vikram Dixit <vi...@hortonworks.com>.
Moved user@tez to bcc. Looks like that jira just added a test case and was closed as 'not reproducible'. However, this jira seems to have found and fixed the issue:
https://issues.apache.org/jira/browse/HIVE-10559
Regards
Vikram.
From: Jim Green <op...@gmail.com>>
Reply-To: "user@tez.apache.org<ma...@tez.apache.org>" <us...@tez.apache.org>>
Date: Wednesday, July 1, 2015 at 1:43 PM
To: "user@tez.apache.org<ma...@tez.apache.org>" <us...@tez.apache.org>>, "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Subject: Hive 1.0 on Tez 0.5.3 reproduced HIVE-8204
Hi Team,
Is HIVE-8204 really fixed or not?
Below test can reproduce exactly the same stacktrace when dynamic partition pruning is enabled.
1. Create test tables
create table h1_fact(id int, id2 string) PARTITIONED BY (yyyymmdd string) row format delimited fields terminated by ',';
create table h1_dim1(id int, id2 string) PARTITIONED BY (yyyymmdd string) row format delimited fields terminated by ',';
2. Load some data.
3. Below SQL will fail:
set hive.execution.engine=tez;
select h1_fact.id2 from h1_fact join h1_dim1
on h1_fact.yyyymmdd=h1_dim1.yyyymmdd
where h1_dim1.id2 <> 'adfsdfd';
2015-07-01 20:37:51,210 ERROR [main]: ql.Driver (SessionState.java:printError(833)) - FAILED: IndexOutOfBoundsException Index: 0, Size: 0
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at java.util.ArrayList.rangeCheck(ArrayList.java:653)
at java.util.ArrayList.get(ArrayList.java:429)
at org.apache.hadoop.hive.ql.optimizer.RemoveDynamicPruningBySize.process(RemoveDynamicPruningBySize.java:61)
at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
at org.apache.hadoop.hive.ql.lib.ForwardWalker.walk(ForwardWalker.java:79)
at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
at org.apache.hadoop.hive.ql.parse.TezCompiler.runStatsDependentOptimizations(TezCompiler.java:281)
at org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:123)
at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:99)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10250)
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:221)
at org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:221)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:429)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:315)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1082)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1144)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1019)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1009)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:201)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:153)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:364)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:712)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:631)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:570)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
--
Thanks,
www.openkb.info<http://www.openkb.info>
(Open KnowledgeBase for Hadoop/Database/OS/Network/Tool)
Re: Hive 1.0 on Tez 0.5.3 reproduced HIVE-8204
Posted by Vikram Dixit <vi...@hortonworks.com>.
Moved user@tez to bcc. Looks like that jira just added a test case and was closed as 'not reproducible'. However, this jira seems to have found and fixed the issue:
https://issues.apache.org/jira/browse/HIVE-10559
Regards
Vikram.
From: Jim Green <op...@gmail.com>>
Reply-To: "user@tez.apache.org<ma...@tez.apache.org>" <us...@tez.apache.org>>
Date: Wednesday, July 1, 2015 at 1:43 PM
To: "user@tez.apache.org<ma...@tez.apache.org>" <us...@tez.apache.org>>, "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Subject: Hive 1.0 on Tez 0.5.3 reproduced HIVE-8204
Hi Team,
Is HIVE-8204 really fixed or not?
Below test can reproduce exactly the same stacktrace when dynamic partition pruning is enabled.
1. Create test tables
create table h1_fact(id int, id2 string) PARTITIONED BY (yyyymmdd string) row format delimited fields terminated by ',';
create table h1_dim1(id int, id2 string) PARTITIONED BY (yyyymmdd string) row format delimited fields terminated by ',';
2. Load some data.
3. Below SQL will fail:
set hive.execution.engine=tez;
select h1_fact.id2 from h1_fact join h1_dim1
on h1_fact.yyyymmdd=h1_dim1.yyyymmdd
where h1_dim1.id2 <> 'adfsdfd';
2015-07-01 20:37:51,210 ERROR [main]: ql.Driver (SessionState.java:printError(833)) - FAILED: IndexOutOfBoundsException Index: 0, Size: 0
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at java.util.ArrayList.rangeCheck(ArrayList.java:653)
at java.util.ArrayList.get(ArrayList.java:429)
at org.apache.hadoop.hive.ql.optimizer.RemoveDynamicPruningBySize.process(RemoveDynamicPruningBySize.java:61)
at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
at org.apache.hadoop.hive.ql.lib.ForwardWalker.walk(ForwardWalker.java:79)
at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
at org.apache.hadoop.hive.ql.parse.TezCompiler.runStatsDependentOptimizations(TezCompiler.java:281)
at org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:123)
at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:99)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10250)
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:221)
at org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:221)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:429)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:315)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1082)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1144)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1019)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1009)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:201)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:153)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:364)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:712)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:631)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:570)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
--
Thanks,
www.openkb.info<http://www.openkb.info>
(Open KnowledgeBase for Hadoop/Database/OS/Network/Tool)