[jira] [Created] (HIVE-11924) cbo_gby q test fails with a client exception if bucketmapjoin is enabled

["Siddharth Seth (JIRA)" <ji...@apache.org>]

Siddharth Seth created HIVE-11924:
-------------------------------------

             Summary: cbo_gby q test fails with a client exception if bucketmapjoin is enabled
                 Key: HIVE-11924
                 URL: https://issues.apache.org/jira/browse/HIVE-11924
             Project: Hive
          Issue Type: Bug
    Affects Versions: 1.2.1, 2.0.0
            Reporter: Siddharth Seth
            Priority: Critical


Affects branch-1 and the llap branch. Not sure how far back this goes.

Steps to reproduce.
Option1: 
1. Apply patch from HIVE-11923.
2. Run all tez / llap tests or the following q files (-Dqfile=auto_sortmerge_join_9.q,cbo_gby.q)
Sequence is important.

Option2:
1. Modify cbo_gby.q to set the following
{code}
set hive.optimize.bucketmapjoin=true;
set hive.optimize.bucketmapjoin.sortedmerge=true;
set hive.auto.convert.sortmerge.join=true;
set hive.auto.convert.sortmerge.join.to.mapjoin=false;
{code}
2. Run only the cbo_gby.q with tez or llap

Fails with the following exception in the client
{code}
2015-09-22 15:14:18,854 INFO  [main]: optimizer.ConvertJoinMapJoin (ConvertJoinMapJoin.java:fallbackToReduceSideJoin(805)) - Fallback to common merge join operator
2015-09-22 15:14:18,855 INFO  [main]: optimizer.ConvertJoinMapJoin (ConvertJoinMapJoin.java:process(110)) - Estimated number of buckets 1
2015-09-22 15:14:18,855 INFO  [main]: optimizer.ConvertJoinMapJoin (ConvertJoinMapJoin.java:fallbackToReduceSideJoin(805)) - Fallback to common merge join operator
2015-09-22 15:14:18,856 ERROR [main]: ql.Driver (SessionState.java:printError(955)) - FAILED: ArrayIndexOutOfBoundsException -1
java.lang.ArrayIndexOutOfBoundsException: -1
        at java.util.ArrayList.elementData(ArrayList.java:418)
        at java.util.ArrayList.remove(ArrayList.java:495)
        at org.apache.hadoop.hive.ql.optimizer.ConvertJoinMapJoin.convertJoinSMBJoin(ConvertJoinMapJoin.java:241)
        at org.apache.hadoop.hive.ql.optimizer.ConvertJoinMapJoin.fallbackToReduceSideJoin(ConvertJoinMapJoin.java:806)
        at org.apache.hadoop.hive.ql.optimizer.ConvertJoinMapJoin.process(ConvertJoinMapJoin.java:119)
        at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
        at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
        at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
        at org.apache.hadoop.hive.ql.lib.ForwardWalker.walk(ForwardWalker.java:74)
        at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
        at org.apache.hadoop.hive.ql.parse.TezCompiler.runStatsDependentOptimizations(TezCompiler.java:280)
        at org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:125)
        at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:102)
        at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10234)
        at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:210)
        at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:240)
        at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:420)
        at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:308)
        at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1140)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1188)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1077)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1067)
        at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:311)
        at org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1028)
        at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:1002)
        at org.apache.hadoop.hive.cli.TestMiniTezCliDriver.runTest(TestMiniTezCliDriver.java:146)
        at org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_gby(TestMiniTezCliDriver.java:130)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at junit.framework.TestCase.runTest(TestCase.java:176)
        at junit.framework.TestCase.runBare(TestCase.java:141)
        at junit.framework.TestResult$1.protect(TestResult.java:122)
        at junit.framework.TestResult.runProtected(TestResult.java:142)
        at junit.framework.TestResult.run(TestResult.java:125)
        at junit.framework.TestCase.run(TestCase.java:129)
        at junit.framework.TestSuite.runTest(TestSuite.java:255)
        at junit.framework.TestSuite.run(TestSuite.java:250)
        at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
        at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
        at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
        at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
        at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
        at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
        at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
{code}

Overall looks like this is something that could be hit during regular usage as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)