[jira] [Created] (HIVE-19101) Apply rule [HiveJoinPushTransitivePredicatesRule] is getting stuck when there are huge number of predicates

["Ganesha Shreedhara (JIRA)" <ji...@apache.org>]

Ganesha Shreedhara created HIVE-19101:
-----------------------------------------

             Summary: Apply rule [HiveJoinPushTransitivePredicatesRule] is getting stuck when there are huge number of predicates 
                 Key: HIVE-19101
                 URL: https://issues.apache.org/jira/browse/HIVE-19101
             Project: Hive
          Issue Type: Bug
          Components: CBO
    Affects Versions: 2.3.2, 2.3.1, 2.3.0, 2.2.0, 2.1.1
            Reporter: Ganesha Shreedhara
         Attachments: queries

Hive query is getting stuck during the optimisation phase while applying HiveJoinPushTransitivePredicatesRule when there are huge number of predicates.

 

*DEBUG Log:*
{code:java}
2018-04-04T11:22:47,991 [user: ganeshas] -1 DEBUG [6f5e8faa-505c-48e3-a2cd-ce7bfced27f0 main] plan.RelOptPlanner: call#10963: Apply rule [ReduceExpressionsRule(Join)] to [rel#881:HiveJoin.HIVE.[](left=HepRelVertex#879,right=HepRelVertex#887,condition==($1, $73),joinType=inner,algorithm=none,cost=not available)]
2018-04-04T11:22:48,359 [user: ganeshas] -1 DEBUG [6f5e8faa-505c-48e3-a2cd-ce7bfced27f0 main] plan.RelOptPlanner: call#10964: Apply rule [HiveJoinAddNotNullRule] to [rel#881:HiveJoin.HIVE.[](left=HepRelVertex#879,right=HepRelVertex#887,condition==($1, $73),joinType=inner,algorithm=none,cost=not available)]
2018-04-04T11:22:48,360 [user: ganeshas] -1 DEBUG [6f5e8faa-505c-48e3-a2cd-ce7bfced27f0 main] plan.RelOptPlanner: call#10965: Apply rule [HiveJoinPushTransitivePredicatesRule] to [rel#881:HiveJoin.HIVE.[](left=HepRelVertex#879,right=HepRelVertex#887,condition==($1, $73),joinType=inner,algorithm=none,cost=not available)]{code}
 

*Thread Status:*

 
{code:java}
"6f5e8faa-505c-48e3-a2cd-ce7bfced27f0 main" prio=5 tid=0x00007ff18e006800 nid=0x1c03 runnable [0x0000700008176000]
 java.lang.Thread.State: RUNNABLE
 at java.util.Arrays.copyOfRange(Arrays.java:2694)
 at java.lang.String.<init>(String.java:203)
 at java.lang.StringBuilder.toString(StringBuilder.java:405)
 at org.apache.calcite.rex.RexCall.computeDigest(RexCall.java:95)
 at org.apache.calcite.rex.RexCall.toString(RexCall.java:100)
 at org.apache.calcite.rex.RexCall.computeDigest(RexCall.java:84)
 at org.apache.calcite.rex.RexCall.toString(RexCall.java:100)
 at org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdPredicates$JoinConditionBasedPredicateInference.infer(HiveRelMdPredicates.java:516)
 at org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdPredicates$JoinConditionBasedPredicateInference.inferPredicates(HiveRelMdPredicates.java:426)
 at org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdPredicates.getPredicates(HiveRelMdPredicates.java:186)
 at GeneratedMetadataHandler_Predicates.getPredicates_$(Unknown Source)
 at GeneratedMetadataHandler_Predicates.getPredicates(Unknown Source)
 at org.apache.calcite.rel.metadata.RelMetadataQuery.getPulledUpPredicates(RelMetadataQuery.java:721)
 at org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveJoinPushTransitivePredicatesRule.onMatch(HiveJoinPushTransitivePredicatesRule.java:83)
 at org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:314)
 at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:502)
 at org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:381)
 at org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:275)
 at org.apache.calcite.plan.hep.HepInstruction$RuleCollection.execute(HepInstruction.java:72)
 at org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:206)
 at org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:193)
 at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.hepPlan(CalcitePlanner.java:1575)
 at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.applyPreJoinOrderingTransforms(CalcitePlanner.java:1448)
 at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1174)
 at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1096)
 at org.apache.calcite.tools.Frameworks$1.apply(Frameworks.java:113)
 at org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:997)
 at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:149)
 at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:106)
 at org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:905)
 at org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedAST(CalcitePlanner.java:920)
 at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:330)
 at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11206)
 at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:251)
 at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:265)
 at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:527)
 at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1358)
 at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1499)
 at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1278)
 at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1268)
 at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
 at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:213)
 at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:447)
 at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:924)
 at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:848)
 at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:735)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:606)
 at org.apache.hadoop.util.RunJar.run(RunJar.java:223)
 at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Locked ownable synchronizers:
 - <0x00000007d5cfcab0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
{code}
 

 

This issue seems to be  because of optimisation done on CBO (Specifically on calcite transitive predicate inference rule) in hive 2.1.1 and later versions. 

This issue is not there in hive1.2.1 version.

 

The queries to reproduce this issue is in this attachment : [^queries]. 

There is a view (test_view) created by combining the results from 16 tables using UNION ALL. 

The query is getting stuck when I run a select query which performs join between view (test_view) and a table (test1). 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)