You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Ganesha Shreedhara (JIRA)" <ji...@apache.org> on 2018/04/09 14:36:00 UTC
[jira] [Commented] (HIVE-19101) Apply rule
[HiveJoinPushTransitivePredicatesRule] is getting stuck when there are huge
number of predicates
[ https://issues.apache.org/jira/browse/HIVE-19101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16430613#comment-16430613 ]
Ganesha Shreedhara commented on HIVE-19101:
-------------------------------------------
[~jcamachorodriguez] Could you please help with this? Please let me know if you need more details on this.
> Apply rule [HiveJoinPushTransitivePredicatesRule] is getting stuck when there are huge number of predicates
> ------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-19101
> URL: https://issues.apache.org/jira/browse/HIVE-19101
> Project: Hive
> Issue Type: Bug
> Components: CBO
> Affects Versions: 2.1.1, 2.2.0, 2.3.0, 2.3.1, 2.3.2
> Reporter: Ganesha Shreedhara
> Priority: Major
> Attachments: queries
>
>
> Hive query is getting stuck during the optimisation phase while applying HiveJoinPushTransitivePredicatesRule when there are huge number of predicates.
>
> *DEBUG Log:*
> {code:java}
> 2018-04-04T11:22:47,991 [user: ganeshas] -1 DEBUG [6f5e8faa-505c-48e3-a2cd-ce7bfced27f0 main] plan.RelOptPlanner: call#10963: Apply rule [ReduceExpressionsRule(Join)] to [rel#881:HiveJoin.HIVE.[](left=HepRelVertex#879,right=HepRelVertex#887,condition==($1, $73),joinType=inner,algorithm=none,cost=not available)]
> 2018-04-04T11:22:48,359 [user: ganeshas] -1 DEBUG [6f5e8faa-505c-48e3-a2cd-ce7bfced27f0 main] plan.RelOptPlanner: call#10964: Apply rule [HiveJoinAddNotNullRule] to [rel#881:HiveJoin.HIVE.[](left=HepRelVertex#879,right=HepRelVertex#887,condition==($1, $73),joinType=inner,algorithm=none,cost=not available)]
> 2018-04-04T11:22:48,360 [user: ganeshas] -1 DEBUG [6f5e8faa-505c-48e3-a2cd-ce7bfced27f0 main] plan.RelOptPlanner: call#10965: Apply rule [HiveJoinPushTransitivePredicatesRule] to [rel#881:HiveJoin.HIVE.[](left=HepRelVertex#879,right=HepRelVertex#887,condition==($1, $73),joinType=inner,algorithm=none,cost=not available)]{code}
>
> *Thread Status:*
>
> {code:java}
> "6f5e8faa-505c-48e3-a2cd-ce7bfced27f0 main" prio=5 tid=0x00007ff18e006800 nid=0x1c03 runnable [0x0000700008176000]
> java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOfRange(Arrays.java:2694)
> at java.lang.String.<init>(String.java:203)
> at java.lang.StringBuilder.toString(StringBuilder.java:405)
> at org.apache.calcite.rex.RexCall.computeDigest(RexCall.java:95)
> at org.apache.calcite.rex.RexCall.toString(RexCall.java:100)
> at org.apache.calcite.rex.RexCall.computeDigest(RexCall.java:84)
> at org.apache.calcite.rex.RexCall.toString(RexCall.java:100)
> at org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdPredicates$JoinConditionBasedPredicateInference.infer(HiveRelMdPredicates.java:516)
> at org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdPredicates$JoinConditionBasedPredicateInference.inferPredicates(HiveRelMdPredicates.java:426)
> at org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdPredicates.getPredicates(HiveRelMdPredicates.java:186)
> at GeneratedMetadataHandler_Predicates.getPredicates_$(Unknown Source)
> at GeneratedMetadataHandler_Predicates.getPredicates(Unknown Source)
> at org.apache.calcite.rel.metadata.RelMetadataQuery.getPulledUpPredicates(RelMetadataQuery.java:721)
> at org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveJoinPushTransitivePredicatesRule.onMatch(HiveJoinPushTransitivePredicatesRule.java:83)
> at org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:314)
> at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:502)
> at org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:381)
> at org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:275)
> at org.apache.calcite.plan.hep.HepInstruction$RuleCollection.execute(HepInstruction.java:72)
> at org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:206)
> at org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:193)
> at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.hepPlan(CalcitePlanner.java:1575)
> at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.applyPreJoinOrderingTransforms(CalcitePlanner.java:1448)
> at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1174)
> at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1096)
> at org.apache.calcite.tools.Frameworks$1.apply(Frameworks.java:113)
> at org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:997)
> at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:149)
> at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:106)
> at org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:905)
> at org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedAST(CalcitePlanner.java:920)
> at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:330)
> at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11206)
> at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:251)
> at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:265)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:527)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1358)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1499)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1278)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1268)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:213)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:447)
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:924)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:848)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:735)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:223)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Locked ownable synchronizers:
> - <0x00000007d5cfcab0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
> {code}
>
>
> This issue seems to be because of optimisation done on CBO (Specifically on calcite transitive predicate inference rule) in hive 2.1.1 and later versions.
> This issue is not there in hive1.2.1 version.
>
> The queries to reproduce this issue is in this attachment : [^queries].
> There is a view (test_view) created by combining the results from 16 tables using UNION ALL.
> The query is getting stuck when I run a select query which performs join between view (test_view) and a table (test1).
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)