You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Stamatis Zampetakis (Jira)" <ji...@apache.org> on 2021/09/16 14:36:00 UTC
[jira] [Commented] (HIVE-25530) AssertionError when query involves
multiple JDBC tables and views
[ https://issues.apache.org/jira/browse/HIVE-25530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17416145#comment-17416145 ]
Stamatis Zampetakis commented on HIVE-25530:
--------------------------------------------
Basically any query with more than one external JDBC table and at least one view can trigger the problem.
From a quick look the culprit seems to be that we are [instantiating|https://github.com/apache/hive/blob/3e861c5f2775cda4821199681e0e2e9d25654371/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java#L3023] a new {{JdbcConvention}} for every table in the query. Then when the {{VolcanoPlanner}} runs it will [register the rules|https://github.com/apache/calcite/blob/f3baf348598fcc6bc4f97a0abee3f99309e5bf76/core/src/main/java/org/apache/calcite/plan/AbstractRelOptPlanner.java#L239] for every convention that is not registered till now. Since there is a new convention per table it will trigger the registering of the rules as many times as distinct tables in the query.
> AssertionError when query involves multiple JDBC tables and views
> -----------------------------------------------------------------
>
> Key: HIVE-25530
> URL: https://issues.apache.org/jira/browse/HIVE-25530
> Project: Hive
> Issue Type: Bug
> Components: CBO, HiveServer2
> Affects Versions: 4.0.0
> Reporter: Stamatis Zampetakis
> Assignee: Soumyakanti Das
> Priority: Major
> Fix For: 4.0.0
>
> Attachments: engesc_6056.q
>
>
> An {{AssertionError}} is thrown during compilation when a query contains multiple external JDBC tables and there are available materialized views which can be used to answer the query.
> The problem can be reproduced by running the scenario in [^engesc_6056.q].
> {code:bash}
> mvn test -Dtest=TestMiniLlapLocalCliDriver -Dqfile=engesc_6056.q -Dtest.output.overwrite
> {code}
> The stacktrace is shown below:
> {noformat}
> java.lang.AssertionError: Rule's description should be unique; existing rule=JdbcToEnumerableConverterRule(in:JDBC.DERBY,out:ENUMERABLE); new rule=JdbcToEnumerableConverterRule(in:JDBC.DERBY,out:ENUMERABLE)
> at org.apache.calcite.plan.AbstractRelOptPlanner.addRule(AbstractRelOptPlanner.java:158)
> at org.apache.calcite.plan.volcano.VolcanoPlanner.addRule(VolcanoPlanner.java:406)
> at org.apache.calcite.adapter.jdbc.JdbcConvention.register(JdbcConvention.java:66)
> at org.apache.calcite.plan.AbstractRelOptPlanner.registerClass(AbstractRelOptPlanner.java:233)
> at org.apache.hadoop.hive.ql.optimizer.calcite.cost.HiveVolcanoPlanner.registerClass(HiveVolcanoPlanner.java:90)
> at org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1224)
> at org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:589)
> at org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:604)
> at org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:84)
> at org.apache.calcite.rel.AbstractRelNode.onRegister(AbstractRelNode.java:268)
> at org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1132)
> at org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:589)
> at org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:604)
> at org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:84)
> at org.apache.calcite.rel.AbstractRelNode.onRegister(AbstractRelNode.java:268)
> at org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1132)
> at org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:589)
> at org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:604)
> at org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:148)
> at org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:268)
> at org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:283)
> at org.apache.hadoop.hive.ql.optimizer.calcite.rules.views.HiveMaterializedViewBoxing$HiveMaterializedViewUnboxingRule.onMatch(HiveMaterializedViewBoxing.java:210)
> at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:229)
> at org.apache.calcite.plan.volcano.IterativeRuleDriver.drive(IterativeRuleDriver.java:58)
> at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:510)
> at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.applyMaterializedViewRewriting(CalcitePlanner.java:2027)
> at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1717)
> at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1589)
> at org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:131)
> at org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:914)
> at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:180)
> at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:126)
> at org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1341)
> at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:559)
> at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12549)
> at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:452)
> at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:317)
> at org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:175)
> at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:317)
> at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:223)
> at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:105)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:500)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:453)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:417)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:411)
> at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:125)
> at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:229)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:256)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:201)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:127)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:422)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:353)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)