You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Stamatis Zampetakis (Jira)" <ji...@apache.org> on 2024/04/25 12:16:00 UTC
[jira] [Commented] (HIVE-28218) AssertionError in VolcanoPlanner#fixUpInputs during MV rewritting
[ https://issues.apache.org/jira/browse/HIVE-28218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17840775#comment-17840775 ]
Stamatis Zampetakis commented on HIVE-28218:
--------------------------------------------
The CBO plan before the MV rewritting kicks in is shown below.
{noformat}
HiveJoin(condition=[true], joinType=[inner], algorithm=[none], cost=[not available])
HiveProject(empid=[$0], deptno=[$1], name=[$2], salary=[$3], commission=[$4])
HiveTableScan(table=[[default, emps_acid]], table:alias=[e])
HiveJdbcConverter(convention=[JDBC.DERBY])
JdbcHiveTableScan(table=[[default, depts_jdbc]], table:alias=[d])
{noformat}
In order for the problem to surface it is necessary to have a HiveJdbcConverter (or basically any instance of org.apache.calcite.rel.convert.Converter) directly under a HiveJoin. The ACID part of the join as well as anything under the HiveJdbcConverter can be arbitrarily complex.
> AssertionError in VolcanoPlanner#fixUpInputs during MV rewritting
> -----------------------------------------------------------------
>
> Key: HIVE-28218
> URL: https://issues.apache.org/jira/browse/HIVE-28218
> Project: Hive
> Issue Type: Bug
> Components: CBO
> Affects Versions: 4.0.0
> Reporter: Stamatis Zampetakis
> Assignee: Stamatis Zampetakis
> Priority: Major
> Attachments: materialized_view_rewrite_acid_jdbc_tables.q
>
>
> The minimal scenario to reproduce the problem requires one ACID table, one JDBC table, and a view over the ACID table so that materialized view (MV) rewritting triggers. A minimal set of DDLs is outlined below.
> {code:sql}
> CREATE TABLE emps_acid ( empid int, deptno int) STORED AS ORC TBLPROPERTIES ('transactional'='true');
> CREATE MATERIALIZED VIEW mv_emps AS SELECT * FROM emps_acid;
> CREATE EXTERNAL TABLE depts_jdbc (deptno int, name varchar(256)) STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler' TBLPROPERTIES (...);
> {code}
> {code:sql}
> EXPLAIN CBO SELECT * FROM emps_acid e JOIN depts_jdbc d;
> {code}
> When the MV rewritting is enabled the JOIN query above fails with the following stacktrace.
> {noformat}
> java.lang.AssertionError
> at org.apache.calcite.plan.volcano.VolcanoPlanner.fixUpInputs(VolcanoPlanner.java:1011)
> at org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1186)
> at org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:589)
> at org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:604)
> at org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:148)
> at org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:268)
> at org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:283)
> at org.apache.hadoop.hive.ql.optimizer.calcite.rules.views.HiveMaterializedViewBoxing$HiveMaterializedViewUnboxingRule.onMatch(HiveMaterializedViewBoxing.java:206)
> at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:229)
> at org.apache.calcite.plan.volcano.IterativeRuleDriver.drive(IterativeRuleDriver.java:58)
> at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:510)
> at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.applyMaterializedViewRewriting(CalcitePlanner.java:2053)
> at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1715)
> at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1572)
> at org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:131)
> at org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:914)
> at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:180)
> at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:126)
> at org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1324)
> at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:572)
> at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:13178)
> at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:465)
> at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:180)
> at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:224)
> at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:107)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:519)
> {noformat}
> A full reproducer is attached in [^materialized_view_rewrite_acid_jdbc_tables.q] and can be run using:
> {noformat}
> mvn test -Dtest=TestMiniLlapLocalCliDriver -Dqfile=materialized_view_rewrite_acid_jdbc_tables.q -Dtest.output.overwrite
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)