You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Artem Velykorodnyi (JIRA)" <ji...@apache.org> on 2019/07/22 13:39:00 UTC
[jira] [Created] (HIVE-22031) HiveRelDecorrelator fails with
IndexOutOfBoundsException if the query contains several "constant" columns
Artem Velykorodnyi created HIVE-22031:
-----------------------------------------
Summary: HiveRelDecorrelator fails with IndexOutOfBoundsException if the query contains several "constant" columns
Key: HIVE-22031
URL: https://issues.apache.org/jira/browse/HIVE-22031
Project: Hive
Issue Type: Bug
Components: CBO
Affects Versions: 2.3.5
Reporter: Artem Velykorodnyi
Assignee: Artem Velykorodnyi
Steps for reproducing:
{code}
1. Create table orders
create table orders (ORD_NUM INT, CUST_CODE STRING);
2. Create table customers
create table customers (CUST_CODE STRING);
3. Make select with constants and with a subquery:
select DISTINCT(CUST_CODE), '777' as ANY, ORD_NUM, '888' as CONSTANT
from orders
WHERE not exists
(select 1
from customers
WHERE CUST_CODE=orders.CUST_CODE
);
{code}
Query fails with IndexOutOfBoundsException
{code}
Exception in thread "main" java.lang.AssertionError: Internal error: While invoking method 'public org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelDecorrelator$Frame org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelDecorrelator.decorrelateRel(org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveProject) throws org.apache.hadoop.hive.ql.parse.SemanticException'
at org.apache.calcite.util.Util.newInternal(Util.java:792)
at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:534)
at org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelDecorrelator.getInvoke(HiveRelDecorrelator.java:660)
at org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelDecorrelator.decorrelate(HiveRelDecorrelator.java:252)
at org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelDecorrelator.decorrelateQuery(HiveRelDecorrelator.java:218)
at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1347)
at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1261)
at org.apache.calcite.tools.Frameworks$1.apply(Frameworks.java:113)
at org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:997)
at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:149)
at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:106)
at org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1069)
at org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedAST(CalcitePlanner.java:1085)
at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:364)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11138)
at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:286)
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:512)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:531)
... 32 more
Caused by: java.lang.AssertionError: Internal error: While invoking method 'public org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelDecorrelator$Frame org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelDecorrelator.decorrelateRel(org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveAggregate) throws org.apache.hadoop.hive.ql.parse.SemanticException'
at org.apache.calcite.util.Util.newInternal(Util.java:792)
at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:534)
at org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelDecorrelator.getInvoke(HiveRelDecorrelator.java:660)
at org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelDecorrelator.decorrelateRel(HiveRelDecorrelator.java:854)
... 37 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:531)
... 39 more
Caused by: java.lang.IndexOutOfBoundsException: Index: 3, Size: 2
at java.util.ArrayList.rangeCheckForAdd(ArrayList.java:665)
at java.util.ArrayList.add(ArrayList.java:477)
at org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelDecorrelator.decorrelateRel(HiveRelDecorrelator.java:833)
... 44 more
{code}
HiveRelDecorrelator looking for omitted constants and put them into TreeMap where keys are the numbers of columns in top-level select query.
For query from example TreeMap contains:
{code}
0 = {TreeMap$Entry@8389} "1" -> "_UTF-16LE'777'"
1 = {TreeMap$Entry@8390} "3" -> "_UTF-16LE'888'"
{code}
After that, there is step where List of fields is combined with contsants from TreeMap
{code}
if (!omittedConstants.isEmpty()) {
final List<RexNode> postProjects = new ArrayList<>(relBuilder.fields());
for (Map.Entry<Integer, RexLiteral> entry
: omittedConstants.descendingMap().entrySet()) {
postProjects.add(entry.getKey() + frame.corDefOutputs.size(),
entry.getValue());
}
relBuilder.project(postProjects);
}
{code}
But TreeMap is descending, so firstly goes constant columns with high position number, greater than target List size.
(For query from example there is an attempt to add an element to the List with index 3, but the size of List is only 2).
If we use TreeMap without descending - everything goes as expected. Also, there is no difference between descending and ascending map, because the List is filled using indexes but not sequential position.
"Q file" with the query from the example works fine but fails on the Hive 2.3.5.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)