[jira] [Assigned] (HIVE-24080) CBO failed: Array and Map are not support when convert calcite types to hive types

["范宜臻 (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HIVE-24080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

范宜臻 reassigned HIVE-24080:
--------------------------

    Assignee: Gunther Hagleitner

> CBO failed: Array and Map are not support when convert calcite types to hive types
> ----------------------------------------------------------------------------------
>
>                 Key: HIVE-24080
>                 URL: https://issues.apache.org/jira/browse/HIVE-24080
>             Project: Hive
>          Issue Type: Bug
>          Components: CBO, Logical Optimizer
>    Affects Versions: 3.0.0, 2.3.4
>            Reporter: 范宜臻
>            Assignee: Gunther Hagleitner
>            Priority: Critical
>
> {code:java}
> drop table temp.test_cbo_fyz_right;
> CREATE EXTERNAL TABLE temp.test_cbo_right(
>   `rid` string,
>   `dtm` string,
>   `map` map<string, string>,
>   `tags` array <string>
> );drop table temp.test_cbo_fyz_left;
> CREATE EXTERNAL TABLE temp.test_cbo_left(
>   `lid` string,
>   `dtm` string
> )
> explain select
> b.tags
> from temp.test_cbo_left a
> left join temp.test_cbo_right b on a.lid = b.rid
> where b.dtm = '20200814'
> and a.dtm = '20200813'{code}
>  
> The Problem will occur in  left join and select array type or map type column in right table(reverse for right join or both for full join).
> code will throw NPE when `calcitePlanner` generate `optimizedOptiqPlan` from calcite and then convert it back to `optimizedAST`, the projection of b.tags has been optimized to be  RexCall => CAST(b.tags) in calcite code. Normally the projection should be an RexInputRef, but in this case, calcite need cast `not null` to `nullable`.  However, TypeConverter in ASTConverter has not support ARRAY and MAP.
>  
> NPE threw here:
> org.apache.hadoop.hive.ql.optimizer.calcite.translator.ASTConverter.RexVisitor visitCall
> {code:java}
> HiveToken ht = TypeConverter.hiveToken(call.getType()); // here ht is null
> ASTBuilder astBldr = ASTBuilder.construct(ht.type, ht.text);
> if (ht.args != null) {
>   for (String castArg : ht.args)
>     astBldr.add(HiveParser.Identifier, castArg);
> }
> {code}
> The final result is cbo failed and skip CalcitePlan and cause partition pruning failed.
> when table is large, sql will explode.
>  
>  I wonder which repo should fix this problem?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)