You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "gaozhan ding (Jira)" <ji...@apache.org> on 2021/08/17 09:25:00 UTC
[jira] [Commented] (HIVE-25456) MapJoin operator shoud not be
vectorized when Projections contains complex types
[ https://issues.apache.org/jira/browse/HIVE-25456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17400282#comment-17400282 ]
gaozhan ding commented on HIVE-25456:
-------------------------------------
In addition, I found https://issues.apache.org/jira/browse/HIVE-24647 and https://issues.apache.org/jira/browse/HIVE-19238 may be the same issue with this ticket.
> MapJoin operator shoud not be vectorized when Projections contains complex types
> --------------------------------------------------------------------------------
>
> Key: HIVE-25456
> URL: https://issues.apache.org/jira/browse/HIVE-25456
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
> Affects Versions: 3.1.0
> Reporter: gaozhan ding
> Priority: Major
>
> reproduce step：
> {code:java}
> create table map_test1(str string,str_map map<string, string>);
> create table map_test2(str string,str_map map<string, string>);
> insert into map_test1 values('xxx',map('key','value')),('yyy',map('key','value'));
> insert into map_test2 values('xxx',map('key','value')),('yyy',map('key','value'));
> set hive.vectorized.execution.enabled=true;
> set hive.auto.convert.join=true;
> select a.str_map from map_test1 a join map_test2 b on a.str=b.str;{code}
> Then ClassCastException occurs.
> {panel:title=My title}
> java.lang.RuntimeException: java.lang.RuntimeException: Hive Runtime Error while closing operators
> at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
> at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
> at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
> at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.RuntimeException: Hive Runtime Error while closing operators
> at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.close(MapRecordProcessor.java:495)
> at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:284)
> ... 16 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.objectinspector.StandardMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector
> at org.apache.hadoop.hive.ql.exec.MapJoinOperator.closeOp(MapJoinOperator.java:621)
> at org.apache.hadoop.hive.ql.exec.vector.VectorMapJoinBaseOperator.closeOp(VectorMapJoinBaseOperator.java:258)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:732)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:756)
> at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.close(MapRecordProcessor.java:484)
> ... 17 more
> Caused by: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.objectinspector.StandardMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector
> at org.apache.hadoop.hive.ql.exec.vector.VectorizedBatchUtil.setVector(VectorizedBatchUtil.java:306)
> at org.apache.hadoop.hive.ql.exec.vector.VectorizedBatchUtil.addProjectedRowToBatchFrom(VectorizedBatchUtil.java:254)
> at org.apache.hadoop.hive.ql.exec.vector.VectorMapJoinBaseOperator.reProcessBigTable(VectorMapJoinBaseOperator.java:287)
> at org.apache.hadoop.hive.ql.exec.MapJoinOperator.continueProcess(MapJoinOperator.java:681)
> at org.apache.hadoop.hive.ql.exec.MapJoinOperator.closeOp(MapJoinOperator.java:614)
> ... 21 more
> {panel}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)