[jira] [Created] (HIVE-27006) ParallelEdgeFixer inserts misconfigured operator and does not connect it in Tez DAG

["Seonggon Namgung (Jira)" <ji...@apache.org>]

Seonggon Namgung created HIVE-27006:
---------------------------------------

             Summary: ParallelEdgeFixer inserts misconfigured operator and does not connect it in Tez DAG
                 Key: HIVE-27006
                 URL: https://issues.apache.org/jira/browse/HIVE-27006
             Project: Hive
          Issue Type: Bug
            Reporter: Seonggon Namgung
            Assignee: Seonggon Namgung
         Attachments: after.PEF.png, tez-dag.png

Hive fails to run the below query on 1TB ORC formatted TPC-DS dataset because of runtime error happens in one Operator.
I found that the problematic operator is inserted by ParallelEdgeFixer.
Also I observed that the corresponding vertex has no descendant vertex although its ReduceSinkOperator has a SemiJoin edge connected to TableScanOperator.
(I attached the figure of Tez DAG and OperatorGraph. One can check that Cluster6 and Cluster7 are connected while Reducer4 and Map7 are not.)

 

Query
{code:java}
set hive.optimize.shared.work=true;
set hive.optimize.shared.work.parallel.edge.support=true;
with
  inv00 as (select inv_item_sk, inv_warehouse_sk from inventory, date_dim where inv_date_sk = d_date_sk and d_year = 2000),
  inv01 as (select inv_item_sk, inv_warehouse_sk from inventory, date_dim where inv_date_sk = d_date_sk and d_year = 2001),
  inv02 as (select inv_item_sk, inv_warehouse_sk from inventory, date_dim where inv_date_sk = d_date_sk and d_year = 2002),
  sd00 as (select inv_item_sk id, w_zip zip from inv00 full outer join warehouse on inv_warehouse_sk = w_warehouse_sk where w_state = 'SD'),
  sd01 as (select inv_item_sk id, w_zip zip from inv01 full outer join warehouse on inv_warehouse_sk = w_warehouse_sk where w_state = 'SD'),
  sd02 as (select inv_item_sk id, w_zip zip from inv02 full outer join warehouse on inv_warehouse_sk = w_warehouse_sk where w_state = 'SD')
select * from sd00, sd01, sd02 where sd00.id = sd01.id and sd00.id = sd02.id; {code}
 

Error message
{code:java}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row
        at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:385)
        at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:301)
        ... 18 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: cannot find field _col0 from []
        at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:384)
        at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
        at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:94)
        at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:370)
        ... 19 more
Caused by: java.lang.RuntimeException: cannot find field _col0 from []
        at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:550)
        at org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:153)
        at org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:56)
        at org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:1073)
        at org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:1099)
        at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:305)
        ... 22 more {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)