[jira] [Created] (HIVE-22294) ConditionalWork cannot be cast to MapredWork When both skew.join and auto.convert is on.

["Qiang.Kang (Jira)" <ji...@apache.org>]

Qiang.Kang created HIVE-22294:
---------------------------------

             Summary: ConditionalWork cannot be cast to MapredWork  When both skew.join and auto.convert is on.  
                 Key: HIVE-22294
                 URL: https://issues.apache.org/jira/browse/HIVE-22294
             Project: Hive
          Issue Type: Bug
          Components: Physical Optimizer
    Affects Versions: 2.3.4, 3.1.1, 2.3.0
            Reporter: Qiang.Kang


Our hive version is 1.2.1 which has merged some patches (including patches mentioned  in https://issues.apache.org/jira/browse/HIVE-14557, https://issues.apache.org/jira/browse/HIVE-16155 ) .

 

My sql query string is like this:

```

set hive.auto.convert.join = true;

set hive.optimize.skewjoin=true;

 

SELECT a.*

FROM

a

JOIN b

ON a.id=b.id AND a.uid = b.uid 

LEFT JOIN c

ON b.id=c.id AND b.uid=c.uid;

```

And we met some error: 

FAILED: ClassCastException org.apache.hadoop.hive.ql.plan.ConditionalWork cannot be cast to org.apache.hadoop.hive.ql.plan.MapredWork

 

The main reason is that there is a conditional task (*MapJoin*) in the list tasks of another Conditional task (*SkewJoin*).  Here is the code snippet where it throws this exception:

`org.apache.hadoop.hive.ql.optimizer.physical.MapJoinResolver:`

```java

public Object dispatch(Node nd, Stack<Node> stack, Object... nodeOutputs)
 throws SemanticException {
 Task<? extends Serializable> currTask = (Task<? extends Serializable>) nd;
 // not map reduce task or not conditional task, just skip
 if (currTask.isMapRedTask()) {
 if (currTask instanceof ConditionalTask) {
 // get the list of task
 List<Task<? extends Serializable>> taskList = ((ConditionalTask) currTask).getListTasks();
 for (Task<? extends Serializable> tsk : taskList) {
 if (tsk.isMapRedTask()) {

 

//  ATTENTION: tsk May be ConditionalTask !!!
 this.processCurrentTask(tsk, ((ConditionalTask) currTask));
 }
 }
 } else {
 this.processCurrentTask(currTask, null);
 }
 }
 return null;
}

private void processCurrentTask(Task<? extends Serializable> currTask,
 ConditionalTask conditionalTask) throws SemanticException {
 // get current mapred work and its local work
 MapredWork mapredWork = (MapredWork) currTask.getWork(); // WRONG!!!!!!
 MapredLocalWork localwork = mapredWork.getMapWork().getMapRedLocalWork();

```

 

Here is some detail Information about query plan:

*-  set hive.auto.convert.join = true; set hive.optimize.skewjoin=false;*

```

Stage-1 is a root stage [a join b]
 Stage-12 [map join]depends on stages: Stage-1 , consists of Stage-13, Stage-2
 Stage-13 has a backup stage: Stage-2
 Stage-11 depends on stages: Stage-13
 Stage-8 depends on stages: Stage-2, Stage-11 , consists of Stage-5, Stage-4, Stage-6
 Stage-5
 Stage-0 depends on stages: Stage-5, Stage-4, Stage-7
 Stage-14 depends on stages: Stage-0
 Stage-3 depends on stages: Stage-14
 Stage-4
 Stage-6
 Stage-7 depends on stages: Stage-6
 Stage-2

```

*-  set hive.auto.convert.join = false; set hive.optimize.skewjoin=true;*

```

STAGE DEPENDENCIES:
 Stage-1 is a root stage
 Stage-12 depends on stages: Stage-1 , consists of Stage-13, Stage-2
 Stage-13 [skew Join map local task]
 Stage-11 depends on stages: Stage-13
 Stage-2 depends on stages: Stage-11
 Stage-8 depends on stages: Stage-2 , consists of Stage-5, Stage-4, Stage-6
 Stage-5
 Stage-0 depends on stages: Stage-5, Stage-4, Stage-7
 Stage-14 depends on stages: Stage-0
 Stage-3 depends on stages: Stage-14
 Stage-4
 Stage-6
 Stage-7 depends on stages: Stage-6

```

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)