[jira] [Commented] (HIVE-22198) Execute unoin-all with childs Join in parallel

["LuGuangMing (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-22198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934016#comment-16934016 ] 

LuGuangMing commented on HIVE-22198:
------------------------------------

【analyze】

explain  SELECT sum(1) as a 
FROM tab1 t1 
INNER JOIN tab2 t2 
ON t1.com = t2.com
UNION ALL
SELECT sum(1) as a 
FROM tab3 t3 
INNER JOIN tab4 t4 
ON t3.com = t4.com;

!image-2019-09-20-11-38-37-433.png|width=456,height=176!!image-2019-09-20-11-39-30-347.png|width=378,height=303!

Analysis explian plan and resouce code, shows that stage-3 is added to the runnable queue when one of stage-2 or stage-6 runs is finished, due to ConditionalTask stage-9 and ConditionalTask stage-11 was invoked by parallel, remove the parent dependency of stage-3 on stage-2 and stage-6, The Driver scheduler finds that one of the stage-2 or stage-6 runtimes is finished, The children stage-3 will be found for scheduling add to runnable queue to run, At this time, there is still one node in stage-2 or stage-6 that has not run finished, and the result of the stage that has not finished will not be read by stage-3, so part of the result data is missing

> Execute unoin-all with childs Join in parallel
> ----------------------------------------------
>
>                 Key: HIVE-22198
>                 URL: https://issues.apache.org/jira/browse/HIVE-22198
>             Project: Hive
>          Issue Type: Bug
>    Affects Versions: 1.2.0, 3.0.0, 3.1.0
>            Reporter: LuGuangMing
>            Assignee: LuGuangMing
>            Priority: Major
>         Attachments: image-2019-09-20-11-38-37-433.png, image-2019-09-20-11-39-30-347.png, test-parallel.sql
>
>
> set parallel is true, set skewjoin is false, set auto convert join is false. run a unoin all, There is nothing error message, but some result data is missing, details check attatchment sql file
> create table tab1(tid int, com string) row format delimited fields terminated by '\t' stored as textfile;
>  create table tab2(tid int, com string) row format delimited fields terminated by '\t' stored as textfile;
>  create table tab3(tid int, com string) row format delimited fields terminated by '\t' stored as textfile;
>  create table tab4(tid int, com string) row format delimited fields terminated by '\t' stored as textfile;
> insert into tab1 values(1,'abc'),(2,'bcd'),(3,'cde');
>  insert into tab2 values(1,'abc'),(2,'bcd'),(3,'cde');
>  insert into tab3 values(1,'abc'),(2,'bcd'),(3,'cde');
>  insert into tab4 values(1,'abc'),(2,'bcd'),(3,'cde');
> set hive.auto.convert.join=false;
>  set hive.optimize.skewjoin=true;
>  set hive.exec.parallel=true;
> SELECT sum(1) as a 
>  FROM tab1 t1 
>  INNER JOIN tab2 t2 
>  ON t1.com = t2.com
>  UNION ALL
>  SELECT sum(1) as a 
>  FROM tab3 t3 
>  INNER JOIN tab4 t4 
>  ON t3.com = t4.com;
> create table test_parallel stored as orcfile as 
>  SELECT sum(1) as a 
>  FROM tab1 t1 
>  INNER JOIN tab2 t2 
>  ON t1.com = t2.com
>  UNION ALL
>  SELECT sum(1) as a 
>  FROM tab3 t3 
>  INNER JOIN tab4 t4 
>  ON t3.com = t4.com;
> select * from test_parallel;
> The result data should be two, but only one.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)