[jira] [Commented] (HIVE-9007) Hive may generate wrong plan for map join queries due to IdentityProjectRemover [Spark Branch]

["Szehon Ho (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-9007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234767#comment-14234767 ] 

Szehon Ho commented on HIVE-9007:
---------------------------------

I'll leave this JIRA for now.

One observation to note here is that it is revealed in ppd_join4.q test, if you add "set hive.auto.convert.join=true" for the test.

The plan has too many HashTableSinks.

{noformat}
STAGE DEPENDENCIES:
  Stage-1 is a root stage
  Stage-0 is a root stage

STAGE PLANS:
  Stage: Stage-1
    Spark
#### A masked pattern was here ####
      Vertices:
        Map 1 
            Map Operator Tree:
                TableScan
                  alias: test_tbl
                  Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE
                  Filter Operator
                    predicate: ((id is not null and (name = 'c')) and (id = 'a')) (type: boolean)
                    Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE
                    Select Operator
                      Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE
                      Spark HashTable Sink Operator
                        condition expressions:
                          0 
                          1 
                        keys:
                          0 'a' (type: string)
                          1 'a' (type: string)
            Local Work:
              Map Reduce Local Work
        Map 2 
            Map Operator Tree:
                TableScan
                  alias: t3
                  Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE
                  Filter Operator
                    predicate: (id = 'a') (type: boolean)
                    Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE
                    Spark HashTable Sink Operator
                      condition expressions:
                        0 
                        1 
                      keys:
                        0 'a' (type: string)
                        1 'a' (type: string)
            Local Work:
              Map Reduce Local Work

  Stage: Stage-0
    Fetch Operator
      limit: -1
      Processor Tree:
        ListSink
{noformat}

It could be related to this issue.  I'll come back to this JIRA at a later point, or others who are free can take it.

> Hive may generate wrong plan for map join queries due to IdentityProjectRemover [Spark Branch]
> ----------------------------------------------------------------------------------------------
>
>                 Key: HIVE-9007
>                 URL: https://issues.apache.org/jira/browse/HIVE-9007
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Spark
>    Affects Versions: spark-branch
>            Reporter: Chao
>            Assignee: Szehon Ho
>
> HIVE-8435 introduces a new logical optimizer called IdentityProjectRemover, which may cause map join in spark branch to generate wrong plan.
> Currently, the map join conversion in spark branch first goes through a method {{convertJoinMapJoin}}, which replaces a join op with a mapjoin op, removes RS associated with big table, and keep RSs for all small tables. Afterwards, in {{SparkReduceSinkMapJoinProc}} it replaces all parent RSs of the mapjoin op with HTS (note it doesn't check whether the RS belongs to small table or big table.)
> The issue arises, when IdentityProjectRemover comes into play, which may result into a situation that a operator tree has two consecutive RSs. Imaging the following example:
> {noformat}
>           Join               MapJoin
>           / \                /   \
>         RS   RS   --->     RS     RS
>        /      \           /         \
>       TS       RS       TS          TS (big table)
>                 \      (small table)
>                  TS
> {noformat}
> In this case, all parents of the mapjoin op will be RS, even the branch for big table! In {{SparkReduceSinkMapJoinProc}}, they will be replaced with HTS, which is obviously incorrect.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)