[jira] [Commented] (HIVE-8024) Find out whether it's possible to remove UnionOperator from original operator tree [Spark Branch]

["Chao (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-8024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14126398#comment-14126398 ] 

Chao commented on HIVE-8024:
----------------------------

I think there's a problem removing union op in place. Suppose there's a plan for multi-insertion looks like this:

{code}
      TS_0       TS_2
        \        /
          UNION_3
        /       \
      SEL_1    SEL_4
{code}

(I ignored some operators)
Currently, {{TS_0}} and {{TS_2}} will be in two MapWorks, which have separate plans, like following:

{code}
          TS_0              TS_2
         /    \            /    \
      SEL_1   SEL_4      SEL_1  SEL_4
{code}

If we remove the union operator from the original tree, the result may not be correct.

> Find out whether it's possible to remove UnionOperator from original operator tree [Spark Branch]
> -------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-8024
>                 URL: https://issues.apache.org/jira/browse/HIVE-8024
>             Project: Hive
>          Issue Type: Task
>          Components: Spark
>            Reporter: Chao
>            Assignee: Chao
>
> Currently, after operator tree is processed, the generated works with union operators will go through {{GenSparkUtils::removeUnionOperators}}, which will clone the original operator plan associated with the work, and remove union operators in it. This caused some issues as seen, for example, in HIVE-7870. This JIRA is created to find out whether it's possible to just remove the union operators in the original plan.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)