[jira] [Commented] (HIVE-7541) Support union all on Spark

["Xuefu Zhang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-7541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14080920#comment-14080920 ] 

Xuefu Zhang commented on HIVE-7541:
-----------------------------------

[~nyang] Thanks for working on this. This task is fairly large, and I think breaking the task into smaller ones would help in sharing the load and tracking progress. Could you please create smaller JIRAs for this? I'd image that supporting union would require work in the following area:

1. SparkCompiler changes: generate a SparkWork that contains UnionWork from logical operator tree.
2. SparkPlan modeling: represent the spark job in terms of a graph (rather than) list of SparkTran instances. We may need to enhance SparkTran interface. 
3. SparkPlanGenerator: need to generate a plan from SparkWork, which needs to use Spark's union transformation to achieve the functionality..
4. other earas.

Tez can be a good reference point

Please feel free to create JIRAs for those or other areas.

> Support union all on Spark
> --------------------------
>
>                 Key: HIVE-7541
>                 URL: https://issues.apache.org/jira/browse/HIVE-7541
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Spark
>            Reporter: Xuefu Zhang
>            Assignee: Na Yang
>
> For union all operator, we will use Spark's union transformation. Refer to the design doc on wiki for more information.



--
This message was sent by Atlassian JIRA
(v6.2#6252)