[jira] [Comment Edited] (TEZ-145) Support a combiner processor that can run non-local to map/reduce nodes

["Hitesh Shah (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13802382#comment-13802382 ] 

Hitesh Shah edited comment on TEZ-145 at 10/22/13 10:50 PM:
------------------------------------------------------------

[~ozawa] Please go ahead and take a crack at it if you like. I have not started on any work related to this jira as of now. 

I will propose what I was thinking of as a possible option for a solution with respect to the current features of Tez. Feel free to change/suggest/tear apart the proposal/suggest improvements for core functionality in tez. 

Possible design: 

Background info: Tez supports concepts of vertices and edges. As part of each edge and vertex, there is possibility for a user to plug-in some user logic to affect different aspects of the run-time. Currently, there are some pieces of this implemented to support things such as dynamic sizing of the no. of reduce tasks to run. Via events, information of the outputs of a map stage can be sampled in the AM to determine how many reducers to run. Once this is decided, the user logic in the AM can then route the information of map outputs ( within DataMovementEvents) to the appropriate reducer to ensure that partitions are assigned correctly.

Today, a MapReduce job consists of a Map vertex connected to a Reduce vertex via a Shuffle edge. For the above, I was thinking along the lines of a Map vertex followed by a Combiner Vertex which is then connected to the Reduce Vertex. The edge between the Map and combiner vertex could also just be a shuffle. 
Using a similar approach for reducer dynamism, the combiner vertex could use events generated by the framework to learn about the locations of where the Map tasks are running. Based on this, the user logic could then decide how many tasks to create for the combiner vertex ( For example, one per physical node or one per rack ) and also define the locality requirements. Note, the shuffle edge works by the map task generating an event publishing the location of the map output which is then passed to the next stage's input. Using this, there could be various optimizations done too. In some cases, the combiner vertex may decide to do no work and therefore pass the event generated by the map directly to the reduce without doing any work. This may require changes in the current shuffle input/output pairs though. 

Tez is still sometime away before we can dynamically introduce new vertices into the DAG. At some point, the combiner vertex would be dynamically introduced by user-logic but at this time, it might be a good start to implement it via a static DAG with optimizations to bypass it as needed.

There is some reference information here: http://hortonworks.com/hadoop/tez/. ( We plan to create better docs and publish to the apache tez website soon ).


















 


was (Author: hitesh):
[~ozawa] Please go ahead and take a crack at it if you like. I have not started on any work related to this jira as of now. 

I will propose what I was thinking of as a possible option for a solution with respect to the current features of Tez. Feel free to change/suggest/tear apart the proposal/suggest improvements for core functionality in tez. 

Possible design: 

Background info: Tez supports concepts of vertices and edges. As part of each edge and vertex, there is possibility for a user to plug-in some user logic to affect different aspects of the run-time. Currently, there are some pieces of this implemented to support things such as dynamic sizing of the no. of reduce tasks to run. Via events, information of the outputs of a map stage can be sampled in the AM to determine how many reducers to run. Once this is decided, the user logic in the AM can then route the information of map outputs ( within DataMovementEvents) to the appropriate reducer to ensure that partitions are assigned correctly.

Today, a MapReduce job consists of a Map vertex connected to a Reduce vertex via a Shuffle edge. For the above, I was thinking along the lines of a Map vertex followed by a Combiner Vertex which is then connected to the Reduce Vertex. The edge between the Map and combiner vertex could also just be a shuffle. 
Using a similar approach for reducer dynamism, the combiner vertex could use events generated by the framework to learn about the locations of where the Map tasks are running. Based on this, the user logic could then decide how many tasks to create for the combiner vertex ( For example, one per physical node or one per rack ) and also define the locality requirements. Note, the shuffle edge works by the map task generating an event publishing the location of the map output which is then passed to the next stage's input. Using this, there could be various optimizations done too. In some cases, the combiner vertex may decide to do no work and therefore pass the event generated by the map directly to the reduce without doing any work. This may require changes in the current shuffle input/output pairs though. 

Tez is still sometime away before we can dynamically introduce new vertices into the DAG. At some point, the combiner vertex would be dynamically introduced by user-logic but at this time, it might be a good start to implement it via a static DAG with optimizations to bypass it as needed.

There is some reference information here: http://hortonworks.com/hadoop/tez/. ( We plan to create better docs and publish to the tez website soon ).


















 

> Support a combiner processor that can run non-local to map/reduce nodes
> -----------------------------------------------------------------------
>
>                 Key: TEZ-145
>                 URL: https://issues.apache.org/jira/browse/TEZ-145
>             Project: Apache Tez
>          Issue Type: Bug
>            Reporter: Hitesh Shah
>            Assignee: Hitesh Shah
>              Labels: TEZ-1
>
> For aggregate operators that can benefit by running in multi-level trees, support of being able to run a combiner in a non-local mode would allow performance efficiencies to be gained by running a combiner at a rack-level. 



--
This message was sent by Atlassian JIRA
(v6.1#6144)