[jira] [Updated] (MAPREDUCE-4502) Multi-level aggregation with combining the result of maps per node/rack

["Tsuyoshi OZAWA (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/MAPREDUCE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsuyoshi OZAWA updated MAPREDUCE-4502:
--------------------------------------

    Attachment: MAPREDUCE-4525-pof.diff

I've prototyped proof of concept code of node-level aggregation. I made sure that it works well, so I share it! Current situation is as follows:

* Node level aggregation in distributed mode is supported. I made sure of the correctness of the result by using diff command.
* Fault tolerance is partially supported. At design level, it can be supported(WIP).
* No test codes.
* No switch on/off feature.

Quick benchmark is as follows:

|Default Hadoop(sec)|Node level aggregation(sec)|
|105.0|82.0|

Experiment Environment is as follows:

* Input file is 1GB, generated by RandomTextWriter.
* 8 CPU/machines.
* 2 machines.

I'm appreciate if you review the design and code.

And, I have several questions to deal with:
1. How can I write tests such a complex case? Any ideas or references?
2. I've extended mr_protos.proto. Is it acceptable?

My next step is creating patch sets with tests and fully fault tolerance support to merge it into trunk. I'm also planning to benchmark of node-level aggregation. I'll show you here when I do it. Note that design note is outdated, so I'm updating it now.

If you'd like to track changes my code changes, you can see it on [my github|https://github.com/oza/hadoop-common/tree/MAPREDUCE-4525-Node-level-aggregation].
                
> Multi-level aggregation with combining the result of maps per node/rack
> -----------------------------------------------------------------------
>
>                 Key: MAPREDUCE-4502
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4502
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: applicationmaster, mrv2
>            Reporter: Tsuyoshi OZAWA
>            Assignee: Tsuyoshi OZAWA
>         Attachments: design_v2.pdf, MAPREDUCE-4525-pof.diff, speculative_draft.pdf
>
>
> The shuffle costs is expensive in Hadoop in spite of the existence of combiner, because the scope of combining is limited within only one MapTask. To solve this problem, it's a good way to aggregate the result of maps per node/rack by launch combiner.
> This JIRA is to implement the multi-level aggregation infrastructure, including combining per container(MAPREDUCE-3902 is related), coordinating containers by application master without breaking fault tolerance of jobs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira