[jira] [Updated] (MAPREDUCE-4808) Allow reduce-side merge to be pluggable

["Arun C Murthy (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/MAPREDUCE-4808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Arun C Murthy updated MAPREDUCE-4808:
-------------------------------------

    Status: Open  (was: Patch Available)

Asokan, sorry I've been away traveling home during the holidays and hence the delay.

I have more comments, but I'll put some here to keep the discussion going.

Thanks for the design doc, but I was looking for thoughts on *how* the plugin was going used for use-cases you've mentioned (hash-join etc.), alternatives on design etc. 

IAC, taking a step back, the 'goal' here is to make the 'merge' pluggable.

Reduce-side has 2 pieces:
# Shuffle - Move data from maps to the reduce.
# Merge - Merge already sorted map-outputs.

The rest (MergeManager etc.) are merely implementation details to manage memory etc., which are irrelevant in several scenarios as soon as we consider alternatives to the current HTTP-based shuffle (several alternatives exist such RDMA etc.).

Your current approach tries to encapsulate and enshrine the current implementation of the reduce task, which I'm not wild about. By this I mean, you are focussing too much on the current state and trying to make interfaces which are unnecessary for now and might not suffice for the future.

I really don't think we should be tying Shuffle & Merge as you have done by introducing yet another new interface (regardless of whether it's public or not).


As I've noted above, adding a simple 'Merge' interface with one 'merge' call will address all of the use-cases you have outlined. If not, let's discuss.

                
> Allow reduce-side merge to be pluggable
> ---------------------------------------
>
>                 Key: MAPREDUCE-4808
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4808
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>    Affects Versions: 2.0.2-alpha
>            Reporter: Arun C Murthy
>            Assignee: Mariappan Asokan
>             Fix For: 2.0.3-alpha
>
>         Attachments: COMBO-mapreduce-4809-4812-4808.patch, mapreduce-4808.patch, mapreduce-4808.patch, mapreduce-4808.patch, mapreduce-4808.patch, mapreduce-4808.patch, MergeManagerPlugin.pdf
>
>
> Allow reduce-side merge to be pluggable for MAPREDUCE-2454

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira