[jira] [Comment Edited] (MAPREDUCE-2454) Allow external sorter plugin for MR

["Alejandro Abdelnur (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495660#comment-13495660 ] 

Alejandro Abdelnur edited comment on MAPREDUCE-2454 at 11/12/12 9:42 PM:
-------------------------------------------------------------------------

Asokan,

Thanks for patience working out the design and implementation of this JIRA.

The latest patch looks good and it addresses the concerns voiced regarding big changes that could destabilize the MR framework.

There are few minor NITs that should be addressed in the patch (at the end of this comment).

+1 after these NITs are addressed and jenkins test-patch OKs the new patch.

Patch NITs:

* PreReduceProcessor.java: unused import: Shuffle
* ReduceTasks.java: line 357, PreReduceProcessor merger = (PreReduceProcessor) ReflectionUtils.newInstance(..., no need for the "(PreReduceProcessor)" casting.
* Shuffle.java, unused imports: FileSystem, LocalDirAllocator, CompressionCodec, MapOutputFile, RawKeyValueIterator, Reducer, CombineOutputCollector
* TestLimitNQuery.java, unused imports: FileInputStream, FileNotFoundException, FileStatus, JobCounter, TaskAttemptID, ReflectionUtils

                
      was (Author: tucu00):
    
Asokan,

Thanks for patience working out the design and implementation of this JIRA.

The latest patch looks good and it addresses the concerns voiced regarding big changes that could destabilize the MR framework.

There are few minor NITs that should be addressed in the patch (at the end of this comment).

+1 after these NITs are addresses and jenkins test-patch OKs the new patch.

Patch NITs:

* PreReduceProcessor.java: unused import: Shuffle
* ReduceTasks.java: line 357, PreReduceProcessor merger = (PreReduceProcessor) ReflectionUtils.newInstance(..., no need for the "(PreReduceProcessor)" casting.
* Shuffle.java, unused imports: FileSystem, LocalDirAllocator, CompressionCodec, MapOutputFile, RawKeyValueIterator, Reducer, CombineOutputCollector
* TestLimitNQuery.java, unused imports: FileInputStream, FileNotFoundException, FileStatus, JobCounter, TaskAttemptID, ReflectionUtils

                  
> Allow external sorter plugin for MR
> -----------------------------------
>
>                 Key: MAPREDUCE-2454
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2454
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>    Affects Versions: 2.0.0-alpha, 3.0.0, 2.0.2-alpha
>            Reporter: Mariappan Asokan
>            Assignee: Mariappan Asokan
>            Priority: Minor
>              Labels: features, performance, plugin, sort
>         Attachments: HadoopSortPlugin.pdf, HadoopSortPlugin.pdf, KeyValueIterator.java, MapOutputSorterAbstract.java, MapOutputSorter.java, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mr-2454-on-mr-279-build82.patch.gz, MR-2454-trunkPatchPreview.gz, ReduceInputSorter.java
>
>
> Define interfaces and some abstract classes in the Hadoop framework to facilitate external sorter plugins both on the Map and Reduce sides.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira