[jira] [Commented] (HBASE-7743) Replace *SortReducers with Hadoop Secondary Sort

["Jerry He (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-7743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15181935#comment-15181935 ] 

Jerry He commented on HBASE-7743:
---------------------------------

Linked to HBASE-13897 as well.

> Replace *SortReducers with Hadoop Secondary Sort
> ------------------------------------------------
>
>                 Key: HBASE-7743
>                 URL: https://issues.apache.org/jira/browse/HBASE-7743
>             Project: HBase
>          Issue Type: Sub-task
>          Components: mapreduce, Performance
>            Reporter: Nick Dimiduk
>             Fix For: 2.0.0
>
>
> The mapreduce package provides two Reducer implementations, KeyValueSortReducer and PutSortReducer, which are used by Import, ImportTsv, and WALPlayer in conjunction with the HFileOutputFormat. Both of these implementations make use of a TreeSet to sort values matching a key. This reducer will OOM when rows are large.
> A better solution would be to implement secondary sort of the values. That way hadoop sorts the records, spilling to disk when necessary.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)