[jira] Commented: (HBASE-987) We need a Hbase Partitioner for TableMapReduceUtil.initTableReduceJob MR Jobs

["Billy Pearson (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12646198#action_12646198 ] 

Billy Pearson commented on HBASE-987:
-------------------------------------

Ran a import job importing 9,972,859 rows in to 8 regions on 4 servers

With out HRegionPartitioner:
Average time taken by Reduce tasks: 5mins, 18sec

With HRegionPartitioner:
Average time taken by Reduce tasks: 3mins, 11sec = +166.5% improvement

Saves quite a bit time on imports with just  a small set of server and spreads 
out the average load on the regions so no one server is getting hit hard while others
are set idle.

Would still like to see some stats form someone that has a larger server group then me.


> We need a Hbase Partitioner for TableMapReduceUtil.initTableReduceJob MR Jobs
> -----------------------------------------------------------------------------
>
>                 Key: HBASE-987
>                 URL: https://issues.apache.org/jira/browse/HBASE-987
>             Project: Hadoop HBase
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Billy Pearson
>            Assignee: Billy Pearson
>            Priority: Minor
>             Fix For: 0.19.0
>
>         Attachments: 987.patch.txt
>
>
> When we run say 20 reducers they all get ~1/20th of the data to output to the table.
> The problem for us on large import jobs is the data gets sorted by key and the all Reducers 
> pound one region at a time.
> we need to add onto the TableMapReduceUtil.initTableReduceJob method so it can set the partitioner 
> and set the number of reducers = number of regions as the table map does for maps.
> Then the Partitioner will send all the BatchUpdates for one region to one reducer.
> So we get a more even spread of writers to the regions this would assure that only one reducer will send 
> updates to one region keeping any one region from getting more overloaded the others.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.