You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Reid Chan (JIRA)" <ji...@apache.org> on 2017/11/13 05:28:00 UTC
[jira] [Comment Edited] (HBASE-19226) Limit the reduce tasks number
of incremental load
[ https://issues.apache.org/jira/browse/HBASE-19226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16249114#comment-16249114 ]
Reid Chan edited comment on HBASE-19226 at 11/13/17 5:27 AM:
-------------------------------------------------------------
Descriptions from HBASE-17130 shows that sometimes 1 reducer would bring benefits, number is much more fine-grained than ratio, though to use which depends on case.
bq. The number of regions in the same table also changes, and the reduce tasks number also needs to be dynamically adjusted.
I got your mind, what about if the config(string) is between (0, 1), treats it as ratio; else [1, +), treats it as a specified number. Judge it with regular expression. WDYT. [~Yun Zhao]
was (Author: reidchan):
Descriptions from HBASE-17130 shows that sometimes 1 reducer would bring benefits, number is much more fine-grained than ratio, though to use which depends on case.
bq. The number of regions in the same table also changes, and the reduce tasks number also needs to be dynamically adjusted.
I got your mind, what about if the config(string) is between (0, 1), treats it as ratio; else [1, +), treats it as a specified number. Judge it with regular expression. WDYT.
> Limit the reduce tasks number of incremental load
> -------------------------------------------------
>
> Key: HBASE-19226
> URL: https://issues.apache.org/jira/browse/HBASE-19226
> Project: HBase
> Issue Type: Improvement
> Reporter: Yun Zhao
> Assignee: Yun Zhao
> Priority: Minor
> Attachments: HBASE-19226.master.001.patch, HBASE-19226.master.002.patch
>
>
> When using MapReduce job to perform an incremental load into a table,the number of reduce tasks is the current number of regions. If there are too many regions, will lead to network+disk I/O is too large, affecting the real-time request.
> Need to use a configuration to set a number or ratio?
> Limit running reduce tasks since [https://issues.apache.org/jira/browse/MAPREDUCE-5583], the old version can only be set reduce number.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)