[jira] [Commented] (HBASE-4365) Add a decent heuristic for region size

["Todd Lipcon (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-4365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101668#comment-13101668 ] 

Todd Lipcon commented on HBASE-4365:
------------------------------------

One idea that I'd like to propose is the following. We would introduce a special value for max region size (eg 0 or -1) which configures HBase to use a heuristic instead of a constant. The heuristic would be:
- on region-open (and periodically, perhaps on flush or compaction) the RS scans META to find how many regions are in the table. Let's call this _R_.
- the region server also periodically scans ZK or asks the master to determine how many RS are in the cluster. Let's call this _S_.
- If _R_ < _S_ (ie there are fewer regions than servers), we use a max region size of perhaps 256MB. This is so that early in a table's lifetime, even if not pre-split, the table will split rapidly and be able to spread load across the cluster.
- If _R_ < _5S_, we use a max region size of perhaps 1GB.
- If _R_ > _5S_, we use a max region size of something rather large, like 10GB.

As an example, if we started putting a 1T table on a cluster with 20 nodes, we'd quickly expand to 20 regions of 256M. The size would switch to 1G, and we'd get another ~100 regions. The size would switch to 10G and we'd get another 100 regions. So at the end, we'd have around 200-250 regions, or about 10 per server.

The specific numbers above aren't well thought out, but does this seem to be more "foolproof" than any predetermined default.

> Add a decent heuristic for region size
> --------------------------------------
>
>                 Key: HBASE-4365
>                 URL: https://issues.apache.org/jira/browse/HBASE-4365
>             Project: HBase
>          Issue Type: Improvement
>    Affects Versions: 0.94.0
>            Reporter: Todd Lipcon
>
> A few of us were brainstorming this morning about what the default region size should be. There were a few general points made:
> - in some ways it's better to be too-large than too-small, since you can always split a table further, but you can't merge regions currently
> - with HFile v2 and multithreaded compactions there are fewer reasons to avoid very-large regions (10GB+)
> - for small tables you may want a small region size just so you can distribute load better across a cluster
> - for big tables, multi-GB is probably best

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira