You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hbase.apache.org by mi...@apache.org on 2014/11/06 23:47:06 UTC
hbase git commit: HBASE-12409 Add actual tunable parameters to
regions per RS calculations
Repository: hbase
Updated Branches:
refs/heads/master 0c2314b07 -> bbd681541
HBASE-12409 Add actual tunable parameters to regions per RS calculations
Project: http://git-wip-us.apache.org/repos/asf/hbase/repo
Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/bbd68154
Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/bbd68154
Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/bbd68154
Branch: refs/heads/master
Commit: bbd681541428d3983a9c4e114b1ee479ab22f020
Parents: 0c2314b
Author: Misty Stanley-Jones <ms...@cloudera.com>
Authored: Mon Nov 3 14:28:31 2014 +1000
Committer: Misty Stanley-Jones <ms...@cloudera.com>
Committed: Fri Nov 7 08:46:57 2014 +1000
----------------------------------------------------------------------
src/main/docbkx/ops_mgt.xml | 41 ++++++++++++++++++++++++----------------
1 file changed, 25 insertions(+), 16 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hbase/blob/bbd68154/src/main/docbkx/ops_mgt.xml
----------------------------------------------------------------------
diff --git a/src/main/docbkx/ops_mgt.xml b/src/main/docbkx/ops_mgt.xml
index 81be6c7..af598ab 100644
--- a/src/main/docbkx/ops_mgt.xml
+++ b/src/main/docbkx/ops_mgt.xml
@@ -2279,22 +2279,31 @@ hbase> restore_snapshot 'myTableSnapshot-122112'
xml:id="ops.capacity.regions.count">
<title>Number of regions per RS - upper bound</title>
<para>In production scenarios, where you have a lot of data, you are normally concerned with
- the maximum number of regions you can have per server. <xref
- linkend="too_many_regions" /> has technical discussion on the subject; in short, maximum
- number of regions is mostly determined by memstore memory usage. Each region has its own
- memstores; these grow up to a configurable size; usually in 128-256Mb range, see <xref
- linkend="hbase.hregion.memstore.flush.size" />. There's one memstore per column family
- (so there's only one per region if there's one CF in the table). RS dedicates some
- fraction of total memory (see <xref
- linkend="hbase.regionserver.global.memstore.size" />) to region memstores. If this
- memory is exceeded (too much memstore usage), undesirable consequences such as
- unresponsive server, or later compaction storms, can result. Thus, a good starting point
- for the number of regions per RS (assuming one table) is:</para>
-
- <programlisting>(RS memory)*(total memstore fraction)/((memstore size)*(# column families))</programlisting>
- <para> E.g. if RS has 16Gb RAM, with default settings, it is 16384*0.4/128 ~ 51 regions per
- RS is a starting point. The formula can be extended to multiple tables; if they all have
- the same configuration, just use total number of families.</para>
+ the maximum number of regions you can have per server. <xref linkend="too_many_regions"/>
+ has technical discussion on the subject. Basically, the maximum number of regions is
+ mostly determined by memstore memory usage. Each region has its own memstores; these grow
+ up to a configurable size; usually in 128-256 MB range, see <xref
+ linkend="hbase.hregion.memstore.flush.size"/>. One memstore exists per column family (so
+ there's only one per region if there's one CF in the table). The RS dedicates some
+ fraction of total memory to its memstores (see <xref
+ linkend="hbase.regionserver.global.memstore.size"/>). If this memory is exceeded (too
+ much memstore usage), it can cause undesirable consequences such as unresponsive server or
+ compaction storms. A good starting point for the number of regions per RS (assuming one
+ table) is:</para>
+
+ <programlisting>((RS memory) * (total memstore fraction)) / ((memstore size)*(# column families))</programlisting>
+ <para>This formula is pseudo-code. Here are two formulas using the actual tunable
+ parameters, first for HBase 0.98+ and second for HBase 0.94.x.</para>
+ <itemizedlist>
+ <listitem><para>HBase 0.98.x:<code>((RS Xmx) * hbase.regionserver.global.memstore.size) /
+ (hbase.hregion.memstore.flush.size * (# column families))</code></para></listitem>
+ <listitem><para>HBase 0.94.x:<code>((RS Xmx) * hbase.regionserver.global.memstore.upperLimit) /
+ (hbase.hregion.memstore.flush.size * (# column families))</code></para></listitem>
+ </itemizedlist>
+ <para>If a given RegionServer has 16 GB of RAM, with default settings, the formula works out
+ to 16384*0.4/128 ~ 51 regions per RS is a starting point. The formula can be extended to
+ multiple tables; if they all have the same configuration, just use the total number of
+ families.</para>
<para>This number can be adjusted; the formula above assumes all your regions are filled at
approximately the same rate. If only a fraction of your regions are going to be actively
written to, you can divide the result by that fraction to get a larger region count. Then,