You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hbase.apache.org by mb...@apache.org on 2014/08/20 10:45:23 UTC
[1/2] git commit: HBASE-11737 Document callQueue improvements from
HBASE-11355 and HBASE-11724 (Misty Stanley-Jones)
Repository: hbase
Updated Branches:
refs/heads/branch-1 cbc1bc7d4 -> 5c1ae840f
refs/heads/master e28ec7246 -> a55a65017
HBASE-11737 Document callQueue improvements from HBASE-11355 and HBASE-11724 (Misty Stanley-Jones)
Project: http://git-wip-us.apache.org/repos/asf/hbase/repo
Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/a55a6501
Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/a55a6501
Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/a55a6501
Branch: refs/heads/master
Commit: a55a65017cc182e3efd4639e3959af09f178d7d1
Parents: e28ec72
Author: Matteo Bertozzi <ma...@cloudera.com>
Authored: Wed Aug 20 09:40:58 2014 +0100
Committer: Matteo Bertozzi <ma...@cloudera.com>
Committed: Wed Aug 20 09:41:46 2014 +0100
----------------------------------------------------------------------
src/main/docbkx/performance.xml | 126 ++++++++++++++++++++++++++++++++++-
1 file changed, 125 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hbase/blob/a55a6501/src/main/docbkx/performance.xml
----------------------------------------------------------------------
diff --git a/src/main/docbkx/performance.xml b/src/main/docbkx/performance.xml
index 57c866a..941ecb7 100644
--- a/src/main/docbkx/performance.xml
+++ b/src/main/docbkx/performance.xml
@@ -254,7 +254,131 @@
release note on <link xlink:href="https://issues.apache.org/jira/browse/HBASE-5074">HBASE-5074 support checksums in HBase block cache</link>.
</para>
</section>
-
+ <section>
+ <title>Tuning <code>callQueue</code> Options</title>
+ <para><link xlink:href="https://issues.apache.org/jira/browse/HBASE-11355">HBASE-11355</link>
+ introduces several callQueue tuning mechanisms which can increase performance. See the JIRA
+ for some benchmarking information.</para>
+ <itemizedlist>
+ <listitem>
+ <para>To increase the number of callqueues, set
+ <option>hbase.ipc.server.num.callqueue</option> to a value greater than
+ <literal>1</literal>.</para>
+ </listitem>
+ <listitem>
+ <para>To split the callqueue into separate read and write queues, set
+ <code>hbase.ipc.server.callqueue.read.ratio</code> to a value between
+ <literal>0</literal> and <literal>1</literal>. This factor weights the queues toward
+ writes (if below .5) or reads (if above .5). Another way to say this is that the factor
+ determines what percentage of the split queues are used for reads. The following
+ examples illustrate some of the possibilities. Note that you always have at least one
+ write queue, no matter what setting you use.</para>
+ <itemizedlist>
+ <listitem>
+ <para>The default value of <literal>0</literal> does not split the queue.</para>
+ </listitem>
+ <listitem>
+ <para>A value of <literal>.3</literal> uses 30% of the queues for reading and 60% for
+ writing. Given a value of <literal>10</literal> for
+ <option>hbase.ipc.server.num.callqueue</option>, 3 queues would be used for reads
+ and 7 for writes.</para>
+ </listitem>
+ <listitem>
+ <para>A value of <literal>.5</literal> uses the same number of read queues and write
+ queues. Given a value of <literal>10</literal> for
+ <option>hbase.ipc.server.num.callqueue</option>, 5 queues would be used for reads
+ and 5 for writes.</para>
+ </listitem>
+ <listitem>
+ <para>A value of <literal>.6</literal> uses 60% of the queues for reading and 30% for
+ reading. Given a value of <literal>10</literal> for
+ <option>hbase.ipc.server.num.callqueue</option>, 7 queues would be used for reads
+ and 3 for writes.</para>
+ </listitem>
+ <listitem>
+ <para>A value of <literal>1.0</literal> uses one queue to process write requests, and
+ all other queues process read requests. A value higher than <literal>1.0</literal>
+ has the same effect as a value of <literal>1.0</literal>. Given a value of
+ <literal>10</literal> for <option>hbase.ipc.server.num.callqueue</option>, 9
+ queues would be used for reads and 1 for writes.</para>
+ </listitem>
+ </itemizedlist>
+ </listitem>
+ <listitem>
+ <para>You can also split the read queues so that separate queues are used for short reads
+ (from Get operations) and long reads (from Scan operations), by setting the
+ <option>hbase.ipc.server.callqueue.scan.ratio</option> option. This option is a factor
+ between 0 and 1, which determine the ratio of read queues used for Gets and Scans. More
+ queues are used for Gets if the value is below <literal>.5</literal> and more are used
+ for scans if the value is above <literal>.5</literal>. No matter what setting you use,
+ at least one read queue is used for Get operations.</para>
+ <itemizedlist>
+ <listitem>
+ <para>A value of <literal>0</literal> does not split the read queue.</para>
+ </listitem>
+ <listitem>
+ <para>A value of <literal>.3</literal> uses 60% of the read queues for Gets and 30%
+ for Scans. Given a value of <literal>20</literal> for
+ <option>hbase.ipc.server.num.callqueue</option> and a value of <literal>.5
+ </literal> for <option>hbase.ipc.server.callqueue.read.ratio</option>, 10 queues
+ would be used for reads, out of those 10, 7 would be used for Gets and 3 for
+ Scans.</para>
+ </listitem>
+ <listitem>
+ <para>A value of <literal>.5</literal> uses half the read queues for Gets and half for
+ Scans. Given a value of <literal>20</literal> for
+ <option>hbase.ipc.server.num.callqueue</option> and a value of <literal>.5
+ </literal> for <option>hbase.ipc.server.callqueue.read.ratio</option>, 10 queues
+ would be used for reads, out of those 10, 5 would be used for Gets and 5 for
+ Scans.</para>
+ </listitem>
+ <listitem>
+ <para>A value of <literal>.6</literal> uses 30% of the read queues for Gets and 60%
+ for Scans. Given a value of <literal>20</literal> for
+ <option>hbase.ipc.server.num.callqueue</option> and a value of <literal>.5
+ </literal> for <option>hbase.ipc.server.callqueue.read.ratio</option>, 10 queues
+ would be used for reads, out of those 10, 3 would be used for Gets and 7 for
+ Scans.</para>
+ </listitem>
+ <listitem>
+ <para>A value of <literal>1.0</literal> uses all but one of the read queues for Scans.
+ Given a value of <literal>20</literal> for
+ <option>hbase.ipc.server.num.callqueue</option> and a value of <literal>.5
+ </literal> for <option>hbase.ipc.server.callqueue.read.ratio</option>, 10 queues
+ would be used for reads, out of those 10, 1 would be used for Gets and 9 for
+ Scans.</para>
+ </listitem>
+ </itemizedlist>
+ </listitem>
+ <listitem>
+ <para>You can use the new option
+ <option>hbase.ipc.server.callqueue.handler.factor</option> to programmatically tune
+ the number of queues:</para>
+ <itemizedlist>
+ <listitem>
+ <para>A value of <literal>0</literal> uses a single shared queue between all the
+ handlers.</para>
+ </listitem>
+ <listitem>
+ <para>A value of <literal>1</literal> uses a separate queue for each handler.</para>
+ </listitem>
+ <listitem>
+ <para>A value between <literal>0</literal> and <literal>1</literal> tunes the number
+ of queues against the number of handlers. For instance, a value of
+ <literal>.5</literal> shares one queue between each two handlers.</para>
+ </listitem>
+ </itemizedlist>
+ <para>Having more queues, such as in a situation where you have one queue per handler,
+ reduces contention when adding a task to a queue or selecting it from a queue. The
+ trade-off is that if you have some queues with long-running tasks, a handler may end up
+ waiting to execute from that queue rather than processing another queue which has
+ waiting tasks.</para>
+ </listitem>
+ </itemizedlist>
+ <para>For these values to take effect on a given Region Server, the Region Server must be
+ restarted. These parameters are intended for testing purposes and should be used
+ carefully.</para>
+ </section>
</section>
[2/2] git commit: HBASE-11737 Document callQueue improvements from
HBASE-11355 and HBASE-11724 (Misty Stanley-Jones)
Posted by mb...@apache.org.
HBASE-11737 Document callQueue improvements from HBASE-11355 and HBASE-11724 (Misty Stanley-Jones)
Project: http://git-wip-us.apache.org/repos/asf/hbase/repo
Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/5c1ae840
Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/5c1ae840
Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/5c1ae840
Branch: refs/heads/branch-1
Commit: 5c1ae840f21f7a3857543e408ef20a63be2b0751
Parents: cbc1bc7
Author: Matteo Bertozzi <ma...@cloudera.com>
Authored: Wed Aug 20 09:40:58 2014 +0100
Committer: Matteo Bertozzi <ma...@cloudera.com>
Committed: Wed Aug 20 09:42:41 2014 +0100
----------------------------------------------------------------------
src/main/docbkx/performance.xml | 126 ++++++++++++++++++++++++++++++++++-
1 file changed, 125 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hbase/blob/5c1ae840/src/main/docbkx/performance.xml
----------------------------------------------------------------------
diff --git a/src/main/docbkx/performance.xml b/src/main/docbkx/performance.xml
index 187d8c2..58557db 100644
--- a/src/main/docbkx/performance.xml
+++ b/src/main/docbkx/performance.xml
@@ -252,7 +252,131 @@
release note on <link xlink:href="https://issues.apache.org/jira/browse/HBASE-5074">HBASE-5074 support checksums in HBase block cache</link>.
</para>
</section>
-
+ <section>
+ <title>Tuning <code>callQueue</code> Options</title>
+ <para><link xlink:href="https://issues.apache.org/jira/browse/HBASE-11355">HBASE-11355</link>
+ introduces several callQueue tuning mechanisms which can increase performance. See the JIRA
+ for some benchmarking information.</para>
+ <itemizedlist>
+ <listitem>
+ <para>To increase the number of callqueues, set
+ <option>hbase.ipc.server.num.callqueue</option> to a value greater than
+ <literal>1</literal>.</para>
+ </listitem>
+ <listitem>
+ <para>To split the callqueue into separate read and write queues, set
+ <code>hbase.ipc.server.callqueue.read.ratio</code> to a value between
+ <literal>0</literal> and <literal>1</literal>. This factor weights the queues toward
+ writes (if below .5) or reads (if above .5). Another way to say this is that the factor
+ determines what percentage of the split queues are used for reads. The following
+ examples illustrate some of the possibilities. Note that you always have at least one
+ write queue, no matter what setting you use.</para>
+ <itemizedlist>
+ <listitem>
+ <para>The default value of <literal>0</literal> does not split the queue.</para>
+ </listitem>
+ <listitem>
+ <para>A value of <literal>.3</literal> uses 30% of the queues for reading and 60% for
+ writing. Given a value of <literal>10</literal> for
+ <option>hbase.ipc.server.num.callqueue</option>, 3 queues would be used for reads
+ and 7 for writes.</para>
+ </listitem>
+ <listitem>
+ <para>A value of <literal>.5</literal> uses the same number of read queues and write
+ queues. Given a value of <literal>10</literal> for
+ <option>hbase.ipc.server.num.callqueue</option>, 5 queues would be used for reads
+ and 5 for writes.</para>
+ </listitem>
+ <listitem>
+ <para>A value of <literal>.6</literal> uses 60% of the queues for reading and 30% for
+ reading. Given a value of <literal>10</literal> for
+ <option>hbase.ipc.server.num.callqueue</option>, 7 queues would be used for reads
+ and 3 for writes.</para>
+ </listitem>
+ <listitem>
+ <para>A value of <literal>1.0</literal> uses one queue to process write requests, and
+ all other queues process read requests. A value higher than <literal>1.0</literal>
+ has the same effect as a value of <literal>1.0</literal>. Given a value of
+ <literal>10</literal> for <option>hbase.ipc.server.num.callqueue</option>, 9
+ queues would be used for reads and 1 for writes.</para>
+ </listitem>
+ </itemizedlist>
+ </listitem>
+ <listitem>
+ <para>You can also split the read queues so that separate queues are used for short reads
+ (from Get operations) and long reads (from Scan operations), by setting the
+ <option>hbase.ipc.server.callqueue.scan.ratio</option> option. This option is a factor
+ between 0 and 1, which determine the ratio of read queues used for Gets and Scans. More
+ queues are used for Gets if the value is below <literal>.5</literal> and more are used
+ for scans if the value is above <literal>.5</literal>. No matter what setting you use,
+ at least one read queue is used for Get operations.</para>
+ <itemizedlist>
+ <listitem>
+ <para>A value of <literal>0</literal> does not split the read queue.</para>
+ </listitem>
+ <listitem>
+ <para>A value of <literal>.3</literal> uses 60% of the read queues for Gets and 30%
+ for Scans. Given a value of <literal>20</literal> for
+ <option>hbase.ipc.server.num.callqueue</option> and a value of <literal>.5
+ </literal> for <option>hbase.ipc.server.callqueue.read.ratio</option>, 10 queues
+ would be used for reads, out of those 10, 7 would be used for Gets and 3 for
+ Scans.</para>
+ </listitem>
+ <listitem>
+ <para>A value of <literal>.5</literal> uses half the read queues for Gets and half for
+ Scans. Given a value of <literal>20</literal> for
+ <option>hbase.ipc.server.num.callqueue</option> and a value of <literal>.5
+ </literal> for <option>hbase.ipc.server.callqueue.read.ratio</option>, 10 queues
+ would be used for reads, out of those 10, 5 would be used for Gets and 5 for
+ Scans.</para>
+ </listitem>
+ <listitem>
+ <para>A value of <literal>.6</literal> uses 30% of the read queues for Gets and 60%
+ for Scans. Given a value of <literal>20</literal> for
+ <option>hbase.ipc.server.num.callqueue</option> and a value of <literal>.5
+ </literal> for <option>hbase.ipc.server.callqueue.read.ratio</option>, 10 queues
+ would be used for reads, out of those 10, 3 would be used for Gets and 7 for
+ Scans.</para>
+ </listitem>
+ <listitem>
+ <para>A value of <literal>1.0</literal> uses all but one of the read queues for Scans.
+ Given a value of <literal>20</literal> for
+ <option>hbase.ipc.server.num.callqueue</option> and a value of <literal>.5
+ </literal> for <option>hbase.ipc.server.callqueue.read.ratio</option>, 10 queues
+ would be used for reads, out of those 10, 1 would be used for Gets and 9 for
+ Scans.</para>
+ </listitem>
+ </itemizedlist>
+ </listitem>
+ <listitem>
+ <para>You can use the new option
+ <option>hbase.ipc.server.callqueue.handler.factor</option> to programmatically tune
+ the number of queues:</para>
+ <itemizedlist>
+ <listitem>
+ <para>A value of <literal>0</literal> uses a single shared queue between all the
+ handlers.</para>
+ </listitem>
+ <listitem>
+ <para>A value of <literal>1</literal> uses a separate queue for each handler.</para>
+ </listitem>
+ <listitem>
+ <para>A value between <literal>0</literal> and <literal>1</literal> tunes the number
+ of queues against the number of handlers. For instance, a value of
+ <literal>.5</literal> shares one queue between each two handlers.</para>
+ </listitem>
+ </itemizedlist>
+ <para>Having more queues, such as in a situation where you have one queue per handler,
+ reduces contention when adding a task to a queue or selecting it from a queue. The
+ trade-off is that if you have some queues with long-running tasks, a handler may end up
+ waiting to execute from that queue rather than processing another queue which has
+ waiting tasks.</para>
+ </listitem>
+ </itemizedlist>
+ <para>For these values to take effect on a given Region Server, the Region Server must be
+ restarted. These parameters are intended for testing purposes and should be used
+ carefully.</para>
+ </section>
</section>