You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hbase.apache.org by en...@apache.org on 2013/03/30 01:19:57 UTC
svn commit: r1462679 [6/14] - in /hbase/hbase.apache.org/trunk: ./ book/
case_studies/ community/ configuration/ developer/ getting_started/
ops_mgt/ performance/ rpc/
Modified: hbase/hbase.apache.org/trunk/book/hbase.accesscontrol.configuration.html
URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/hbase.accesscontrol.configuration.html?rev=1462679&r1=1462678&r2=1462679&view=diff
==============================================================================
--- hbase/hbase.apache.org/trunk/book/hbase.accesscontrol.configuration.html (original)
+++ hbase/hbase.apache.org/trunk/book/hbase.accesscontrol.configuration.html Sat Mar 30 00:19:55 2013
@@ -8,7 +8,7 @@
This describes how to set up Secure HBase for access control, with an
example of granting and revoking user permission on table resources
provided.
- </p><div class="section" title="8.2.1. Prerequisites"><div class="titlepage"><div><div><h3 class="title"><a name="d2274e4367"></a>8.2.1. Prerequisites</h3></div></div></div><p>
+ </p><div class="section" title="8.2.1. Prerequisites"><div class="titlepage"><div><div><h3 class="title"><a name="d2279e4367"></a>8.2.1. Prerequisites</h3></div></div></div><p>
You must configure HBase for secure operation. Refer to the section
"Secure Client Access to HBase" and complete all of the steps described
there.
@@ -19,7 +19,7 @@
possible to subvert HBase access control via direct client access to
ZooKeeper. Refer to the section on secure ZooKeeper configuration and
complete all of the steps described there.
- </p></div><div class="section" title="8.2.2. Overview"><div class="titlepage"><div><div><h3 class="title"><a name="d2274e4374"></a>8.2.2. Overview</h3></div></div></div><p>
+ </p></div><div class="section" title="8.2.2. Overview"><div class="titlepage"><div><div><h3 class="title"><a name="d2279e4374"></a>8.2.2. Overview</h3></div></div></div><p>
With Secure RPC and Access Control enabled, client access to HBase is
authenticated and user data is private unless access has been
explicitly granted. Access to data can be granted at a table or per
@@ -29,7 +29,7 @@
implementation for simplicity:
</p><div class="orderedlist"><ol class="orderedlist" type="1"><li class="listitem"><p>Row-level or per value (cell): This would require broader changes for storing the ACLs inline with rows. It is a future goal.</p></li><li class="listitem"><p>Push down of file ownership to HDFS: HBase is not designed for the case where files may have different permissions than the HBase system principal. Pushing file ownership down into HDFS would necessitate changes to core code. Also, while HDFS file ownership would make applying quotas easy, and possibly make bulk imports more straightforward, it is not clear that it would offer a more secure setup.</p></li><li class="listitem"><p>HBase managed "roles" as collections of permissions: We will not model "roles" internally in HBase to begin with. We instead allow group names to be granted permissions, which allows external modeling of roles via group membership. Groups are created and manipulated externally to HBase, via the Hadoop group
mapping service.</p></li></ol></div><p>
Access control mechanisms are mature and fairly standardized in the relational database world. The HBase implementation approximates current convention, but HBase has a simpler feature set than relational databases, especially in terms of client operations. We don't distinguish between an insert (new record) and update (of existing record), for example, as both collapse down into a Put. Accordingly, the important operations condense to four permissions: READ, WRITE, CREATE, and ADMIN.
- </p><table id="d2274e4393">Operation To Permission Mapping<thead>PermissionOperation</thead><tbody>ReadGetExistsScanWritePutDeleteLock/UnlockRowIncrementColumnValueCheckAndDelete/PutFlushCompactCreateCreateAlterDropAdminEnable/DisableSplitMajor CompactGrantRevokeShutdown</tbody></table><p>
+ </p><table id="d2279e4393">Operation To Permission Mapping<thead>PermissionOperation</thead><tbody>ReadGetExistsScanWritePutDeleteLock/UnlockRowIncrementColumnValueCheckAndDelete/PutFlushCompactCreateCreateAlterDropAdminEnable/DisableSplitMajor CompactGrantRevokeShutdown</tbody></table><p>
Permissions can be granted in any of the following scopes, though
CREATE and ADMIN permissions are effective only at table scope.
</p><p>
@@ -57,7 +57,7 @@ Access control mechanisms are mature and
alter operation by setting or changing the OWNER table attribute. Only a
single user principal can own a table at a given time. A table owner will
have all permissions over a given table.
- </p></div><div class="section" title="8.2.3. Server-side Configuration for Access Control"><div class="titlepage"><div><div><h3 class="title"><a name="d2274e4531"></a>8.2.3. Server-side Configuration for Access Control</h3></div></div></div><p>
+ </p></div><div class="section" title="8.2.3. Server-side Configuration for Access Control"><div class="titlepage"><div><div><h3 class="title"><a name="d2279e4531"></a>8.2.3. Server-side Configuration for Access Control</h3></div></div></div><p>
Enable the AccessController coprocessor in the cluster configuration
and restart HBase. The restart can be a rolling one. Complete the
restart of all Master and RegionServer processes before setting up
@@ -74,7 +74,7 @@ Access control mechanisms are mature and
<value>org.apache.hadoop.hbase.security.token.TokenProvider,
org.apache.hadoop.hbase.security.access.AccessController</value>
</property>
- </pre></div><div class="section" title="8.2.4. Shell Enhancements for Access Control"><div class="titlepage"><div><div><h3 class="title"><a name="d2274e4543"></a>8.2.4. Shell Enhancements for Access Control</h3></div></div></div><p>
+ </pre></div><div class="section" title="8.2.4. Shell Enhancements for Access Control"><div class="titlepage"><div><div><h3 class="title"><a name="d2279e4543"></a>8.2.4. Shell Enhancements for Access Control</h3></div></div></div><p>
The HBase shell has been extended to provide simple commands for editing and updating user permissions. The following commands have been added for access control list management:
</p>
Grant
Modified: hbase/hbase.apache.org/trunk/book/hbase.org.html
URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/hbase.org.html?rev=1462679&r1=1462678&r2=1462679&view=diff
==============================================================================
--- hbase/hbase.apache.org/trunk/book/hbase.org.html (original)
+++ hbase/hbase.apache.org/trunk/book/hbase.org.html Sat Mar 30 00:19:55 2013
@@ -12,7 +12,7 @@
# Getting the javadoc into site is a little tricky. You have to build it independent, then
# 'aggregate' it at top-level so the pre-site site lifecycle step can find it; that is
# what the javadoc:javadoc and javadoc:aggregate is about.
- $ MAVEN_OPTS=" -Xmx3g" mvn clean -DskipTests javadoc:javadoc javadoc:aggregate site site:stage -DstagingDirectory=/Users/stack/checkouts/hbase.apache.org/trunk
+ $ MAVEN_OPTS=" -Xmx3g" mvn clean -DskipTests javadoc:aggregate site site:stage -DstagingDirectory=/Users/stack/checkouts/hbase.apache.org/trunk
# Check the deployed site by viewing in a brower.
# If all is good, commit it and it will show up at http://hbase.apache.org
#
Modified: hbase/hbase.apache.org/trunk/book/hbase.rpc.html
URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/hbase.rpc.html?rev=1462679&r1=1462678&r2=1462679&view=diff
==============================================================================
--- hbase/hbase.apache.org/trunk/book/hbase.rpc.html (original)
+++ hbase/hbase.apache.org/trunk/book/hbase.rpc.html Sat Mar 30 00:19:55 2013
@@ -1,13 +1,13 @@
<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
- <title>Appendix J. 0.95 RPC Specification</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="prev" href="tracing.client.modifications.html" title="I.2. Client Modifications"><link rel="next" href="apjs02.html" title="J.2. TODO"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">Appendix J. 0.95 RPC Specification</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="tracing.client.modifications.html">Prev</a> </td><th width="60%" align="center"> </th><td width="20%" align="right"> <a accesskey="n" href="apjs02.htm
l">Next</a></td></tr></table><hr></div><div class="appendix" title="Appendix J. 0.95 RPC Specification"><div class="titlepage"><div><div><h2 class="title"><a name="hbase.rpc"></a>Appendix J. 0.95 RPC Specification</h2></div></div></div><div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="section"><a href="hbase.rpc.html#d2274e12696">J.1. Goals</a></span></dt><dt><span class="section"><a href="apjs02.html">J.2. TODO</a></span></dt><dt><span class="section"><a href="apjs03.html">J.3. RPC</a></span></dt><dd><dl><dt><span class="section"><a href="apjs03.html#d2274e12739">J.3.1. Connection Setup</a></span></dt><dt><span class="section"><a href="apjs03.html#d2274e12771">J.3.2. Request</a></span></dt><dt><span class="section"><a href="apjs03.html#d2274e12804">J.3.3. Response</a></span></dt><dt><span class="section"><a href="apjs03.html#d2274e12834">J.3.4. Exceptions</a></span></dt><dt><span class="section"><a href="apjs03.html#d2274e12841">J.3.5.
CellBlocks</a></span></dt></dl></dd><dt><span class="section"><a href="apjs04.html">J.4. Notes</a></span></dt><dd><dl><dt><span class="section"><a href="apjs04.html#d2274e12849">J.4.1. Constraints</a></span></dt><dt><span class="section"><a href="apjs04.html#d2274e12854">J.4.2. One fat pb request or header+param</a></span></dt><dt><span class="section"><a href="apjs04.html#d2274e12874">J.4.3. Compression</a></span></dt></dl></dd></dl></div><p>In 0.95, all client/server communication is done with
+ <title>Appendix J. 0.95 RPC Specification</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="prev" href="tracing.client.modifications.html" title="I.2. Client Modifications"><link rel="next" href="apjs02.html" title="J.2. TODO"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">Appendix J. 0.95 RPC Specification</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="tracing.client.modifications.html">Prev</a> </td><th width="60%" align="center"> </th><td width="20%" align="right"> <a accesskey="n" href="apjs02.htm
l">Next</a></td></tr></table><hr></div><div class="appendix" title="Appendix J. 0.95 RPC Specification"><div class="titlepage"><div><div><h2 class="title"><a name="hbase.rpc"></a>Appendix J. 0.95 RPC Specification</h2></div></div></div><div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="section"><a href="hbase.rpc.html#d2279e12710">J.1. Goals</a></span></dt><dt><span class="section"><a href="apjs02.html">J.2. TODO</a></span></dt><dt><span class="section"><a href="apjs03.html">J.3. RPC</a></span></dt><dd><dl><dt><span class="section"><a href="apjs03.html#d2279e12753">J.3.1. Connection Setup</a></span></dt><dt><span class="section"><a href="apjs03.html#d2279e12785">J.3.2. Request</a></span></dt><dt><span class="section"><a href="apjs03.html#d2279e12818">J.3.3. Response</a></span></dt><dt><span class="section"><a href="apjs03.html#d2279e12848">J.3.4. Exceptions</a></span></dt><dt><span class="section"><a href="apjs03.html#d2279e12855">J.3.5.
CellBlocks</a></span></dt></dl></dd><dt><span class="section"><a href="apjs04.html">J.4. Notes</a></span></dt><dd><dl><dt><span class="section"><a href="apjs04.html#d2279e12863">J.4.1. Constraints</a></span></dt><dt><span class="section"><a href="apjs04.html#d2279e12868">J.4.2. One fat pb request or header+param</a></span></dt><dt><span class="section"><a href="apjs04.html#d2279e12888">J.4.3. Compression</a></span></dt></dl></dd></dl></div><p>In 0.95, all client/server communication is done with
<a class="link" href="https://code.google.com/p/protobuf/" target="_top">protobuf’ed</a> Messages rather than with
<a class="link" href="http://hadoop.apache.org/docs/current/api/org/apache/hadoop/io/Writable.html" target="_top">Hadoop Writables</a>.
Our RPC wire format therefore changes.
This document describes the client/server request/response protocol and our new RPC wire-format.</p><p></p><p>For what RPC is like in 0.94 and previous,
see Benoît/Tsuna’s <a class="link" href="https://github.com/OpenTSDB/asynchbase/blob/master/src/HBaseRpc.java#L164" target="_top">Unofficial Hadoop / HBase RPC protocol documentation</a>.
For more background on how we arrived at this spec., see
- <a class="link" href="https://docs.google.com/document/d/1WCKwgaLDqBw2vpux0jPsAu2WPTRISob7HGCO8YhfDTA/edit#" target="_top">HBase RPC: WIP</a></p><p></p><div class="section" title="J.1. Goals"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="d2274e12696"></a>J.1. Goals</h2></div></div></div><p>
+ <a class="link" href="https://docs.google.com/document/d/1WCKwgaLDqBw2vpux0jPsAu2WPTRISob7HGCO8YhfDTA/edit#" target="_top">HBase RPC: WIP</a></p><p></p><div class="section" title="J.1. Goals"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="d2279e12710"></a>J.1. Goals</h2></div></div></div><p>
</p><div class="orderedlist"><ol class="orderedlist" type="1"><li class="listitem"><p>A wire-format we can evolve</p></li><li class="listitem"><p>A format that does not require our rewriting server core or
radically changing its current architecture (for later).</p></li></ol></div><p>
</p></div></div><div id="disqus_thread"></div><script type="text/javascript">
Modified: hbase/hbase.apache.org/trunk/book/hbase.tests.html
URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/hbase.tests.html?rev=1462679&r1=1462678&r2=1462679&view=diff
==============================================================================
--- hbase/hbase.apache.org/trunk/book/hbase.tests.html (original)
+++ hbase/hbase.apache.org/trunk/book/hbase.tests.html Sat Mar 30 00:19:55 2013
@@ -46,18 +46,18 @@ Apache HBase uses a patched maven surefi
its unit test characterizations.
</p><p>Read the below to figure which annotation of the set small, medium, and large to
put on your new HBase unit test.
-</p><div class="section" title="15.7.2.1. Small Tests"><div class="titlepage"><div><div><h4 class="title"><a name="hbase.unittests.small"></a>15.7.2.1. Small Tests<a class="indexterm" name="d2274e10131"></a></h4></div></div></div><p>
+</p><div class="section" title="15.7.2.1. Small Tests"><div class="titlepage"><div><div><h4 class="title"><a name="hbase.unittests.small"></a>15.7.2.1. Small Tests<a class="indexterm" name="d2279e10145"></a></h4></div></div></div><p>
<span class="emphasis"><em>Small</em></span> tests are executed in a shared JVM. We put in this category all the tests that can
be executed quickly in a shared JVM. The maximum execution time for a small test is 15 seconds,
-and small tests should not use a (mini)cluster.</p></div><div class="section" title="15.7.2.2. Medium Tests"><div class="titlepage"><div><div><h4 class="title"><a name="hbase.unittests.medium"></a>15.7.2.2. Medium Tests<a class="indexterm" name="d2274e10142"></a></h4></div></div></div><p><span class="emphasis"><em>Medium</em></span> tests represent tests that must be executed
+and small tests should not use a (mini)cluster.</p></div><div class="section" title="15.7.2.2. Medium Tests"><div class="titlepage"><div><div><h4 class="title"><a name="hbase.unittests.medium"></a>15.7.2.2. Medium Tests<a class="indexterm" name="d2279e10156"></a></h4></div></div></div><p><span class="emphasis"><em>Medium</em></span> tests represent tests that must be executed
before proposing a patch. They are designed to run in less than 30 minutes altogether,
and are quite stable in their results. They are designed to last less than 50 seconds
individually. They can use a cluster, and each of them is executed in a separate JVM.
-</p></div><div class="section" title="15.7.2.3. Large Tests"><div class="titlepage"><div><div><h4 class="title"><a name="hbase.unittests.large"></a>15.7.2.3. Large Tests<a class="indexterm" name="d2274e10152"></a></h4></div></div></div><p><span class="emphasis"><em>Large</em></span> tests are everything else. They are typically large-scale
+</p></div><div class="section" title="15.7.2.3. Large Tests"><div class="titlepage"><div><div><h4 class="title"><a name="hbase.unittests.large"></a>15.7.2.3. Large Tests<a class="indexterm" name="d2279e10166"></a></h4></div></div></div><p><span class="emphasis"><em>Large</em></span> tests are everything else. They are typically large-scale
tests, regression tests for specific bugs, timeout tests, performance tests.
They are executed before a commit on the pre-integration machines. They can be run on
the developer machine as well.
-</p></div><div class="section" title="15.7.2.4. Integration Tests"><div class="titlepage"><div><div><h4 class="title"><a name="hbase.unittests.integration"></a>15.7.2.4. Integration Tests<a class="indexterm" name="d2274e10162"></a></h4></div></div></div><p><span class="emphasis"><em>Integration</em></span> tests are system level tests. See
+</p></div><div class="section" title="15.7.2.4. Integration Tests"><div class="titlepage"><div><div><h4 class="title"><a name="hbase.unittests.integration"></a>15.7.2.4. Integration Tests<a class="indexterm" name="d2279e10176"></a></h4></div></div></div><p><span class="emphasis"><em>Integration</em></span> tests are system level tests. See
<a class="xref" href="hbase.tests.html#integration.tests" title="15.7.5. Integration Tests">Section 15.7.5, “Integration Tests”</a> for more info.
</p></div></div><div class="section" title="15.7.3. Running tests"><div class="titlepage"><div><div><h3 class="title"><a name="hbase.unittests.cmds"></a>15.7.3. Running tests</h3></div></div></div><p>Below we describe how to run the Apache HBase junit categories.</p><div class="section" title="15.7.3.1. Default: small and medium category tests"><div class="titlepage"><div><div><h4 class="title"><a name="hbase.unittests.cmds.test"></a>15.7.3.1. Default: small and medium category tests
</h4></div></div></div><p>Running </p><pre class="programlisting">mvn test</pre><p> will execute all small tests in a single JVM
@@ -112,7 +112,7 @@ It must be executed from the directory w
Running </p><pre class="programlisting">./dev-support/hbasetests.sh runAllTests</pre><p> will execute all tests.
Running </p><pre class="programlisting">./dev-support/hbasetests.sh replayFailed</pre><p> will rerun the failed tests a
second time, in a separate jvm and without parallelisation.
-</p></div><div class="section" title="15.7.3.7. Test Resource Checker"><div class="titlepage"><div><div><h4 class="title"><a name="hbase.unittests.resource.checker"></a>15.7.3.7. Test Resource Checker<a class="indexterm" name="d2274e10290"></a></h4></div></div></div><p>
+</p></div><div class="section" title="15.7.3.7. Test Resource Checker"><div class="titlepage"><div><div><h4 class="title"><a name="hbase.unittests.resource.checker"></a>15.7.3.7. Test Resource Checker<a class="indexterm" name="d2279e10304"></a></h4></div></div></div><p>
A custom Maven SureFire plugin listener checks a number of resources before
and after each HBase unit test runs and logs its findings at the end of the test
output files which can be found in <code class="filename">target/surefire-reports</code>
Modified: hbase/hbase.apache.org/trunk/book/hbck.in.depth.html
URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/hbck.in.depth.html?rev=1462679&r1=1462678&r2=1462679&view=diff
==============================================================================
--- hbase/hbase.apache.org/trunk/book/hbck.in.depth.html (original)
+++ hbase/hbase.apache.org/trunk/book/hbck.in.depth.html Sat Mar 30 00:19:55 2013
@@ -1,9 +1,9 @@
<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
- <title>Appendix B. hbck In Depth</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="prev" href="faq.html" title="Appendix A. FAQ"><link rel="next" href="apbs02.html" title="B.2. Inconsistencies"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">Appendix B. hbck In Depth</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="faq.html">Prev</a> </td><th width="60%" align="center"> </th><td width="20%" align="right"> <a accesskey="n" href="apbs02.html">Next</a></td></tr></table><hr></div><div class="appendix" t
itle="Appendix B. hbck In Depth"><div class="titlepage"><div><div><h2 class="title"><a name="hbck.in.depth"></a>Appendix B. hbck In Depth</h2></div></div></div><div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="section"><a href="hbck.in.depth.html#d2274e11695">B.1. Running hbck to identify inconsistencies</a></span></dt><dt><span class="section"><a href="apbs02.html">B.2. Inconsistencies</a></span></dt><dt><span class="section"><a href="apbs03.html">B.3. Localized repairs</a></span></dt><dt><span class="section"><a href="apbs04.html">B.4. Region Overlap Repairs</a></span></dt><dd><dl><dt><span class="section"><a href="apbs04.html#d2274e11807">B.4.1. Special cases: Meta is not properly assigned</a></span></dt><dt><span class="section"><a href="apbs04.html#d2274e11816">B.4.2. Special cases: HBase version file is missing</a></span></dt><dt><span class="section"><a href="apbs04.html#d2274e11823">B.4.3. Special case: Root and META are corrupt
.</a></span></dt><dt><span class="section"><a href="apbs04.html#d2274e11830">B.4.4. Special cases: Offline split parent</a></span></dt></dl></dd></dl></div><p>HBaseFsck (hbck) is a tool for checking for region consistency and table integrity problems
+ <title>Appendix B. hbck In Depth</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="prev" href="faq.html" title="Appendix A. FAQ"><link rel="next" href="apbs02.html" title="B.2. Inconsistencies"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">Appendix B. hbck In Depth</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="faq.html">Prev</a> </td><th width="60%" align="center"> </th><td width="20%" align="right"> <a accesskey="n" href="apbs02.html">Next</a></td></tr></table><hr></div><div class="appendix" t
itle="Appendix B. hbck In Depth"><div class="titlepage"><div><div><h2 class="title"><a name="hbck.in.depth"></a>Appendix B. hbck In Depth</h2></div></div></div><div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="section"><a href="hbck.in.depth.html#d2279e11709">B.1. Running hbck to identify inconsistencies</a></span></dt><dt><span class="section"><a href="apbs02.html">B.2. Inconsistencies</a></span></dt><dt><span class="section"><a href="apbs03.html">B.3. Localized repairs</a></span></dt><dt><span class="section"><a href="apbs04.html">B.4. Region Overlap Repairs</a></span></dt><dd><dl><dt><span class="section"><a href="apbs04.html#d2279e11821">B.4.1. Special cases: Meta is not properly assigned</a></span></dt><dt><span class="section"><a href="apbs04.html#d2279e11830">B.4.2. Special cases: HBase version file is missing</a></span></dt><dt><span class="section"><a href="apbs04.html#d2279e11837">B.4.3. Special case: Root and META are corrupt
.</a></span></dt><dt><span class="section"><a href="apbs04.html#d2279e11844">B.4.4. Special cases: Offline split parent</a></span></dt></dl></dd></dl></div><p>HBaseFsck (hbck) is a tool for checking for region consistency and table integrity problems
and repairing a corrupted HBase. It works in two basic modes -- a read-only inconsistency
identifying mode and a multi-phase read-write repair mode.
- </p><div class="section" title="B.1. Running hbck to identify inconsistencies"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="d2274e11695"></a>B.1. Running hbck to identify inconsistencies</h2></div></div></div>
+ </p><div class="section" title="B.1. Running hbck to identify inconsistencies"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="d2279e11709"></a>B.1. Running hbck to identify inconsistencies</h2></div></div></div>
To check to see if your HBase cluster has corruptions, run hbck against your HBase cluster:
<pre class="programlisting">
$ ./bin/hbase hbck
Modified: hbase/hbase.apache.org/trunk/book/hfilev2.html
URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/hfilev2.html?rev=1462679&r1=1462678&r2=1462679&view=diff
==============================================================================
--- hbase/hbase.apache.org/trunk/book/hfilev2.html (original)
+++ hbase/hbase.apache.org/trunk/book/hfilev2.html Sat Mar 30 00:19:55 2013
@@ -1,12 +1,12 @@
<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
- <title>Appendix E. HFile format version 2</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="prev" href="apd.html" title="Appendix D. YCSB: The Yahoo! Cloud Serving Benchmark and HBase"><link rel="next" href="apes02.html" title="E.2. HFile format version 1 overview"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">Appendix E. HFile format version 2</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="apd.html">Prev</a> </td><th width="60%" align="center"> </th><td width="20%" align="right"> <a accesskey="
n" href="apes02.html">Next</a></td></tr></table><hr></div><div class="appendix" title="Appendix E. HFile format version 2"><div class="titlepage"><div><div><h2 class="title"><a name="hfilev2"></a>Appendix E. HFile format version 2</h2></div></div></div><div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="section"><a href="hfilev2.html#d2274e11991">E.1. Motivation </a></span></dt><dt><span class="section"><a href="apes02.html">E.2. HFile format version 1 overview </a></span></dt><dd><dl><dt><span class="section"><a href="apes02.html#d2274e12026">E.2.1. Block index format in version 1 </a></span></dt></dl></dd><dt><span class="section"><a href="apes03.html">E.3.
+ <title>Appendix E. HFile format version 2</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="prev" href="apd.html" title="Appendix D. YCSB: The Yahoo! Cloud Serving Benchmark and HBase"><link rel="next" href="apes02.html" title="E.2. HFile format version 1 overview"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">Appendix E. HFile format version 2</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="apd.html">Prev</a> </td><th width="60%" align="center"> </th><td width="20%" align="right"> <a accesskey="
n" href="apes02.html">Next</a></td></tr></table><hr></div><div class="appendix" title="Appendix E. HFile format version 2"><div class="titlepage"><div><div><h2 class="title"><a name="hfilev2"></a>Appendix E. HFile format version 2</h2></div></div></div><div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="section"><a href="hfilev2.html#d2279e12005">E.1. Motivation </a></span></dt><dt><span class="section"><a href="apes02.html">E.2. HFile format version 1 overview </a></span></dt><dd><dl><dt><span class="section"><a href="apes02.html#d2279e12040">E.2.1. Block index format in version 1 </a></span></dt></dl></dd><dt><span class="section"><a href="apes03.html">E.3.
HBase file format with inline blocks (version 2)
- </a></span></dt><dd><dl><dt><span class="section"><a href="apes03.html#d2274e12053">E.3.1. Overview</a></span></dt><dt><span class="section"><a href="apes03.html#d2274e12068">E.3.2. Unified version 2 block format</a></span></dt><dt><span class="section"><a href="apes03.html#d2274e12137">E.3.3. Block index in version 2</a></span></dt><dt><span class="section"><a href="apes03.html#d2274e12162">E.3.4.
- Root block index format in version 2</a></span></dt><dt><span class="section"><a href="apes03.html#d2274e12215">E.3.5.
- Non-root block index format in version 2</a></span></dt><dt><span class="section"><a href="apes03.html#d2274e12240">E.3.6.
- Bloom filters in version 2</a></span></dt><dt><span class="section"><a href="apes03.html#d2274e12277">E.3.7. File Info format in versions 1 and 2</a></span></dt><dt><span class="section"><a href="apes03.html#d2274e12323">E.3.8.
- Fixed file trailer format differences between versions 1 and 2</a></span></dt></dl></dd></dl></div><div class="section" title="E.1. Motivation"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="d2274e11991"></a>E.1. Motivation </h2></div></div></div><p>Note: this feature was introduced in HBase 0.92</p><p>We found it necessary to revise the HFile format after encountering high memory usage and slow startup times caused by large Bloom filters and block indexes in the region server. Bloom filters can get as large as 100 MB per HFile, which adds up to 2 GB when aggregated over 20 regions. Block indexes can grow as large as 6 GB in aggregate size over the same set of regions. A region is not considered opened until all of its block index data is loaded. Large Bloom filters produce a different performance problem: the first get request that requires a Bloom filter lookup will incur the latency of loading the entire Bloom filter bit a
rray.</p><p>To speed up region server startup we break Bloom filters and block indexes into multiple blocks and write those blocks out as they fill up, which also reduces the HFile writer’s memory footprint. In the Bloom filter case, “filling up a block” means accumulating enough keys to efficiently utilize a fixed-size bit array, and in the block index case we accumulate an “index block” of the desired size. Bloom filter blocks and index blocks (we call these “inline blocks”) become interspersed with data blocks, and as a side effect we can no longer rely on the difference between block offsets to determine data block length, as it was done in version 1.</p><p>HFile is a low-level file format by design, and it should not deal with application-specific details such as Bloom filters, which are handled at StoreFile level. Therefore, we call Bloom filter blocks in an HFile "inline" blocks. We also supply HFile with an interface to write
those inline blocks. </p><p>Another format modification aimed at reducing the region server startup time is to use a contiguous “load-on-open” section that has to be loaded in memory at the time an HFile is being opened. Currently, as an HFile opens, there are separate seek operations to read the trailer, data/meta indexes, and file info. To read the Bloom filter, there are two more seek operations for its “data” and “meta” portions. In version 2, we seek once to read the trailer and seek again to read everything else we need to open the file from a contiguous block.</p></div></div><div id="disqus_thread"></div><script type="text/javascript">
+ </a></span></dt><dd><dl><dt><span class="section"><a href="apes03.html#d2279e12067">E.3.1. Overview</a></span></dt><dt><span class="section"><a href="apes03.html#d2279e12082">E.3.2. Unified version 2 block format</a></span></dt><dt><span class="section"><a href="apes03.html#d2279e12151">E.3.3. Block index in version 2</a></span></dt><dt><span class="section"><a href="apes03.html#d2279e12176">E.3.4.
+ Root block index format in version 2</a></span></dt><dt><span class="section"><a href="apes03.html#d2279e12229">E.3.5.
+ Non-root block index format in version 2</a></span></dt><dt><span class="section"><a href="apes03.html#d2279e12254">E.3.6.
+ Bloom filters in version 2</a></span></dt><dt><span class="section"><a href="apes03.html#d2279e12291">E.3.7. File Info format in versions 1 and 2</a></span></dt><dt><span class="section"><a href="apes03.html#d2279e12337">E.3.8.
+ Fixed file trailer format differences between versions 1 and 2</a></span></dt></dl></dd></dl></div><div class="section" title="E.1. Motivation"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="d2279e12005"></a>E.1. Motivation </h2></div></div></div><p>Note: this feature was introduced in HBase 0.92</p><p>We found it necessary to revise the HFile format after encountering high memory usage and slow startup times caused by large Bloom filters and block indexes in the region server. Bloom filters can get as large as 100 MB per HFile, which adds up to 2 GB when aggregated over 20 regions. Block indexes can grow as large as 6 GB in aggregate size over the same set of regions. A region is not considered opened until all of its block index data is loaded. Large Bloom filters produce a different performance problem: the first get request that requires a Bloom filter lookup will incur the latency of loading the entire Bloom filter bit a
rray.</p><p>To speed up region server startup we break Bloom filters and block indexes into multiple blocks and write those blocks out as they fill up, which also reduces the HFile writer’s memory footprint. In the Bloom filter case, “filling up a block” means accumulating enough keys to efficiently utilize a fixed-size bit array, and in the block index case we accumulate an “index block” of the desired size. Bloom filter blocks and index blocks (we call these “inline blocks”) become interspersed with data blocks, and as a side effect we can no longer rely on the difference between block offsets to determine data block length, as it was done in version 1.</p><p>HFile is a low-level file format by design, and it should not deal with application-specific details such as Bloom filters, which are handled at StoreFile level. Therefore, we call Bloom filter blocks in an HFile "inline" blocks. We also supply HFile with an interface to write
those inline blocks. </p><p>Another format modification aimed at reducing the region server startup time is to use a contiguous “load-on-open” section that has to be loaded in memory at the time an HFile is being opened. Currently, as an HFile opens, there are separate seek operations to read the trailer, data/meta indexes, and file info. To read the Bloom filter, there are two more seek operations for its “data” and “meta” portions. In version 2, we seek once to read the trailer and seek again to read everything else we need to open the file from a contiguous block.</p></div></div><div id="disqus_thread"></div><script type="text/javascript">
var disqus_shortname = 'hbase'; // required: replace example with your forum shortname
var disqus_url = 'http://hbase.apache.org/book';
var disqus_identifier = 'hfilev2';
Modified: hbase/hbase.apache.org/trunk/book/important_configurations.html
URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/important_configurations.html?rev=1462679&r1=1462678&r2=1462679&view=diff
==============================================================================
--- hbase/hbase.apache.org/trunk/book/important_configurations.html (original)
+++ hbase/hbase.apache.org/trunk/book/important_configurations.html Sat Mar 30 00:19:55 2013
@@ -120,7 +120,7 @@
Keeping 5 regions per RS would be too low for a job, whereas 1000 will generate too many maps.
</p></div></div><div class="section" title="2.5.2.7. Managed Splitting"><div class="titlepage"><div><div><h4 class="title"><a name="disable.splitting"></a>2.5.2.7. Managed Splitting</h4></div></div></div><p>
Rather than let HBase auto-split your Regions, manage the splitting manually
- <sup>[<a name="d2274e2558" href="#ftn.d2274e2558" class="footnote">11</a>]</sup>.
+ <sup>[<a name="d2279e2558" href="#ftn.d2279e2558" class="footnote">11</a>]</sup>.
With growing amounts of data, splits will continually be needed. Since
you always know exactly what regions you have, long-term debugging and
profiling is much easier with manual splits. It is hard to trace the logs to
@@ -180,7 +180,7 @@ of all regions.
<a class="link" href="http://search-hadoop.com/m/pduLg2fydtE/Inconsistent+scan+performance+with+caching+set+&subj=Re+Inconsistent+scan+performance+with+caching+set+to+1" target="_top">Inconsistent scan performance with caching set to 1</a>
and the issue cited therein where setting notcpdelay improved scan speeds. You might also
see the graphs on the tail of <a class="link" href="https://issues.apache.org/jira/browse/HBASE-7008" target="_top">HBASE-7008 Set scanner caching to a better default</a>
- where our Lars Hofhansl tries various data sizes w/ Nagle's on and off measuring the effect.</p></div></div><div class="footnotes"><br><hr width="100" align="left"><div class="footnote"><p><sup>[<a id="ftn.d2274e2558" href="#d2274e2558" class="para">11</a>] </sup>What follows is taken from the javadoc at the head of
+ where our Lars Hofhansl tries various data sizes w/ Nagle's on and off measuring the effect.</p></div></div><div class="footnotes"><br><hr width="100" align="left"><div class="footnote"><p><sup>[<a id="ftn.d2279e2558" href="#d2279e2558" class="para">11</a>] </sup>What follows is taken from the javadoc at the head of
the <code class="classname">org.apache.hadoop.hbase.util.RegionSplitter</code> tool
added to HBase post-0.90.0 release.
</p></div></div></div><div id="disqus_thread"></div><script type="text/javascript">
Modified: hbase/hbase.apache.org/trunk/book/jvm.html
URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/jvm.html?rev=1462679&r1=1462678&r2=1462679&view=diff
==============================================================================
--- hbase/hbase.apache.org/trunk/book/jvm.html (original)
+++ hbase/hbase.apache.org/trunk/book/jvm.html Sat Mar 30 00:19:55 2013
@@ -10,11 +10,11 @@
from defaults. Start at 60 or 70 percent (The lower you bring down the
threshold, the more GCing is done, the more CPU used). To address the
second fragmentation issue, Todd added an experimental facility,
- <a class="indexterm" name="d2274e6867"></a>, that
+ <a class="indexterm" name="d2279e6867"></a>, that
must be explicitly enabled in Apache HBase 0.90.x (Its defaulted to be on in
Apache 0.92.x HBase). See <code class="code">hbase.hregion.memstore.mslab.enabled</code>
to true in your <code class="classname">Configuration</code>. See the cited
- slides for background and detail<sup>[<a name="d2274e6877" href="#ftn.d2274e6877" class="footnote">24</a>]</sup>.
+ slides for background and detail<sup>[<a name="d2279e6877" href="#ftn.d2279e6877" class="footnote">24</a>]</sup>.
Be aware that when enabled, each MemStore instance will occupy at least
an MSLAB instance of memory. If you have thousands of regions or lots
of regions each with many column families, this allocation of MSLAB
@@ -22,7 +22,7 @@
an extreme case cause you to OOME. Disable MSLAB in this case, or
lower the amount of memory it uses or float less regions per server.
</p><p>For more information about GC logs, see <a class="xref" href="trouble.log.html#trouble.log.gc" title="12.2.3. JVM Garbage Collection Logs">Section 12.2.3, “JVM Garbage Collection Logs”</a>.
- </p></div></div><div class="footnotes"><br><hr width="100" align="left"><div class="footnote"><p><sup>[<a id="ftn.d2274e6877" href="#d2274e6877" class="para">24</a>] </sup>The latest jvms do better
+ </p></div></div><div class="footnotes"><br><hr width="100" align="left"><div class="footnote"><p><sup>[<a id="ftn.d2279e6877" href="#d2279e6877" class="para">24</a>] </sup>The latest jvms do better
regards fragmentation so make sure you are running a recent release.
Read down in the message,
<a class="link" href="http://osdir.com/ml/hotspot-gc-use/2011-11/msg00002.html" target="_top">Identifying concurrent mode failures caused by fragmentation</a>.</p></div></div></div><div id="disqus_thread"></div><script type="text/javascript">
Modified: hbase/hbase.apache.org/trunk/book/mapreduce.example.html
URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/mapreduce.example.html?rev=1462679&r1=1462678&r2=1462679&view=diff
==============================================================================
--- hbase/hbase.apache.org/trunk/book/mapreduce.example.html (original)
+++ hbase/hbase.apache.org/trunk/book/mapreduce.example.html Sat Mar 30 00:19:55 2013
@@ -133,12 +133,14 @@ if (!b) {
This value is used as the key to emit from the mapper, and an <code class="classname">IntWritable</code> represents an instance counter.
</p><pre class="programlisting">
public static class MyMapper extends TableMapper<Text, IntWritable> {
+ public static final byte[] CF = "cf".getBytes();
+ public static final byte[] ATTR1 = "attr1".getBytes();
private final IntWritable ONE = new IntWritable(1);
private Text text = new Text();
public void map(ImmutableBytesWritable row, Result value, Context context) throws IOException, InterruptedException {
- String val = new String(value.getValue(Bytes.toBytes("cf"), Bytes.toBytes("attr1")));
+ String val = new String(value.getValue(CF, ATTR1));
text.set(val); // we can only emit Writables...
context.write(text, ONE);
@@ -148,6 +150,8 @@ public static class MyMapper extends Tab
In the reducer, the "ones" are counted (just like any other MR example that does this), and then emits a <code class="classname">Put</code>.
</p><pre class="programlisting">
public static class MyTableReducer extends TableReducer<Text, IntWritable, ImmutableBytesWritable> {
+ public static final byte[] CF = "cf".getBytes();
+ public static final byte[] COUNT = "count".getBytes();
public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
int i = 0;
@@ -155,7 +159,7 @@ public static class MyTableReducer exten
i += val.get();
}
Put put = new Put(Bytes.toBytes(key.toString()));
- put.add(Bytes.toBytes("cf"), Bytes.toBytes("count"), Bytes.toBytes(i));
+ put.add(CF, COUNT, Bytes.toBytes(i));
context.write(null, put);
}
Modified: hbase/hbase.apache.org/trunk/book/master.html
URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/master.html?rev=1462679&r1=1462678&r2=1462679&view=diff
==============================================================================
--- hbase/hbase.apache.org/trunk/book/master.html (original)
+++ hbase/hbase.apache.org/trunk/book/master.html Sat Mar 30 00:19:55 2013
@@ -2,7 +2,7 @@
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>9.5. Master</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="architecture.html" title="Chapter 9. Architecture"><link rel="prev" href="client.filter.html" title="9.4. Client Request Filters"><link rel="next" href="regionserver.arch.html" title="9.6. RegionServer"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">9.5. Master</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="client.filter.html">Prev</a> </td><th width="60%" align="center">Chapter 9. Architecture</th><td width="20%" align="right"> <a accesskey="n" href="regionserver.arch.html">Next</a></td></tr></t
able><hr></div><div class="section" title="9.5. Master"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="master"></a>9.5. Master</h2></div></div></div><p><code class="code">HMaster</code> is the implementation of the Master Server. The Master server
is responsible for monitoring all RegionServer instances in the cluster, and is
- the interface for all metadata changes. In a distributed cluster, the Master typically runs on the <a class="xref" href="arch.hdfs.html#arch.hdfs.nn" title="9.9.1. NameNode">Section 9.9.1, “NameNode”</a><sup>[<a name="d2274e5133" href="#ftn.d2274e5133" class="footnote">21</a>]</sup>
+ the interface for all metadata changes. In a distributed cluster, the Master typically runs on the <a class="xref" href="arch.hdfs.html#arch.hdfs.nn" title="9.9.1. NameNode">Section 9.9.1, “NameNode”</a><sup>[<a name="d2279e5133" href="#ftn.d2279e5133" class="footnote">21</a>]</sup>
</p><div class="section" title="9.5.1. Startup Behavior"><div class="titlepage"><div><div><h3 class="title"><a name="master.startup"></a>9.5.1. Startup Behavior</h3></div></div></div><p>If run in a multi-Master environment, all Masters compete to run the cluster. If the active
Master loses its lease in ZooKeeper (or the Master shuts down), then then the remaining Masters jostle to
take over the Master role.
@@ -22,7 +22,7 @@
</p><div class="section" title="9.5.4.1. LoadBalancer"><div class="titlepage"><div><div><h4 class="title"><a name="master.processes.loadbalancer"></a>9.5.4.1. LoadBalancer</h4></div></div></div><p>Periodically, and when there are no regions in transition,
a load balancer will run and move regions around to balance the cluster's load.
See <a class="xref" href="important_configurations.html#balancer_config" title="2.5.3.1. Balancer">Section 2.5.3.1, “Balancer”</a> for configuring this property.</p><p>See <a class="xref" href="regions.arch.html#regions.arch.assignment" title="9.7.2. Region-RegionServer Assignment">Section 9.7.2, “Region-RegionServer Assignment”</a> for more information on region assignment.
- </p></div><div class="section" title="9.5.4.2. CatalogJanitor"><div class="titlepage"><div><div><h4 class="title"><a name="master.processes.catalog"></a>9.5.4.2. CatalogJanitor</h4></div></div></div><p>Periodically checks and cleans up the .META. table. See <a class="xref" href="arch.catalog.html#arch.catalog.meta" title="9.2.2. META">Section 9.2.2, “META”</a> for more information on META.</p></div></div><div class="footnotes"><br><hr width="100" align="left"><div class="footnote"><p><sup>[<a id="ftn.d2274e5133" href="#d2274e5133" class="para">21</a>] </sup>J Mohamed Zahoor goes into some more detail on the Master Architecture in this blog posting, <a class="link" href="http://blog.zahoor.in/2012/08/hbase-hmaster-architecture/" target="_top">HBase HMaster Architecture
+ </p></div><div class="section" title="9.5.4.2. CatalogJanitor"><div class="titlepage"><div><div><h4 class="title"><a name="master.processes.catalog"></a>9.5.4.2. CatalogJanitor</h4></div></div></div><p>Periodically checks and cleans up the .META. table. See <a class="xref" href="arch.catalog.html#arch.catalog.meta" title="9.2.2. META">Section 9.2.2, “META”</a> for more information on META.</p></div></div><div class="footnotes"><br><hr width="100" align="left"><div class="footnote"><p><sup>[<a id="ftn.d2279e5133" href="#d2279e5133" class="para">21</a>] </sup>J Mohamed Zahoor goes into some more detail on the Master Architecture in this blog posting, <a class="link" href="http://blog.zahoor.in/2012/08/hbase-hmaster-architecture/" target="_top">HBase HMaster Architecture
</a>.</p></div></div></div><div id="disqus_thread"></div><script type="text/javascript">
var disqus_shortname = 'hbase'; // required: replace example with your forum shortname
var disqus_url = 'http://hbase.apache.org/book';
Modified: hbase/hbase.apache.org/trunk/book/ops.monitoring.html
URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/ops.monitoring.html?rev=1462679&r1=1462678&r2=1462679&view=diff
==============================================================================
--- hbase/hbase.apache.org/trunk/book/ops.monitoring.html (original)
+++ hbase/hbase.apache.org/trunk/book/ops.monitoring.html Sat Mar 30 00:19:55 2013
@@ -14,13 +14,13 @@
</p><p>
For more information on HBase metrics, see <a class="xref" href="hbase_metrics.html" title="14.4. HBase Metrics">Section 14.4, “HBase Metrics”</a>.
</p></div><div class="section" title="14.5.2. Slow Query Log"><div class="titlepage"><div><div><h3 class="title"><a name="ops.slow.query"></a>14.5.2. Slow Query Log</h3></div></div></div><p>The HBase slow query log consists of parseable JSON structures describing the properties of those client operations (Gets, Puts, Deletes, etc.) that either took too long to run, or produced too much output. The thresholds for "too long to run" and "too much output" are configurable, as described below. The output is produced inline in the main region server logs so that it is easy to discover further details from context with other logged events. It is also prepended with identifying tags <code class="constant">(responseTooSlow)</code>, <code class="constant">(responseTooLarge)</code>, <code class="constant">(operationTooSlow)</code>, and <code class="constant">(operationTooLarge)</code> in order to enable easy filtering with grep, in case the user desires to see only slow
queries.
-</p><div class="section" title="14.5.2.1. Configuration"><div class="titlepage"><div><div><h4 class="title"><a name="d2274e9519"></a>14.5.2.1. Configuration</h4></div></div></div><p>There are two configuration knobs that can be used to adjust the thresholds for when queries are logged.
+</p><div class="section" title="14.5.2.1. Configuration"><div class="titlepage"><div><div><h4 class="title"><a name="d2279e9533"></a>14.5.2.1. Configuration</h4></div></div></div><p>There are two configuration knobs that can be used to adjust the thresholds for when queries are logged.
</p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem"><code class="varname">hbase.ipc.warn.response.time</code> Maximum number of milliseconds that a query can be run without being logged. Defaults to 10000, or 10 seconds. Can be set to -1 to disable logging by time.
</li><li class="listitem"><code class="varname">hbase.ipc.warn.response.size</code> Maximum byte size of response that a query can return without being logged. Defaults to 100 megabytes. Can be set to -1 to disable logging by size.
-</li></ul></div></div><div class="section" title="14.5.2.2. Metrics"><div class="titlepage"><div><div><h4 class="title"><a name="d2274e9533"></a>14.5.2.2. Metrics</h4></div></div></div><p>The slow query log exposes to metrics to JMX.
+</li></ul></div></div><div class="section" title="14.5.2.2. Metrics"><div class="titlepage"><div><div><h4 class="title"><a name="d2279e9547"></a>14.5.2.2. Metrics</h4></div></div></div><p>The slow query log exposes to metrics to JMX.
</p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem"><code class="varname">hadoop.regionserver_rpc_slowResponse</code> a global metric reflecting the durations of all responses that triggered logging.</li><li class="listitem"><code class="varname">hadoop.regionserver_rpc_methodName.aboveOneSec</code> A metric reflecting the durations of all responses that lasted for more than one second.</li></ul></div><p>
-</p></div><div class="section" title="14.5.2.3. Output"><div class="titlepage"><div><div><h4 class="title"><a name="d2274e9548"></a>14.5.2.3. Output</h4></div></div></div><p>The output is tagged with operation e.g. <code class="constant">(operationTooSlow)</code> if the call was a client operation, such as a Put, Get, or Delete, which we expose detailed fingerprint information for. If not, it is tagged <code class="constant">(responseTooSlow)</code> and still produces parseable JSON output, but with less verbose information solely regarding its duration and size in the RPC itself. <code class="constant">TooLarge</code> is substituted for <code class="constant">TooSlow</code> if the response size triggered the logging, with <code class="constant">TooLarge</code> appearing even in the case that both size and duration triggered logging.
-</p></div><div class="section" title="14.5.2.4. Example"><div class="titlepage"><div><div><h4 class="title"><a name="d2274e9568"></a>14.5.2.4. Example</h4></div></div></div><p>
+</p></div><div class="section" title="14.5.2.3. Output"><div class="titlepage"><div><div><h4 class="title"><a name="d2279e9562"></a>14.5.2.3. Output</h4></div></div></div><p>The output is tagged with operation e.g. <code class="constant">(operationTooSlow)</code> if the call was a client operation, such as a Put, Get, or Delete, which we expose detailed fingerprint information for. If not, it is tagged <code class="constant">(responseTooSlow)</code> and still produces parseable JSON output, but with less verbose information solely regarding its duration and size in the RPC itself. <code class="constant">TooLarge</code> is substituted for <code class="constant">TooSlow</code> if the response size triggered the logging, with <code class="constant">TooLarge</code> appearing even in the case that both size and duration triggered logging.
+</p></div><div class="section" title="14.5.2.4. Example"><div class="titlepage"><div><div><h4 class="title"><a name="d2279e9582"></a>14.5.2.4. Example</h4></div></div></div><p>
</p><pre class="programlisting">2011-09-08 10:01:25,824 WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow): {"tables":{"riley2":{"puts":[{"totalColumns":11,"families":{"actions":[{"timestamp":1315501284459,"qualifier":"0","vlen":9667580},{"timestamp":1315501284459,"qualifier":"1","vlen":10122412},{"timestamp":1315501284459,"qualifier":"2","vlen":11104617},{"timestamp":1315501284459,"qualifier":"3","vlen":13430635}]},"row":"cfcd208495d565ef66e7dff9f98764da:0"}],"families":["actions"]}},"processingtimems":956,"client":"10.47.34.63:33623","starttimems":1315501284456,"queuetimems":0,"totalPuts":1,"class":"HRegionServer","responsesize":0,"method":"multiPut"}</pre><p>
</p><p>Note that everything inside the "tables" structure is output produced by MultiPut's fingerprint, while the rest of the information is RPC-specific, such as processing time and client IP/port. Other client operations follow the same pattern and the same general structure, with necessary differences due to the nature of the individual operations. In the case that the call is not a client operation, that detailed fingerprint information will be completely absent.
</p><p>This particular example, for example, would indicate that the likely cause of slowness is simply a very large (on the order of 100MB) multiput, as we can tell by the "vlen," or value length, fields of each put in the multiPut.
Modified: hbase/hbase.apache.org/trunk/book/ops.regionmgt.html
URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/ops.regionmgt.html?rev=1462679&r1=1462678&r2=1462679&view=diff
==============================================================================
--- hbase/hbase.apache.org/trunk/book/ops.regionmgt.html (original)
+++ hbase/hbase.apache.org/trunk/book/ops.regionmgt.html Sat Mar 30 00:19:55 2013
@@ -3,7 +3,7 @@
<title>14.2. Region Management</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="ops_mgt.html" title="Chapter 14. Apache HBase (TM) Operational Management"><link rel="prev" href="ops_mgt.html" title="Chapter 14. Apache HBase (TM) Operational Management"><link rel="next" href="node.management.html" title="14.3. Node Management"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">14.2. Region Management</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="ops_mgt.html">Prev</a> </td><th width="60%" align="center">Chapter 14. Apache HBase (TM) Operational Management</th><td wid
th="20%" align="right"> <a accesskey="n" href="node.management.html">Next</a></td></tr></table><hr></div><div class="section" title="14.2. Region Management"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="ops.regionmgt"></a>14.2. Region Management</h2></div></div></div><div class="section" title="14.2.1. Major Compaction"><div class="titlepage"><div><div><h3 class="title"><a name="ops.regionmgt.majorcompact"></a>14.2.1. Major Compaction</h3></div></div></div><p>Major compactions can be requested via the HBase shell or <a class="link" href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HBaseAdmin.html#majorCompact%28java.lang.String%29" target="_top">HBaseAdmin.majorCompact</a>.
</p><p>Note: major compactions do NOT do region merges. See <a class="xref" href="regions.arch.html#compaction" title="9.7.5.5. Compaction">Section 9.7.5.5, “Compaction”</a> for more information about compactions.
- </p></div><div class="section" title="14.2.2. Merge"><div class="titlepage"><div><div><h3 class="title"><a name="ops.regionmgt.merge"></a>14.2.2. Merge</h3></div></div></div><p>Merge is a utility that can merge adjoining regions in the same table (see org.apache.hadoop.hbase.util.Merge).</p><pre class="programlisting">$ bin/hbase org.apache.hbase.util.Merge <tablename> <region1> <region2>
+ </p></div><div class="section" title="14.2.2. Merge"><div class="titlepage"><div><div><h3 class="title"><a name="ops.regionmgt.merge"></a>14.2.2. Merge</h3></div></div></div><p>Merge is a utility that can merge adjoining regions in the same table (see org.apache.hadoop.hbase.util.Merge).</p><pre class="programlisting">$ bin/hbase org.apache.hadoop.hbase.util.Merge <tablename> <region1> <region2>
</pre><p>If you feel you have too many regions and want to consolidate them, Merge is the utility you need. Merge must
run be done when the cluster is down.
See the <a class="link" href="http://ofps.oreilly.com/titles/9781449396107/performance.html" target="_top">O'Reilly HBase Book</a> for
Modified: hbase/hbase.apache.org/trunk/book/perf.casestudy.html
URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/perf.casestudy.html?rev=1462679&r1=1462678&r2=1462679&view=diff
==============================================================================
--- hbase/hbase.apache.org/trunk/book/perf.casestudy.html (original)
+++ hbase/hbase.apache.org/trunk/book/perf.casestudy.html Sat Mar 30 00:19:55 2013
@@ -1,6 +1,6 @@
<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
- <title>11.12. Case Studies</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="performance.html" title="Chapter 11. Apache HBase (TM) Performance Tuning"><link rel="prev" href="perf.ec2.html" title="11.11. Amazon EC2"><link rel="next" href="trouble.html" title="Chapter 12. Troubleshooting and Debugging Apache HBase (TM)"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">11.12. Case Studies</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="perf.ec2.html">Prev</a> </td><th width="60%" align="center">Chapter 11. Apache HBase (TM) Performance Tuning</th><td width="20%" align=
"right"> <a accesskey="n" href="trouble.html">Next</a></td></tr></table><hr></div><div class="section" title="11.12. Case Studies"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="perf.casestudy"></a>11.12. Case Studies</h2></div></div></div><p>For Performance and Troubleshooting Case Studies, see <a class="xref" href="casestudies.html" title="Chapter 13. Apache HBase (TM) Case Studies">Chapter 13, <i>Apache HBase (TM) Case Studies</i></a>.
+ <title>11.13. Case Studies</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="performance.html" title="Chapter 11. Apache HBase (TM) Performance Tuning"><link rel="prev" href="perf.ec2.html" title="11.12. Amazon EC2"><link rel="next" href="trouble.html" title="Chapter 12. Troubleshooting and Debugging Apache HBase (TM)"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">11.13. Case Studies</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="perf.ec2.html">Prev</a> </td><th width="60%" align="center">Chapter 11. Apache HBase (TM) Performance Tuning</th><td width="20%" align=
"right"> <a accesskey="n" href="trouble.html">Next</a></td></tr></table><hr></div><div class="section" title="11.13. Case Studies"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="perf.casestudy"></a>11.13. Case Studies</h2></div></div></div><p>For Performance and Troubleshooting Case Studies, see <a class="xref" href="casestudies.html" title="Chapter 13. Apache HBase (TM) Case Studies">Chapter 13, <i>Apache HBase (TM) Case Studies</i></a>.
</p></div><div id="disqus_thread"></div><script type="text/javascript">
var disqus_shortname = 'hbase'; // required: replace example with your forum shortname
var disqus_url = 'http://hbase.apache.org/book';
@@ -12,4 +12,4 @@
dsq.src = 'http://' + disqus_shortname + '.disqus.com/embed.js';
(document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
})();
-</script><noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript">comments powered by Disqus.</a></noscript><a href="http://disqus.com" class="dsq-brlink">comments powered by <span class="logo-disqus">Disqus</span></a><div class="navfooter"><hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="perf.ec2.html">Prev</a> </td><td width="20%" align="center"><a accesskey="u" href="performance.html">Up</a></td><td width="40%" align="right"> <a accesskey="n" href="trouble.html">Next</a></td></tr><tr><td width="40%" align="left" valign="top">11.11. Amazon EC2 </td><td width="20%" align="center"><a accesskey="h" href="book.html">Home</a></td><td width="40%" align="right" valign="top"> Chapter 12. Troubleshooting and Debugging Apache HBase (TM)</td></tr></table></div></body></html>
\ No newline at end of file
+</script><noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript">comments powered by Disqus.</a></noscript><a href="http://disqus.com" class="dsq-brlink">comments powered by <span class="logo-disqus">Disqus</span></a><div class="navfooter"><hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="perf.ec2.html">Prev</a> </td><td width="20%" align="center"><a accesskey="u" href="performance.html">Up</a></td><td width="40%" align="right"> <a accesskey="n" href="trouble.html">Next</a></td></tr><tr><td width="40%" align="left" valign="top">11.12. Amazon EC2 </td><td width="20%" align="center"><a accesskey="h" href="book.html">Home</a></td><td width="40%" align="right" valign="top"> Chapter 12. Troubleshooting and Debugging Apache HBase (TM)</td></tr></table></div></body></html>
\ No newline at end of file
Modified: hbase/hbase.apache.org/trunk/book/perf.deleting.html
URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/perf.deleting.html?rev=1462679&r1=1462678&r2=1462679&view=diff
==============================================================================
--- hbase/hbase.apache.org/trunk/book/perf.deleting.html (original)
+++ hbase/hbase.apache.org/trunk/book/perf.deleting.html Sat Mar 30 00:19:55 2013
@@ -1,10 +1,10 @@
<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
- <title>11.9. Deleting from HBase</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="performance.html" title="Chapter 11. Apache HBase (TM) Performance Tuning"><link rel="prev" href="perf.reading.html" title="11.8. Reading from HBase"><link rel="next" href="perf.hdfs.html" title="11.10. HDFS"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">11.9. Deleting from HBase</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="perf.reading.html">Prev</a> </td><th width="60%" align="center">Chapter 11. Apache HBase (TM) Performance Tuning</th><td width="20%" align="right"> <a accessk
ey="n" href="perf.hdfs.html">Next</a></td></tr></table><hr></div><div class="section" title="11.9. Deleting from HBase"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="perf.deleting"></a>11.9. Deleting from HBase</h2></div></div></div><div class="section" title="11.9.1. Using HBase Tables as Queues"><div class="titlepage"><div><div><h3 class="title"><a name="perf.deleting.queue"></a>11.9.1. Using HBase Tables as Queues</h3></div></div></div><p>HBase tables are sometimes used as queues. In this case, special care must be taken to regularly perform major compactions on tables used in
+ <title>11.10. Deleting from HBase</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="performance.html" title="Chapter 11. Apache HBase (TM) Performance Tuning"><link rel="prev" href="perf.reading.html" title="11.9. Reading from HBase"><link rel="next" href="perf.hdfs.html" title="11.11. HDFS"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">11.10. Deleting from HBase</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="perf.reading.html">Prev</a> </td><th width="60%" align="center">Chapter 11. Apache HBase (TM) Performance Tuning</th><td width="20%" align="right"> <a acces
skey="n" href="perf.hdfs.html">Next</a></td></tr></table><hr></div><div class="section" title="11.10. Deleting from HBase"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="perf.deleting"></a>11.10. Deleting from HBase</h2></div></div></div><div class="section" title="11.10.1. Using HBase Tables as Queues"><div class="titlepage"><div><div><h3 class="title"><a name="perf.deleting.queue"></a>11.10.1. Using HBase Tables as Queues</h3></div></div></div><p>HBase tables are sometimes used as queues. In this case, special care must be taken to regularly perform major compactions on tables used in
this manner. As is documented in <a class="xref" href="datamodel.html" title="Chapter 5. Data Model">Chapter 5, <i>Data Model</i></a>, marking rows as deleted creates additional StoreFiles which then need to be processed
on reads. Tombstones only get cleaned up with major compactions.
</p><p>See also <a class="xref" href="regions.arch.html#compaction" title="9.7.5.5. Compaction">Section 9.7.5.5, “Compaction”</a> and <a class="link" href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HBaseAdmin.html#majorCompact%28java.lang.String%29" target="_top">HBaseAdmin.majorCompact</a>.
- </p></div><div class="section" title="11.9.2. Delete RPC Behavior"><div class="titlepage"><div><div><h3 class="title"><a name="perf.deleting.rpc"></a>11.9.2. Delete RPC Behavior</h3></div></div></div><p>Be aware that <code class="code">htable.delete(Delete)</code> doesn't use the writeBuffer. It will execute an RegionServer RPC with each invocation.
+ </p></div><div class="section" title="11.10.2. Delete RPC Behavior"><div class="titlepage"><div><div><h3 class="title"><a name="perf.deleting.rpc"></a>11.10.2. Delete RPC Behavior</h3></div></div></div><p>Be aware that <code class="code">htable.delete(Delete)</code> doesn't use the writeBuffer. It will execute an RegionServer RPC with each invocation.
For a large number of deletes, consider <code class="code">htable.delete(List)</code>.
</p><p>See <a class="link" href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HTable.html#delete%28org.apache.hadoop.hbase.client.Delete%29" target="_top">http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HTable.html#delete%28org.apache.hadoop.hbase.client.Delete%29</a>
</p></div></div><div id="disqus_thread"></div><script type="text/javascript">
@@ -18,4 +18,4 @@
dsq.src = 'http://' + disqus_shortname + '.disqus.com/embed.js';
(document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
})();
-</script><noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript">comments powered by Disqus.</a></noscript><a href="http://disqus.com" class="dsq-brlink">comments powered by <span class="logo-disqus">Disqus</span></a><div class="navfooter"><hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="perf.reading.html">Prev</a> </td><td width="20%" align="center"><a accesskey="u" href="performance.html">Up</a></td><td width="40%" align="right"> <a accesskey="n" href="perf.hdfs.html">Next</a></td></tr><tr><td width="40%" align="left" valign="top">11.8. Reading from HBase </td><td width="20%" align="center"><a accesskey="h" href="book.html">Home</a></td><td width="40%" align="right" valign="top"> 11.10. HDFS</td></tr></table></div></body></html>
\ No newline at end of file
+</script><noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript">comments powered by Disqus.</a></noscript><a href="http://disqus.com" class="dsq-brlink">comments powered by <span class="logo-disqus">Disqus</span></a><div class="navfooter"><hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="perf.reading.html">Prev</a> </td><td width="20%" align="center"><a accesskey="u" href="performance.html">Up</a></td><td width="40%" align="right"> <a accesskey="n" href="perf.hdfs.html">Next</a></td></tr><tr><td width="40%" align="left" valign="top">11.9. Reading from HBase </td><td width="20%" align="center"><a accesskey="h" href="book.html">Home</a></td><td width="40%" align="right" valign="top"> 11.11. HDFS</td></tr></table></div></body></html>
\ No newline at end of file
Modified: hbase/hbase.apache.org/trunk/book/perf.ec2.html
URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/perf.ec2.html?rev=1462679&r1=1462678&r2=1462679&view=diff
==============================================================================
--- hbase/hbase.apache.org/trunk/book/perf.ec2.html (original)
+++ hbase/hbase.apache.org/trunk/book/perf.ec2.html Sat Mar 30 00:19:55 2013
@@ -1,6 +1,6 @@
<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
- <title>11.11. Amazon EC2</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="performance.html" title="Chapter 11. Apache HBase (TM) Performance Tuning"><link rel="prev" href="perf.hdfs.html" title="11.10. HDFS"><link rel="next" href="perf.casestudy.html" title="11.12. Case Studies"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">11.11. Amazon EC2</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="perf.hdfs.html">Prev</a> </td><th width="60%" align="center">Chapter 11. Apache HBase (TM) Performance Tuning</th><td width="20%" align="right"> <a accesskey="n" href="perf.case
study.html">Next</a></td></tr></table><hr></div><div class="section" title="11.11. Amazon EC2"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="perf.ec2"></a>11.11. Amazon EC2</h2></div></div></div><p>Performance questions are common on Amazon EC2 environments because it is a shared environment. You will
+ <title>11.12. Amazon EC2</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="performance.html" title="Chapter 11. Apache HBase (TM) Performance Tuning"><link rel="prev" href="perf.hdfs.html" title="11.11. HDFS"><link rel="next" href="perf.casestudy.html" title="11.13. Case Studies"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">11.12. Amazon EC2</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="perf.hdfs.html">Prev</a> </td><th width="60%" align="center">Chapter 11. Apache HBase (TM) Performance Tuning</th><td width="20%" align="right"> <a accesskey="n" href="perf.case
study.html">Next</a></td></tr></table><hr></div><div class="section" title="11.12. Amazon EC2"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="perf.ec2"></a>11.12. Amazon EC2</h2></div></div></div><p>Performance questions are common on Amazon EC2 environments because it is a shared environment. You will
not see the same throughput as a dedicated server. In terms of running tests on EC2, run them several times for the same
reason (i.e., it's a shared environment and you don't know what else is happening on the server).
</p><p>If you are running on EC2 and post performance questions on the dist-list, please state this fact up-front that
@@ -16,4 +16,4 @@
dsq.src = 'http://' + disqus_shortname + '.disqus.com/embed.js';
(document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
})();
-</script><noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript">comments powered by Disqus.</a></noscript><a href="http://disqus.com" class="dsq-brlink">comments powered by <span class="logo-disqus">Disqus</span></a><div class="navfooter"><hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="perf.hdfs.html">Prev</a> </td><td width="20%" align="center"><a accesskey="u" href="performance.html">Up</a></td><td width="40%" align="right"> <a accesskey="n" href="perf.casestudy.html">Next</a></td></tr><tr><td width="40%" align="left" valign="top">11.10. HDFS </td><td width="20%" align="center"><a accesskey="h" href="book.html">Home</a></td><td width="40%" align="right" valign="top"> 11.12. Case Studies</td></tr></table></div></body></html>
\ No newline at end of file
+</script><noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript">comments powered by Disqus.</a></noscript><a href="http://disqus.com" class="dsq-brlink">comments powered by <span class="logo-disqus">Disqus</span></a><div class="navfooter"><hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="perf.hdfs.html">Prev</a> </td><td width="20%" align="center"><a accesskey="u" href="performance.html">Up</a></td><td width="40%" align="right"> <a accesskey="n" href="perf.casestudy.html">Next</a></td></tr><tr><td width="40%" align="left" valign="top">11.11. HDFS </td><td width="20%" align="center"><a accesskey="h" href="book.html">Home</a></td><td width="40%" align="right" valign="top"> 11.13. Case Studies</td></tr></table></div></body></html>
\ No newline at end of file
Added: hbase/hbase.apache.org/trunk/book/perf.general.html
URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/perf.general.html?rev=1462679&view=auto
==============================================================================
--- hbase/hbase.apache.org/trunk/book/perf.general.html (added)
+++ hbase/hbase.apache.org/trunk/book/perf.general.html Sat Mar 30 00:19:55 2013
@@ -0,0 +1,31 @@
+<html><head>
+ <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
+ <title>11.7. HBase General Patterns</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="performance.html" title="Chapter 11. Apache HBase (TM) Performance Tuning"><link rel="prev" href="perf.schema.html" title="11.6. Schema Design"><link rel="next" href="perf.writing.html" title="11.8. Writing to HBase"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">11.7. HBase General Patterns</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="perf.schema.html">Prev</a> </td><th width="60%" align="center">Chapter 11. Apache HBase (TM) Performance Tuning</th><td width="20%" align="right">&nb
sp;<a accesskey="n" href="perf.writing.html">Next</a></td></tr></table><hr></div><div class="section" title="11.7. HBase General Patterns"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="perf.general"></a>11.7. HBase General Patterns</h2></div></div></div><div class="section" title="11.7.1. Constants"><div class="titlepage"><div><div><h3 class="title"><a name="perf.general.constants"></a>11.7.1. Constants</h3></div></div></div><p>When people get started with HBase they have a tendency to write code that looks like this:
+</p><pre class="programlisting">
+Get get = new Get(rowkey);
+Result r = htable.get(get);
+byte[] b = r.getValue(Bytes.toBytes("cf"), Bytes.toBytes("attr")); // returns current version of value
+</pre><p>
+ But especially when inside loops (and MapReduce jobs), converting the columnFamily and column-names
+ to byte-arrays repeatedly is surprisingly expensive.
+ It's better to use constants for the byte-arrays, like this:
+</p><pre class="programlisting">
+public static final byte[] CF = "cf".getBytes();
+public static final byte[] ATTR = "attr".getBytes();
+...
+Get get = new Get(rowkey);
+Result r = htable.get(get);
+byte[] b = r.getValue(CF, ATTR); // returns current version of value
+</pre><p>
+ </p></div></div><div id="disqus_thread"></div><script type="text/javascript">
+ var disqus_shortname = 'hbase'; // required: replace example with your forum shortname
+ var disqus_url = 'http://hbase.apache.org/book';
+ var disqus_identifier = 'perf.general';
+
+ /* * * DON'T EDIT BELOW THIS LINE * * */
+ (function() {
+ var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
+ dsq.src = 'http://' + disqus_shortname + '.disqus.com/embed.js';
+ (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
+ })();
+</script><noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript">comments powered by Disqus.</a></noscript><a href="http://disqus.com" class="dsq-brlink">comments powered by <span class="logo-disqus">Disqus</span></a><div class="navfooter"><hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="perf.schema.html">Prev</a> </td><td width="20%" align="center"><a accesskey="u" href="performance.html">Up</a></td><td width="40%" align="right"> <a accesskey="n" href="perf.writing.html">Next</a></td></tr><tr><td width="40%" align="left" valign="top">11.6. Schema Design </td><td width="20%" align="center"><a accesskey="h" href="book.html">Home</a></td><td width="40%" align="right" valign="top"> 11.8. Writing to HBase</td></tr></table></div></body></html>
\ No newline at end of file