You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hbase.apache.org by st...@apache.org on 2011/02/25 00:12:21 UTC

svn commit: r1074349 - in /hbase/trunk: CHANGES.txt pom.xml src/docbkx/book.xml

Author: stack
Date: Thu Feb 24 23:12:21 2011
New Revision: 1074349

URL: http://svn.apache.org/viewvc?rev=1074349&view=rev
Log:
HBASE-3563 [site] Add one-page-only version of hbase doc

Modified:
    hbase/trunk/CHANGES.txt
    hbase/trunk/pom.xml
    hbase/trunk/src/docbkx/book.xml

Modified: hbase/trunk/CHANGES.txt
URL: http://svn.apache.org/viewvc/hbase/trunk/CHANGES.txt?rev=1074349&r1=1074348&r2=1074349&view=diff
==============================================================================
--- hbase/trunk/CHANGES.txt (original)
+++ hbase/trunk/CHANGES.txt Thu Feb 24 23:12:21 2011
@@ -78,6 +78,7 @@ Release 0.91.0 - Unreleased
                coprocesssors -- instead version each individually
    HBASE-3520  Update our bundled hadoop from branch-0.20-append to latest
                (rpc version 43)
+   HBASE-3563  [site] Add one-page-only version of hbase doc
 
 
   NEW FEATURES

Modified: hbase/trunk/pom.xml
URL: http://svn.apache.org/viewvc/hbase/trunk/pom.xml?rev=1074349&r1=1074348&r2=1074349&view=diff
==============================================================================
--- hbase/trunk/pom.xml (original)
+++ hbase/trunk/pom.xml Thu Feb 24 23:12:21 2011
@@ -287,10 +287,38 @@
         <version>2.0.11</version>
         <executions>
           <execution>
+              <id>multipage</id>
+              <phase>pre-site</phase>
+        <configuration>
+          <xincludeSupported>true</xincludeSupported>
+          <navigShowtitles>true</navigShowtitles>
+          <chunkedOutput>true</chunkedOutput>
+          <useIdAsFilename>true</useIdAsFilename>
+          <sectionAutolabelMaxDepth>100</sectionAutolabelMaxDepth>
+          <sectionAutolabel>true</sectionAutolabel>
+          <sectionLabelIncludesComponentLabel>true</sectionLabelIncludesComponentLabel>
+          <targetDirectory>${basedir}/target/site/book/</targetDirectory>
+          <htmlStylesheet>../css/freebsd_docbook.css</htmlStylesheet>
+        </configuration>
             <goals>
               <goal>generate-html</goal>
             </goals>
+          </execution>
+          <execution>
+              <id>onepage</id>
             <phase>pre-site</phase>
+        <configuration>
+          <xincludeSupported>true</xincludeSupported>
+          <useIdAsFilename>true</useIdAsFilename>
+          <sectionAutolabelMaxDepth>100</sectionAutolabelMaxDepth>
+          <sectionAutolabel>true</sectionAutolabel>
+          <sectionLabelIncludesComponentLabel>true</sectionLabelIncludesComponentLabel>
+          <targetDirectory>${basedir}/target/site/</targetDirectory>
+          <htmlStylesheet>css/freebsd_docbook.css</htmlStylesheet>
+        </configuration>
+            <goals>
+              <goal>generate-html</goal>
+            </goals>
           </execution>
         </executions>
         <dependencies>
@@ -301,16 +329,6 @@
             <scope>runtime</scope>
           </dependency>
         </dependencies>
-        <configuration>
-          <xincludeSupported>true</xincludeSupported>
-          <chunkedOutput>true</chunkedOutput>
-          <useIdAsFilename>true</useIdAsFilename>
-          <sectionAutolabelMaxDepth>100</sectionAutolabelMaxDepth>
-          <sectionAutolabel>true</sectionAutolabel>
-          <sectionLabelIncludesComponentLabel>true</sectionLabelIncludesComponentLabel>
-          <targetDirectory>${basedir}/target/site/</targetDirectory>
-          <htmlStylesheet>css/freebsd_docbook.css</htmlStylesheet>
-        </configuration>
       </plugin>
       <plugin>
         <artifactId>maven-assembly-plugin</artifactId>

Modified: hbase/trunk/src/docbkx/book.xml
URL: http://svn.apache.org/viewvc/hbase/trunk/src/docbkx/book.xml?rev=1074349&r1=1074348&r2=1074349&view=diff
==============================================================================
--- hbase/trunk/src/docbkx/book.xml (original)
+++ hbase/trunk/src/docbkx/book.xml Thu Feb 24 23:12:21 2011
@@ -251,7 +251,7 @@ hbase(main):013:0&gt; drop 'test'
             <para><programlisting>hbase(main):014:0&gt; exit</programlisting></para>
             </section>
 
-          <section>
+          <section xml:id="stopping">
           <title>Stopping HBase</title>
             <para>Stop your hbase instance by running the stop script.</para>
 
@@ -456,7 +456,7 @@ guide.
 
       </section>
 
-      <section><title>HBase run modes: Standalone and Distributed</title>
+      <section xml:id="standalone_dist"><title>HBase run modes: Standalone and Distributed</title>
           <para>HBase has two run modes: <link linkend="standalone">standalone</link>
               and <link linkend="distributed">distributed</link>.
               Out of the box, HBase runs in standalone mode.  To set up a
@@ -479,7 +479,7 @@ Set <varname>JAVA_HOME</varname> to poin
         talk to HBase.
       </para>
       </section>
-      <section><title>Distributed</title>
+      <section xml:id="distributed"><title>Distributed</title>
           <para>Distributed mode can be subdivided into distributed but all daemons run on a
           single node -- a.k.a <emphasis>pseudo-distributed</emphasis>-- and
           <emphasis>fully-distributed</emphasis> where the daemons 
@@ -595,7 +595,7 @@ make the following configuration.</para>
 &lt;/configuration&gt;
 </programlisting>
 
-<section><title><filename>regionservers</filename></title>
+<section xml:id="regionserver"><title><filename>regionservers</filename></title>
 <para>In addition, a fully-distributed mode requires that you
 modify <filename>conf/regionservers</filename>.
 The <filename><link linkend="regionservrers">regionservers</link></filename> file lists all hosts
@@ -820,7 +820,7 @@ before stopping the Hadoop daemons.</par
 
 
 
-    <section><title>Example Configurations</title>
+    <section xml:id="example_config"><title>Example Configurations</title>
     <section><title>Basic Distributed HBase Install</title>
     <para>Here is an example basic configuration for a distributed ten node cluster.
     The nodes are named <varname>example0</varname>, <varname>example1</varname>, etc., through
@@ -1002,7 +1002,7 @@ to ensure well-formedness of your docume
     Use <command>rsync</command>.</para>
 
 
-    <section>
+    <section xml:id="hbase.site">
     <title><filename>hbase-site.xml</filename> and <filename>hbase-default.xml</filename></title>
     <para>Just as in Hadoop where you add site-specific HDFS configuration
     to the <filename>hdfs-site.xml</filename> file,
@@ -1032,7 +1032,7 @@ to ensure well-formedness of your docume
       href="../../target/site/hbase-default.xml" />
     </section>
 
-      <section>
+      <section xml:id="hbase.env.sh">
       <title><filename>hbase-env.sh</filename></title>
       <para>Set HBase environment variables in this file.
       Examples include options to pass the JVM on start of
@@ -1280,7 +1280,7 @@ of all regions.
             <para>See <link linkend="shell_exercises">Shell Exercises</link>
             for example basic shell operation.</para>
 
-    <section><title>Scripting</title>
+    <section xml:id="scripting"><title>Scripting</title>
         <para>For examples scripting HBase, look in the
             HBase <filename>bin</filename> directory.  Look at the files
             that end in <filename>*.rb</filename>.  To run one of these
@@ -1349,14 +1349,31 @@ of all regions.
 
   <chapter xml:id="schema">
   <title>HBase and Schema Design</title>
+  <section xml:id="number.of.cfs">
+  <title>
+      On the number of column families
+  </title>
+  <para>
+      HBase currently does not do well with anything about two or three column families so keep the number
+      of column families in your schema low.  Currently, flushing and compactions are done on a per Region basis so
+      if one column family is carrying the bulk of the data bringing on flushes, the adjacent families
+      will also be flushed though the amount of data they carry is small.  Compaction is currently triggered
+      by the total number of files under a column family.  Its not size based.  When many column families the
+      flushing and compaction interaction can make for a bunch of needless i/o loading (To be addressed by
+      changing flushing and compaction to work on a per column family basis).
+    </para>
+    <para>Try to make do with one column famliy if you can in your schemas.  Only introduce a
+        second and third column family in the case where data access is usually column scoped;
+        i.e. you query one column family or the other but usually not both at the one time.
+    </para>
+  </section>
   <section>
   <title>
   Monotonically Increasing Row Keys/Timeseries Data
   </title>
   <para>See this comic by IKai Lan on why monotically increasing row keys are
   problematic in BigTable-like datastores:
-  <link xlink:href="http://ikaisays.com/2011/01/25/app-engine-datastore-tip-monotonically-increasing-values-are-bad/">monotonically increasing values are bad</link>
-  in BigTable-like stores.</para>
+  <link xlink:href="http://ikaisays.com/2011/01/25/app-engine-datastore-tip-monotonically-increasing-values-are-bad/">monotonically increasing values are bad</link>.</para>
   <para>If you need to upload time series data into HBase, you should
   study <link xlink:href="http://opentsdb.net/">OpenTSDB</link> as a
   successful example.  It has a page describing the schema it uses in
@@ -1434,7 +1451,7 @@ of all regions.
 
       <para></para>
     </section>
-    <section>
+    <section xml:id="cells">
       <title>Cells<indexterm><primary>Cells</primary></indexterm></title>
       <para>A <emphasis>{row, column, version} </emphasis>tuple exactly
       specifies a <literal>cell</literal> in HBase. 
@@ -1493,7 +1510,7 @@ of all regions.
           basically a synopsis of this article by Bruno Dumon.</para>
         </footnote>.</para>
 
-      <section>
+      <section xml:id="versions">
         <title>Versions and HBase Operations</title>
 
         <para>In this section we look at the behavior of the version dimension
@@ -1635,15 +1652,15 @@ of all regions.
 
   <chapter xml:id="architecture">
     <title>Architecture</title>
-    <section>
+    <section xml:id="daemons">
      <title>Daemons</title>
-     <section><title>Master</title>
+     <section xml:id="master"><title>Master</title>
      </section>
-     <section><title>RegionServer</title>
+     <section xml:id="regionserver.arch"><title>RegionServer</title>
      </section>
     </section>
 
-    <section>
+    <section xml:id="regions.arch">
     <title>Regions</title>
     <para>This chapter is all about Regions.</para>
     <note>
@@ -1758,7 +1775,7 @@ of all regions.
     <para>Each RegionServer adds updates to its Write-ahead Log (WAL)
     first, and then to memory.</para>
 
-    <section>
+    <section xml:id="purpose.wal">
       <title>What is the purpose of the HBase WAL</title>
 
       <para>
@@ -1812,6 +1829,36 @@ of all regions.
     </section>
 
   </chapter>
+  <chapter xml:id="performance">
+    <title>Performance Tuning</title>
+    <para>Start with the <link xlink:href="http://wiki.apache.org/hadoop/PerformanceTuning">wiki Performance Tuning</link> page.
+        It has a general discussion of the main factors involved; RAM, compression, JVM settings, etc.
+        Afterward, come back here for more pointers.
+    </para>
+    <section xml:id="jvm">
+        <title>Java</title>
+    <section xml:id="gc">
+        <title>The Garage Collector and HBase</title>
+        <section xml:id="gcpause">
+            <title>Long GC pauses</title>
+        <para>
+            In his presentation,
+            <link xlink:href="http://www.slideshare.net/cloudera/hbase-hug-presentation">Avoiding Full GCs with MemStore-Local Allocation Buffers</link>,
+            Todd Lipcon describes two cases of stop-the-world garbage collections common in HBase, especially during loading;
+            CMS failure modes and old generation heap fragmentation brought.  To address the first,
+            start the CMS earlier than default by adding <code>-XX:CMSInitiatingOccupancyFraction</code>
+            and setting it down from defaults.  Start at 60 or 70 percent (The lower you bring down
+            the threshold, the more GCing is done, the more CPU used).  To address the second
+            fragmentation issue, Todd added an experimental facility that must be 
+            explicitly enabled in HBase 0.90.x (Its defaulted to be on in 0.92.x HBase).  See
+            <code>hbase.hregion.memstore.mslab.enabled</code> to true in your
+            <classname>Configuration</classname>.  See the cited slides for background and
+            detail.
+        </para>
+      </section>
+    </section>
+    </section>
+  </chapter>
 
   <chapter xml:id="blooms">
     <title>Bloom Filters</title>
@@ -1839,7 +1886,7 @@ of all regions.
         work.</para>
       </footnote></para>
 
-    <section>
+    <section xml:id="bloom.config">
       <title>Configurations</title>
 
       <para>Blooms are enabled by specifying options on a column family in the
@@ -1975,7 +2022,7 @@ of all regions.
         doing:<programlisting> $ ./<code>bin/hbase org.apache.hadoop.hbase.regionserver.wal.HLog --split hdfs://example.org:9000/hbase/.logs/example.org,60020,1283516293161/</code></programlisting></para>
       </section>
     </section>
-    <section><title>Compression Tool</title>
+    <section xml:id="compression.tool"><title>Compression Tool</title>
         <para>See <link linkend="compression.tool" >Compression Tool</link>.</para>
     </section>
   </appendix>
@@ -1984,7 +2031,7 @@ of all regions.
 
     <title >Compression In HBase<indexterm><primary>Compression</primary></indexterm></title>
 
-    <section id="compression.test">
+    <section xml:id="compression.test">
     <title>CompressionTest Tool</title>
     <para>
     HBase includes a tool to test compression is set up properly.
@@ -1993,7 +2040,7 @@ of all regions.
     </para>
     </section>
 
-    <section id="hbase.regionserver.codecs">
+    <section xml:id="hbase.regionserver.codecs">
     <title>
     <varname>
     hbase.regionserver.codecs