You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hbase.apache.org by st...@apache.org on 2010/11/02 05:57:26 UTC
svn commit: r1029952 - in /hbase/trunk: CHANGES.txt src/docbkx/book.xml
Author: stack
Date: Tue Nov 2 04:57:25 2010
New Revision: 1029952
URL: http://svn.apache.org/viewvc?rev=1029952&view=rev
Log:
HBASE-1932 Encourage use of 'lzo' compression... add the wiki page to getting started
Modified:
hbase/trunk/CHANGES.txt
hbase/trunk/src/docbkx/book.xml
Modified: hbase/trunk/CHANGES.txt
URL: http://svn.apache.org/viewvc/hbase/trunk/CHANGES.txt?rev=1029952&r1=1029951&r2=1029952&view=diff
==============================================================================
--- hbase/trunk/CHANGES.txt (original)
+++ hbase/trunk/CHANGES.txt Tue Nov 2 04:57:25 2010
@@ -640,6 +640,8 @@ Release 0.21.0 - Unreleased
HBASE-3179 Enable ReplicationLogsCleaner only if replication is,
and fix its test
HBASE-3185 User-triggered compactions are triggering splits!
+ HBASE-1932 Encourage use of 'lzo' compression... add the wiki page to
+ getting started
IMPROVEMENTS
Modified: hbase/trunk/src/docbkx/book.xml
URL: http://svn.apache.org/viewvc/hbase/trunk/src/docbkx/book.xml?rev=1029952&r1=1029951&r2=1029952&view=diff
==============================================================================
--- hbase/trunk/src/docbkx/book.xml (original)
+++ hbase/trunk/src/docbkx/book.xml Tue Nov 2 04:57:25 2010
@@ -230,6 +230,7 @@ stopping hbase...............</programli
different HBase run modes: standalone, what is described above in <link
linkend="quickstart">Quick Start,</link> pseudo-distributed where all
daemons run on a single server, and distributed.</para>
+ <para>Be sure to read the <link linkend="important_configurations">Important Configurations</link>.</para>
</section>
</chapter>
@@ -242,7 +243,6 @@ stopping hbase...............</programli
<title><filename>hbase-site.xml</filename> and <filename>hbase-default.xml</filename></title>
<para>What are these?
</para>
-
<para>
Not all configuration options make it out to
<filename>hbase-default.xml</filename>. Configuration
@@ -250,38 +250,95 @@ stopping hbase...............</programli
in code; the only way to turn up the configurations is
via a reading of the source code.
</para>
-
<!--The file hbase-default.xml is generated as part of
the build of the hbase site. See the hbase pom.xml.
The generated file is a docbook section with a glossary
in it-->
<xi:include xmlns:xi="http://www.w3.org/2001/XInclude"
href="../../target/site/hbase-default.xml" />
- </section>
+ </section>
<section>
<title><filename>hbase-env.sh</filename></title>
<para></para>
</section>
+
<section>
<title><filename>log4j.properties</filename></title>
<para></para>
</section>
- <section>
- <title>Noteworthy Configuration</title>
- <para>Below we review a couple of the key configurations.
- We'll list those you must to change to suit your context
- and others that you should review and consider moving on
- from defaults after guaging your deploys load and query profiles.
+
+ <section xml:id="important_configurations">
+ <title>The Important Configurations</title>
+ <para>Below we list the important Configurations. We've divided this section into
+ required configuration and worth-a-look recommended configs.
</para>
- <section>
+
+
+ <section xml:id="required_configuration"><title>Required Configurations</title>
+ <para>Here are some configurations you must configure to suit
+ your deploy.
+ </para>
+ <section xml:id="ulimit">
+ <title><varname>ulimit</varname></title>
+ <para>HBase is a database, it uses a lot of files at the same time.
+ The default ulimit -n of 1024 on *nix systems is insufficient.
+ Any significant amount of loading will lead you to
+ <link xlink:href="http://wiki.apache.org/hadoop/Hbase/FAQ#A6">FAQ: Why do I see "java.io.IOException...(Too many open files)" in my logs?</link>.
+ You will also notice errors like:
+ <programlisting>2010-04-06 03:04:37,542 INFO org.apache.hadoop.hdfs.DFSClient: Exception increateBlockOutputStream java.io.EOFException
+ 2010-04-06 03:04:37,542 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_-6935524980745310745_1391901
+ </programlisting>
+ Do yourself a favor and change the upper bound on the number of file descriptors.
+ Set it to north of 10k. See the above referenced FAQ for how.</para>
+ <para>To be clear, upping the file descriptors for the user who is
+ running the HBase process is an operating system configuration, not an
+ HBase configuration.
+ </para>
+ </section>
+ <section xml:id="dfs.datanode.max.xcievers">
+ <title><varname>dfs.datanode.max.xcievers</varname></title>
+ <para>
+ Hadoop HDFS has an upper bound of files that it will serve at one same time,
+ called <varname>xcievers</varname> (yes, this is misspelled). Again, before
+ doing any loading, make sure you have configured Hadoop's <filename>conf/hdfs-site.xml</filename>
+ setting the <varname>xceivers</varname> value to at least the following:
+ <programlisting>
+ <property>
+ <name>dfs.datanode.max.xcievers</name>
+ <value>2047</value>
+ </property>
+ </programlisting>
+ </para>
+ </section>
+ </section>
+
+ <section xml:id="recommended_configurations"><title>Recommended Configuations</title>
+ <section xml:id="lzo">
<title>LZO compression</title>
<para>You should consider enabling LZO compression. Its
near-frictionless and in most all cases boosts performance.
- To enable LZO, TODO...
</para>
+ <para>Unfortunately, HBase cannot ship with LZO because of
+ the licensing issues; HBase is Apache-licensed, LZO is GPL.
+ Therefore LZO install is to be done post-HBase install.
+ See the <link xlink:href="http://wiki.apache.org/hadoop/UsingLzoCompression">Using LZO Compression</link>
+ wiki page for how to make LZO work with HBase.
+ </para>
+ <para>A common problem users run into when using LZO is that while initial
+ setup of the cluster runs smooth, a month goes by and some sysadmin goes to
+ add a machine to the cluster only they'll have forgotten to do the LZO
+ fixup on the new machine. In versions since HBase 0.90.0, we should
+ fail in a way that makes it plain what the problem is, but maybe not.
+ Remember you read this paragraph<footnote><para>See
+ <link linkend="hbase.regionserver.codec">hbase.regionserver.codec</link>
+ for a feature to help protect against failed LZO install</para></footnote>.
+ </para>
+ </section>
</section>
+
+
</section>
</chapter>
@@ -1201,7 +1258,7 @@ stopping hbase...............</programli
xlink:href="http://en.wikipedia.org/wiki/Write-ahead_logging"> Write-Ahead
Log</link></subtitle>
- <para>Each RegionServer adds updates to its <link linkend="???">WAL</link>
+ <para>Each RegionServer adds updates to its Write-ahead Log (WAL)
first, and then to memory.</para>
<section>