You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hbase.apache.org by st...@apache.org on 2014/05/28 16:59:09 UTC
[11/14] HBASE-11199 One-time effort to pretty-print the Docbook XML,
to make further patch review easier (Misty Stanley-Jones)
http://git-wip-us.apache.org/repos/asf/hbase/blob/63e8304e/src/main/docbkx/configuration.xml
----------------------------------------------------------------------
diff --git a/src/main/docbkx/configuration.xml b/src/main/docbkx/configuration.xml
index 25e1f13..cd86bef 100644
--- a/src/main/docbkx/configuration.xml
+++ b/src/main/docbkx/configuration.xml
@@ -1,13 +1,15 @@
<?xml version="1.0"?>
- <chapter xml:id="configuration"
- version="5.0" xmlns="http://docbook.org/ns/docbook"
- xmlns:xlink="http://www.w3.org/1999/xlink"
- xmlns:xi="http://www.w3.org/2001/XInclude"
- xmlns:svg="http://www.w3.org/2000/svg"
- xmlns:m="http://www.w3.org/1998/Math/MathML"
- xmlns:html="http://www.w3.org/1999/xhtml"
- xmlns:db="http://docbook.org/ns/docbook">
-<!--
+<chapter
+ xml:id="configuration"
+ version="5.0"
+ xmlns="http://docbook.org/ns/docbook"
+ xmlns:xlink="http://www.w3.org/1999/xlink"
+ xmlns:xi="http://www.w3.org/2001/XInclude"
+ xmlns:svg="http://www.w3.org/2000/svg"
+ xmlns:m="http://www.w3.org/1998/Math/MathML"
+ xmlns:html="http://www.w3.org/1999/xhtml"
+ xmlns:db="http://docbook.org/ns/docbook">
+ <!--
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
@@ -26,243 +28,331 @@
* limitations under the License.
*/
-->
- <title>Apache HBase Configuration</title>
- <para>This chapter is the Not-So-Quick start guide to Apache HBase configuration. It goes
- over system requirements, Hadoop setup, the different Apache HBase run modes, and the
- various configurations in HBase. Please read this chapter carefully. At a minimum
- ensure that all <xref linkend="basic.prerequisites" /> have
- been satisfied. Failure to do so will cause you (and us) grief debugging strange errors
- and/or data loss.</para>
-
- <para>
- Apache HBase uses the same configuration system as Apache Hadoop.
- To configure a deploy, edit a file of environment variables
- in <filename>conf/hbase-env.sh</filename> -- this configuration
- is used mostly by the launcher shell scripts getting the cluster
- off the ground -- and then add configuration to an XML file to
- do things like override HBase defaults, tell HBase what Filesystem to
- use, and the location of the ZooKeeper ensemble
- <footnote>
-<para>
-Be careful editing XML. Make sure you close all elements.
-Run your file through <command>xmllint</command> or similar
-to ensure well-formedness of your document after an edit session.
-</para>
- </footnote>
- .
- </para>
-
- <para>When running in distributed mode, after you make
- an edit to an HBase configuration, make sure you copy the
- content of the <filename>conf</filename> directory to
- all nodes of the cluster. HBase will not do this for you.
- Use <command>rsync</command>. For most configuration, a restart is
-needed for servers to pick up changes (caveat dynamic config. to be described later below).</para>
-
- <section xml:id="basic.prerequisites">
+ <title>Apache HBase Configuration</title>
+ <para>This chapter is the Not-So-Quick start guide to Apache HBase configuration. It goes over
+ system requirements, Hadoop setup, the different Apache HBase run modes, and the various
+ configurations in HBase. Please read this chapter carefully. At a minimum ensure that all <xref
+ linkend="basic.prerequisites" /> have been satisfied. Failure to do so will cause you (and us)
+ grief debugging strange errors and/or data loss.</para>
+
+ <para> Apache HBase uses the same configuration system as Apache Hadoop. To configure a deploy,
+ edit a file of environment variables in <filename>conf/hbase-env.sh</filename> -- this
+ configuration is used mostly by the launcher shell scripts getting the cluster off the ground --
+ and then add configuration to an XML file to do things like override HBase defaults, tell HBase
+ what Filesystem to use, and the location of the ZooKeeper ensemble. <footnote>
+ <para> Be careful editing XML. Make sure you close all elements. Run your file through
+ <command>xmllint</command> or similar to ensure well-formedness of your document after an
+ edit session. </para>
+ </footnote></para>
+
+ <para>When running in distributed mode, after you make an edit to an HBase configuration, make
+ sure you copy the content of the <filename>conf</filename> directory to all nodes of the
+ cluster. HBase will not do this for you. Use <command>rsync</command>. For most configuration, a
+ restart is needed for servers to pick up changes (caveat dynamic config. to be described later
+ below).</para>
+
+ <section
+ xml:id="basic.prerequisites">
<title>Basic Prerequisites</title>
- <para>This section lists required services and some required system configuration.
- </para>
+ <para>This section lists required services and some required system configuration. </para>
- <section xml:id="java">
- <title>Java</title>
- <para>Just like Hadoop, HBase requires at least Java 6 from
- <link xlink:href="http://www.java.com/download/">Oracle</link>.
- </para>
+ <section
+ xml:id="java">
+ <title>Java</title>
+ <para>Just like Hadoop, HBase requires at least Java 6 from <link
+ xlink:href="http://www.java.com/download/">Oracle</link>. </para>
</section>
- <section xml:id="os">
- <title>Operating System</title>
- <section xml:id="ssh">
+ <section
+ xml:id="os">
+ <title>Operating System</title>
+ <section
+ xml:id="ssh">
<title>ssh</title>
- <para><command>ssh</command> must be installed and
- <command>sshd</command> must be running to use Hadoop's scripts to
- manage remote Hadoop and HBase daemons. You must be able to ssh to all
- nodes, including your local node, using passwordless login (Google
- "ssh passwordless login"). If on mac osx, see the section,
- <link xlink:href="http://wiki.apache.org/hadoop/Running_Hadoop_On_OS_X_10.5_64-bit_%28Single-Node_Cluster%29">SSH: Setting up Remote Desktop and Enabling Self-Login</link>
- on the hadoop wiki.</para>
+ <para><command>ssh</command> must be installed and <command>sshd</command> must be running
+ to use Hadoop's scripts to manage remote Hadoop and HBase daemons. You must be able to ssh
+ to all nodes, including your local node, using passwordless login (Google "ssh
+ passwordless login"). If on mac osx, see the section, <link
+ xlink:href="http://wiki.apache.org/hadoop/Running_Hadoop_On_OS_X_10.5_64-bit_%28Single-Node_Cluster%29">SSH:
+ Setting up Remote Desktop and Enabling Self-Login</link> on the hadoop wiki.</para>
</section>
- <section xml:id="dns">
+ <section
+ xml:id="dns">
<title>DNS</title>
- <para>HBase uses the local hostname to self-report its IP address.
- Both forward and reverse DNS resolving must work in versions of
- HBase previous to 0.92.0
- <footnote><para>The <link xlink:href="https://github.com/sujee/hadoop-dns-checker">hadoop-dns-checker</link> tool can be used to verify
- DNS is working correctly on the cluster. The project README file provides detailed instructions on usage.
-</para></footnote>.</para>
+ <para>HBase uses the local hostname to self-report its IP address. Both forward and reverse
+ DNS resolving must work in versions of HBase previous to 0.92.0 <footnote>
+ <para>The <link
+ xlink:href="https://github.com/sujee/hadoop-dns-checker">hadoop-dns-checker</link>
+ tool can be used to verify DNS is working correctly on the cluster. The project README
+ file provides detailed instructions on usage. </para>
+ </footnote>.</para>
- <para>If your machine has multiple interfaces, HBase will use the
- interface that the primary hostname resolves to.</para>
+ <para>If your machine has multiple interfaces, HBase will use the interface that the primary
+ hostname resolves to.</para>
<para>If this is insufficient, you can set
- <varname>hbase.regionserver.dns.interface</varname> to indicate the
- primary interface. This only works if your cluster configuration is
- consistent and every host has the same network interface
- configuration.</para>
-
- <para>Another alternative is setting
- <varname>hbase.regionserver.dns.nameserver</varname> to choose a
- different nameserver than the system wide default.</para>
+ <varname>hbase.regionserver.dns.interface</varname> to indicate the primary interface.
+ This only works if your cluster configuration is consistent and every host has the same
+ network interface configuration.</para>
+
+ <para>Another alternative is setting <varname>hbase.regionserver.dns.nameserver</varname> to
+ choose a different nameserver than the system wide default.</para>
</section>
- <section xml:id="loopback.ip">
+ <section
+ xml:id="loopback.ip">
<title>Loopback IP</title>
- <para>Previous to hbase-0.96.0, HBase expects the loopback IP address to be 127.0.0.1. See <xref linkend="loopback.ip"/></para>
- </section>
+ <para>Previous to hbase-0.96.0, HBase expects the loopback IP address to be 127.0.0.1. See <xref
+ linkend="loopback.ip" /></para>
+ </section>
- <section xml:id="ntp">
+ <section
+ xml:id="ntp">
<title>NTP</title>
- <para>The clocks on cluster members should be in basic alignments.
- Some skew is tolerable but wild skew could generate odd behaviors. Run
- <link
- xlink:href="http://en.wikipedia.org/wiki/Network_Time_Protocol">NTP</link>
- on your cluster, or an equivalent.</para>
+ <para>The clocks on cluster members should be in basic alignments. Some skew is tolerable
+ but wild skew could generate odd behaviors. Run <link
+ xlink:href="http://en.wikipedia.org/wiki/Network_Time_Protocol">NTP</link> on your
+ cluster, or an equivalent.</para>
- <para>If you are having problems querying data, or "weird" cluster
- operations, check system time!</para>
+ <para>If you are having problems querying data, or "weird" cluster operations, check system
+ time!</para>
</section>
- <section xml:id="ulimit">
+ <section
+ xml:id="ulimit">
<title>
<varname>ulimit</varname><indexterm>
<primary>ulimit</primary>
- </indexterm>
- and
- <varname>nproc</varname><indexterm>
+ </indexterm> and <varname>nproc</varname><indexterm>
<primary>nproc</primary>
- </indexterm>
+ </indexterm>
</title>
- <para>Apache HBase is a database. It uses a lot of files all at the same time.
- The default ulimit -n -- i.e. user file limit -- of 1024 on most *nix systems
- is insufficient (On mac os x its 256). Any significant amount of loading will
- lead you to <xref linkend="trouble.rs.runtime.filehandles"/>.
- You may also notice errors such as... <programlisting>
- 2010-04-06 03:04:37,542 INFO org.apache.hadoop.hdfs.DFSClient: Exception increateBlockOutputStream java.io.EOFException
- 2010-04-06 03:04:37,542 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_-6935524980745310745_1391901
- </programlisting> Do yourself a favor and change the upper bound on the
- number of file descriptors. Set it to north of 10k. The math runs roughly as follows: per ColumnFamily
- there is at least one StoreFile and possibly up to 5 or 6 if the region is under load. Multiply the
- average number of StoreFiles per ColumnFamily times the number of regions per RegionServer. For example, assuming
- that a schema had 3 ColumnFamilies per region with an average of 3 StoreFiles per ColumnFamily,
- and there are 100 regions per RegionServer, the JVM will open 3 * 3 * 100 = 900 file descriptors
- (not counting open jar files, config files, etc.)
- </para>
- <para>You should also up the hbase users'
- <varname>nproc</varname> setting; under load, a low-nproc
- setting could manifest as <classname>OutOfMemoryError</classname>
- <footnote><para>See Jack Levin's <link xlink:href="">major hdfs issues</link>
- note up on the user list.</para></footnote>
- <footnote><para>The requirement that a database requires upping of system limits
- is not peculiar to Apache HBase. See for example the section
- <emphasis>Setting Shell Limits for the Oracle User</emphasis> in
- <link xlink:href="http://www.akadia.com/services/ora_linux_install_10g.html">
- Short Guide to install Oracle 10 on Linux</link>.</para></footnote>.
- </para>
-
- <para>To be clear, upping the file descriptors and nproc for the user who is
- running the HBase process is an operating system configuration, not an
- HBase configuration. Also, a common mistake is that administrators
- will up the file descriptors for a particular user but for whatever
- reason, HBase will be running as some one else. HBase prints in its
- logs as the first line the ulimit its seeing. Ensure its correct.
- <footnote>
- <para>A useful read setting config on you hadoop cluster is Aaron
- Kimballs' <link
- xlink:href="http://www.cloudera.com/blog/2009/03/configuration-parameters-what-can-you-just-ignore/">Configuration
- Parameters: What can you just ignore?</link></para>
+ <para>Apache HBase is a database. It uses a lot of files all at the same time. The default
+ ulimit -n -- i.e. user file limit -- of 1024 on most *nix systems is insufficient (On mac
+ os x its 256). Any significant amount of loading will lead you to <xref
+ linkend="trouble.rs.runtime.filehandles" />. You may also notice errors such as the
+ following:</para>
+ <screen>
+2010-04-06 03:04:37,542 INFO org.apache.hadoop.hdfs.DFSClient: Exception increateBlockOutputStream java.io.EOFException
+2010-04-06 03:04:37,542 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_-6935524980745310745_1391901
+ </screen>
+ <para> Do yourself a favor and change the upper bound on the number of file descriptors. Set
+ it to north of 10k. The math runs roughly as follows: per ColumnFamily there is at least
+ one StoreFile and possibly up to 5 or 6 if the region is under load. Multiply the average
+ number of StoreFiles per ColumnFamily times the number of regions per RegionServer. For
+ example, assuming that a schema had 3 ColumnFamilies per region with an average of 3
+ StoreFiles per ColumnFamily, and there are 100 regions per RegionServer, the JVM will open
+ 3 * 3 * 100 = 900 file descriptors (not counting open jar files, config files, etc.) </para>
+ <para>You should also up the hbase users' <varname>nproc</varname> setting; under load, a
+ low-nproc setting could manifest as <classname>OutOfMemoryError</classname>. <footnote>
+ <para>See Jack Levin's <link
+ xlink:href="">major hdfs issues</link> note up on the user list.</para>
+ </footnote>
+ <footnote>
+ <para>The requirement that a database requires upping of system limits is not peculiar
+ to Apache HBase. See for example the section <emphasis>Setting Shell Limits for the
+ Oracle User</emphasis> in <link
+ xlink:href="http://www.akadia.com/services/ora_linux_install_10g.html"> Short Guide
+ to install Oracle 10 on Linux</link>.</para>
</footnote></para>
- <section xml:id="ulimit_ubuntu">
- <title><varname>ulimit</varname> on Ubuntu</title>
+ <para>To be clear, upping the file descriptors and nproc for the user who is running the
+ HBase process is an operating system configuration, not an HBase configuration. Also, a
+ common mistake is that administrators will up the file descriptors for a particular user
+ but for whatever reason, HBase will be running as some one else. HBase prints in its logs
+ as the first line the ulimit its seeing. Ensure its correct. <footnote>
+ <para>A useful read setting config on you hadoop cluster is Aaron Kimballs' <link
+ xlink:href="http://www.cloudera.com/blog/2009/03/configuration-parameters-what-can-you-just-ignore/">Configuration
+ Parameters: What can you just ignore?</link></para>
+ </footnote></para>
- <para>If you are on Ubuntu you will need to make the following
- changes:</para>
+ <section
+ xml:id="ulimit_ubuntu">
+ <title><varname>ulimit</varname> on Ubuntu</title>
- <para>In the file <filename>/etc/security/limits.conf</filename> add
- a line like: <programlisting>hadoop - nofile 32768</programlisting>
- Replace <varname>hadoop</varname> with whatever user is running
- Hadoop and HBase. If you have separate users, you will need 2
- entries, one for each user. In the same file set nproc hard and soft
- limits. For example: <programlisting>hadoop soft/hard nproc 32000</programlisting>.</para>
+ <para>If you are on Ubuntu you will need to make the following changes:</para>
- <para>In the file <filename>/etc/pam.d/common-session</filename> add
- as the last line in the file: <programlisting>session required pam_limits.so</programlisting>
- Otherwise the changes in <filename>/etc/security/limits.conf</filename> won't be
- applied.</para>
+ <para>In the file <filename>/etc/security/limits.conf</filename> add a line like:</para>
+ <programlisting>hadoop - nofile 32768</programlisting>
+ <para>Replace <varname>hadoop</varname> with whatever user is running Hadoop and HBase. If
+ you have separate users, you will need 2 entries, one for each user. In the same file
+ set nproc hard and soft limits. For example:</para>
+ <programlisting>hadoop soft/hard nproc 32000</programlisting>
+ <para>In the file <filename>/etc/pam.d/common-session</filename> add as the last line in
+ the file: <programlisting>session required pam_limits.so</programlisting> Otherwise the
+ changes in <filename>/etc/security/limits.conf</filename> won't be applied.</para>
- <para>Don't forget to log out and back in again for the changes to
- take effect!</para>
+ <para>Don't forget to log out and back in again for the changes to take effect!</para>
</section>
</section>
- <section xml:id="windows">
+ <section
+ xml:id="windows">
<title>Windows</title>
<para>Previous to hbase-0.96.0, Apache HBase was little tested running on Windows. Running a
- production install of HBase on top of Windows is not recommended.</para>
+ production install of HBase on top of Windows is not recommended.</para>
<para>If you are running HBase on Windows pre-hbase-0.96.0, you must install <link
- xlink:href="http://cygwin.com/">Cygwin</link> to have a *nix-like
- environment for the shell scripts. The full details are explained in
- the <link xlink:href="http://hbase.apache.org/cygwin.html">Windows
- Installation</link> guide. Also
- <link xlink:href="http://search-hadoop.com/?q=hbase+windows&fc_project=HBase&fc_type=mail+_hash_+dev">search our user mailing list</link> to pick
- up latest fixes figured by Windows users.</para>
- <para>Post-hbase-0.96.0, hbase runs natively on windows with supporting <command>*.cmd</command> scripts bundled.
- </para>
+ xlink:href="http://cygwin.com/">Cygwin</link> to have a *nix-like environment for the
+ shell scripts. The full details are explained in the <link
+ xlink:href="http://hbase.apache.org/cygwin.html">Windows Installation</link> guide. Also <link
+ xlink:href="http://search-hadoop.com/?q=hbase+windows&fc_project=HBase&fc_type=mail+_hash_+dev">search
+ our user mailing list</link> to pick up latest fixes figured by Windows users.</para>
+ <para>Post-hbase-0.96.0, hbase runs natively on windows with supporting
+ <command>*.cmd</command> scripts bundled. </para>
</section>
- </section> <!-- OS -->
-
- <section xml:id="hadoop">
- <title><link
- xlink:href="http://hadoop.apache.org">Hadoop</link><indexterm>
- <primary>Hadoop</primary>
- </indexterm></title>
- <para>The below table shows some information about what versions of Hadoop are supported by various HBase versions.
- Based on the version of HBase, you should select the most appropriate version of Hadoop.
- We are not in the Hadoop distro selection business.
- You can use Hadoop distributions from Apache, or learn about vendor distributions of
- Hadoop at <link xlink:href="http://wiki.apache.org/hadoop/Distributions%20and%20Commercial%20Support"/></para>
- <para>
- <tip><title>Hadoop 2.x is better than Hadoop 1.x</title>
- <para>Hadoop 2.x is faster, with more features such as short-circuit reads which will help improve your
- HBase random read profile as well important bug fixes that will improve your overall HBase experience.
- You should run Hadoop 2 rather than Hadoop 1. HBase 0.98 deprecates use of Hadoop1.
- HBase 1.0 will not support Hadoop1.
- </para>
- </tip>
- <table>
- <title>Hadoop version support matrix</title>
- <tgroup cols='6' align='left' colsep='1' rowsep='1'>
- <colspec colname='c1' align='left'/>
- <colspec colname='c2' align='center'/>
- <colspec colname='c3' align='center'/>
- <colspec colname='c4' align='center'/>
- <colspec colname='c5' align='center'/>
- <colspec colname='c6' align='center'/>
- <thead>
- <row><entry> </entry><entry>HBase-0.92.x</entry><entry>HBase-0.94.x</entry><entry>HBase-0.96.x</entry><entry>HBase-0.98.x<footnote><para>Support for Hadoop 1.x is deprecated.</para></footnote></entry><entry>HBase-1.0.x<footnote><para>Hadoop 1.x is NOT supported</para></footnote></entry></row>
- </thead><tbody>
- <row><entry>Hadoop-0.20.205</entry><entry>S</entry> <entry>X</entry> <entry>X</entry><entry>X</entry><entry>X</entry></row>
- <row><entry>Hadoop-0.22.x </entry><entry>S</entry> <entry>X</entry> <entry>X</entry><entry>X</entry><entry>X</entry></row>
- <row><entry>Hadoop-1.0.0-1.0.2<footnote><para>HBase requires hadoop 1.0.3 at a minimum; there is an issue where we cannot find KerberosUtil compiling against earlier versions of Hadoop.</para></footnote> </entry><entry>X</entry> <entry>X</entry> <entry>X</entry><entry>X</entry><entry>X</entry></row>
- <row><entry>Hadoop-1.0.3+</entry><entry>S</entry> <entry>S</entry> <entry>S</entry><entry>X</entry><entry>X</entry></row>
- <row><entry>Hadoop-1.1.x </entry><entry>NT</entry> <entry>S</entry> <entry>S</entry><entry>X</entry><entry>X</entry></row>
- <row><entry>Hadoop-0.23.x </entry><entry>X</entry> <entry>S</entry> <entry>NT</entry><entry>X</entry><entry>X</entry></row>
- <row><entry>Hadoop-2.0.x-alpha </entry><entry>X</entry> <entry>NT</entry> <entry>X</entry><entry>X</entry><entry>X</entry></row>
- <row><entry>Hadoop-2.1.0-beta </entry><entry>X</entry> <entry>NT</entry> <entry>S</entry><entry>X</entry><entry>X</entry></row>
- <row><entry>Hadoop-2.2.0 </entry><entry>X</entry> <entry>NT<footnote><para>To get 0.94.x to run on hadoop 2.2.0,
- you need to change the hadoop 2 and protobuf versions in the <filename>pom.xml</filename>:
- <para>
- Here is a diff with pom.xml changes:
- </para>
- <programlisting><![CDATA[$ svn diff pom.xml
+ </section>
+ <!-- OS -->
+
+ <section
+ xml:id="hadoop">
+ <title><link
+ xlink:href="http://hadoop.apache.org">Hadoop</link><indexterm>
+ <primary>Hadoop</primary>
+ </indexterm></title>
+ <para>The below table shows some information about what versions of Hadoop are supported by
+ various HBase versions. Based on the version of HBase, you should select the most
+ appropriate version of Hadoop. We are not in the Hadoop distro selection business. You can
+ use Hadoop distributions from Apache, or learn about vendor distributions of Hadoop at <link
+ xlink:href="http://wiki.apache.org/hadoop/Distributions%20and%20Commercial%20Support" /></para>
+ <tip>
+ <title>Hadoop 2.x is better than Hadoop 1.x</title>
+ <para>Hadoop 2.x is faster, with more features such as short-circuit reads which will help
+ improve your HBase random read profile as well important bug fixes that will improve your
+ overall HBase experience. You should run Hadoop 2 rather than Hadoop 1. HBase 0.98
+ deprecates use of Hadoop1. HBase 1.0 will not support Hadoop1. </para>
+ </tip>
+ <para>Use the following legend to interpret this table:</para>
+ <simplelist
+ type="vert"
+ columns="1">
+ <member>S = supported and tested,</member>
+ <member>X = not supported,</member>
+ <member>NT = it should run, but not tested enough.</member>
+ </simplelist>
+
+ <table>
+ <title>Hadoop version support matrix</title>
+ <tgroup
+ cols="6"
+ align="left"
+ colsep="1"
+ rowsep="1">
+ <colspec
+ colname="c1"
+ align="left" />
+ <colspec
+ colname="c2"
+ align="center" />
+ <colspec
+ colname="c3"
+ align="center" />
+ <colspec
+ colname="c4"
+ align="center" />
+ <colspec
+ colname="c5"
+ align="center" />
+ <colspec
+ colname="c6"
+ align="center" />
+ <thead>
+ <row>
+ <entry> </entry>
+ <entry>HBase-0.92.x</entry>
+ <entry>HBase-0.94.x</entry>
+ <entry>HBase-0.96.x</entry>
+ <entry>HBase-0.98.x<footnote>
+ <para>Support for Hadoop 1.x is deprecated.</para>
+ </footnote></entry>
+ <entry>HBase-1.0.x<footnote>
+ <para>Hadoop 1.x is NOT supported</para>
+ </footnote></entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>Hadoop-0.20.205</entry>
+ <entry>S</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry>Hadoop-0.22.x </entry>
+ <entry>S</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry>Hadoop-1.0.0-1.0.2<footnote>
+ <para>HBase requires hadoop 1.0.3 at a minimum; there is an issue where we cannot
+ find KerberosUtil compiling against earlier versions of Hadoop.</para>
+ </footnote>
+ </entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry>Hadoop-1.0.3+</entry>
+ <entry>S</entry>
+ <entry>S</entry>
+ <entry>S</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry>Hadoop-1.1.x </entry>
+ <entry>NT</entry>
+ <entry>S</entry>
+ <entry>S</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry>Hadoop-0.23.x </entry>
+ <entry>X</entry>
+ <entry>S</entry>
+ <entry>NT</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry>Hadoop-2.0.x-alpha </entry>
+ <entry>X</entry>
+ <entry>NT</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry>Hadoop-2.1.0-beta </entry>
+ <entry>X</entry>
+ <entry>NT</entry>
+ <entry>S</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry>Hadoop-2.2.0 </entry>
+ <entry>X</entry>
+ <entry>NT <footnote>
+ <para>To get 0.94.x to run on hadoop 2.2.0, you need to change the hadoop 2 and
+ protobuf versions in the <filename>pom.xml</filename>: Here is a diff with
+ pom.xml changes: </para>
+ <programlisting><![CDATA[$ svn diff pom.xml
Index: pom.xml
===================================================================
--- pom.xml (revision 1545157)
@@ -284,235 +374,266 @@ Index: pom.xml
+ <hadoop.version>2.2.0</hadoop.version>
<slf4j.version>1.6.1</slf4j.version>
</properties>
- <dependencies>]]></programlisting>
- <para>The next step is to regenerate Protobuf files and assuming that the Protobuf has been installed:</para>
- <itemizedlist>
- <listitem>Go to the hbase root folder, using the command line;</listitem>
- <listitem>
- <para>Type the following commands:</para>
- <para>
- <programlisting><![CDATA[$ protoc -Isrc/main/protobuf --java_out=src/main/java src/main/protobuf/hbase.proto]]></programlisting>
- </para>
- <para>
- <programlisting><![CDATA[$ protoc -Isrc/main/protobuf --java_out=src/main/java src/main/protobuf/ErrorHandling.proto]]></programlisting>
- </para>
- </listitem>
- </itemizedlist>
- <para>
- Building against the hadoop 2 profile by running something like the following command:
- </para>
- <programlisting>$ mvn clean install assembly:single -Dhadoop.profile=2.0 -DskipTests</programlisting>
- </para></footnote></entry> <entry>S</entry><entry>S</entry><entry>NT</entry></row>
- <row><entry>Hadoop-2.3.x</entry><entry>X</entry> <entry>NT</entry> <entry>S</entry><entry>S</entry><entry>NT</entry></row>
- <row><entry>Hadoop-2.4.x</entry><entry>X</entry> <entry>NT</entry> <entry>S</entry><entry>S</entry><entry>S</entry></row>
- <row><entry>Hadoop-2.5.x</entry><entry>X</entry> <entry>NT</entry> <entry>S</entry><entry>S</entry><entry>S</entry></row>
-
- </tbody></tgroup></table>
-
- Where
- <simplelist type='vert' columns='1'>
- <member>S = supported and tested,</member>
- <member>X = not supported,</member>
- <member>NT = it should run, but not tested enough.</member>
- </simplelist>
+ <dependencies>]]>
+ </programlisting>
+ <para>The next step is to regenerate Protobuf files and assuming that the Protobuf
+ has been installed:</para>
+ <itemizedlist>
+ <listitem>
+ <para>Go to the hbase root folder, using the command line;</para>
+ </listitem>
+ <listitem>
+ <para>Type the following commands:</para>
+ <para>
+ <programlisting><![CDATA[$ protoc -Isrc/main/protobuf --java_out=src/main/java src/main/protobuf/hbase.proto]]></programlisting>
+ </para>
+ <para>
+ <programlisting><![CDATA[$ protoc -Isrc/main/protobuf --java_out=src/main/java src/main/protobuf/ErrorHandling.proto]]></programlisting>
+ </para>
+ </listitem>
+ </itemizedlist>
+ <para> Building against the hadoop 2 profile by running something like the
+ following command: </para>
+ <screen>$ mvn clean install assembly:single -Dhadoop.profile=2.0 -DskipTests</screen>
+ </footnote></entry>
+ <entry>S</entry>
+ <entry>S</entry>
+ <entry>NT</entry>
+ </row>
+ <row>
+ <entry>Hadoop-2.3.x</entry>
+ <entry>X</entry>
+ <entry>NT</entry>
+ <entry>S</entry>
+ <entry>S</entry>
+ <entry>NT</entry>
+ </row>
+ <row>
+ <entry>Hadoop-2.4.x</entry>
+ <entry>X</entry>
+ <entry>NT</entry>
+ <entry>S</entry>
+ <entry>S</entry>
+ <entry>S</entry>
+ </row>
+ <row>
+ <entry>Hadoop-2.5.x</entry>
+ <entry>X</entry>
+ <entry>NT</entry>
+ <entry>S</entry>
+ <entry>S</entry>
+ <entry>S</entry>
+ </row>
+
+ </tbody>
+ </tgroup>
+ </table>
+
+ <note
+ xml:id="replace.hadoop">
+ <title>Replace the Hadoop Bundled With HBase!</title>
+ <para> Because HBase depends on Hadoop, it bundles an instance of the Hadoop jar under its
+ <filename>lib</filename> directory. The bundled jar is ONLY for use in standalone mode.
+ In distributed mode, it is <emphasis>critical</emphasis> that the version of Hadoop that
+ is out on your cluster match what is under HBase. Replace the hadoop jar found in the
+ HBase lib directory with the hadoop jar you are running on your cluster to avoid version
+ mismatch issues. Make sure you replace the jar in HBase everywhere on your cluster. Hadoop
+ version mismatch issues have various manifestations but often all looks like its hung up.
</para>
- <note xml:id="replace.hadoop"><title>Replace the Hadoop Bundled With HBase!</title>
- <para>
- Because HBase depends on Hadoop, it bundles an instance of the Hadoop jar under its <filename>lib</filename> directory. The bundled jar is ONLY for use in standalone mode. In distributed mode, it is <emphasis>critical</emphasis> that the version of Hadoop that is out on your cluster match what is under HBase. Replace the hadoop jar found in the HBase lib directory with the hadoop jar you are running on your cluster to avoid version mismatch issues. Make sure you replace the jar in HBase everywhere on your cluster. Hadoop version mismatch issues have various manifestations but often all looks like its hung up.
- </para>
-</note>
- <section xml:id="hadoop.hbase-0.94">
- <title>Apache HBase 0.92 and 0.94</title>
- <para>HBase 0.92 and 0.94 versions can work with Hadoop versions, 0.20.205, 0.22.x, 1.0.x, and 1.1.x. HBase-0.94 can additionally work with Hadoop-0.23.x and 2.x, but you may have to recompile the code using the specific maven profile (see top level pom.xml)</para>
- </section>
-
- <section xml:id="hadoop.hbase-0.96">
- <title>Apache HBase 0.96</title>
- <para>
- As of Apache HBase 0.96.x, Apache Hadoop 1.0.x at least is required. Hadoop 2 is strongly encouraged (faster but also has fixes that help MTTR).
- We will no longer run properly on older Hadoops such as 0.20.205 or branch-0.20-append. Do not move to Apache HBase 0.96.x if you cannot upgrade your Hadoop<footnote><para>See <link xlink:href="http://search-hadoop.com/m/7vFVx4EsUb2">HBase, mail # dev - DISCUSS: Have hbase require at least hadoop 1.0.0 in hbase 0.96.0?</link></para></footnote>.</para>
- </section>
-
- <section xml:id="hadoop.older.versions">
- <title>Hadoop versions 0.20.x - 1.x</title>
- <para>
- HBase will lose data unless it is running on an HDFS that has a durable
- <code>sync</code> implementation. DO NOT use Hadoop 0.20.2, Hadoop 0.20.203.0, and Hadoop 0.20.204.0 which DO NOT have this attribute. Currently only Hadoop versions 0.20.205.x or any release in excess of this version -- this includes hadoop-1.0.0 -- have a working, durable sync
- <footnote>
- <para>The Cloudera blog post <link xlink:href="http://www.cloudera.com/blog/2012/01/an-update-on-apache-hadoop-1-0/">An update on Apache Hadoop 1.0</link>
- by Charles Zedlweski has a nice exposition on how all the Hadoop versions relate.
- Its worth checking out if you are having trouble making sense of the
- Hadoop version morass.
- </para>
- </footnote>. Sync has to be explicitly enabled by setting
- <varname>dfs.support.append</varname> equal
- to true on both the client side -- in <filename>hbase-site.xml</filename>
- -- and on the serverside in <filename>hdfs-site.xml</filename> (The sync
- facility HBase needs is a subset of the append code path).
- <programlisting>
- <property>
- <name>dfs.support.append</name>
- <value>true</value>
- </property>
- </programlisting>
- You will have to restart your cluster after making this edit. Ignore the chicken-little
- comment you'll find in the <filename>hdfs-default.xml</filename> in the
- description for the <varname>dfs.support.append</varname> configuration.
- </para>
- </section>
- <section xml:id="hadoop.security">
- <title>Apache HBase on Secure Hadoop</title>
- <para>Apache HBase will run on any Hadoop 0.20.x that incorporates Hadoop
- security features as long as you do as
- suggested above and replace the Hadoop jar that ships with HBase
- with the secure version. If you want to read more about how to setup
- Secure HBase, see <xref linkend="hbase.secure.configuration" />.</para>
- </section>
-
- <section xml:id="dfs.datanode.max.transfer.threads">
- <title><varname>dfs.datanode.max.transfer.threads</varname><indexterm>
- <primary>xcievers</primary>
- </indexterm></title>
-
- <para>An Hadoop HDFS datanode has an upper bound on the number of
- files that it will serve at any one time. The upper bound parameter is
- called <varname>xcievers</varname> (yes, this is misspelled). Again,
- before doing any loading, make sure you have configured Hadoop's
- <filename>conf/hdfs-site.xml</filename> setting the
- <varname>xceivers</varname> value to at least the following:
- <programlisting>
- <property>
- <name>dfs.datanode.max.transfer.threads</name>
- <value>4096</value>
- </property>
- </programlisting></para>
-
- <para>Be sure to restart your HDFS after making the above
- configuration.</para>
-
- <para>Not having this configuration in place makes for strange looking
- failures. Eventually you'll see a complain in the datanode logs
- complaining about the xcievers exceeded, but on the run up to this one
- manifestation is complaint about missing blocks. For example:
- <code>10/12/08 20:10:31 INFO hdfs.DFSClient: Could not obtain block
- blk_XXXXXXXXXXXXXXXXXXXXXX_YYYYYYYY from any node:
- java.io.IOException: No live nodes contain current block. Will get new
- block locations from namenode and retry...</code>
- <footnote><para>See <link xlink:href="http://ccgtech.blogspot.com/2010/02/hadoop-hdfs-deceived-by-xciever.html">Hadoop HDFS: Deceived by Xciever</link> for an informative rant on xceivering.</para></footnote></para>
- <para>See also <xref linkend="casestudies.xceivers"/>
- </para>
+ </note>
+ <section
+ xml:id="hadoop.hbase-0.94">
+ <title>Apache HBase 0.92 and 0.94</title>
+ <para>HBase 0.92 and 0.94 versions can work with Hadoop versions, 0.20.205, 0.22.x, 1.0.x,
+ and 1.1.x. HBase-0.94 can additionally work with Hadoop-0.23.x and 2.x, but you may have
+ to recompile the code using the specific maven profile (see top level pom.xml)</para>
</section>
- </section> <!-- hadoop -->
- </section>
-
- <section xml:id="standalone_dist">
- <title>HBase run modes: Standalone and Distributed</title>
-
- <para>HBase has two run modes: <xref linkend="standalone" /> and <xref linkend="distributed" />. Out of the box, HBase runs in
- standalone mode. Whatever your mode, you will need to configure HBase by editing files in the HBase <filename>conf</filename>
- directory. At a minimum, you must edit <code>conf/hbase-env.sh</code> to tell HBase which
- <command>java</command> to use. In this file you set HBase environment
- variables such as the heapsize and other options for the
- <application>JVM</application>, the preferred location for log files,
- etc. Set <varname>JAVA_HOME</varname> to point at the root of your
- <command>java</command> install.</para>
-
- <section xml:id="standalone">
- <title>Standalone HBase</title>
-
- <para>This is the default mode. Standalone mode is what is described
- in the <xref linkend="quickstart" /> section. In
- standalone mode, HBase does not use HDFS -- it uses the local
- filesystem instead -- and it runs all HBase daemons and a local
- ZooKeeper all up in the same JVM. Zookeeper binds to a well known port
- so clients may talk to HBase.</para>
+ <section
+ xml:id="hadoop.hbase-0.96">
+ <title>Apache HBase 0.96</title>
+ <para> As of Apache HBase 0.96.x, Apache Hadoop 1.0.x at least is required. Hadoop 2 is
+ strongly encouraged (faster but also has fixes that help MTTR). We will no longer run
+ properly on older Hadoops such as 0.20.205 or branch-0.20-append. Do not move to Apache
+ HBase 0.96.x if you cannot upgrade your Hadoop.<footnote>
+ <para>See <link
+ xlink:href="http://search-hadoop.com/m/7vFVx4EsUb2">HBase, mail # dev - DISCUSS:
+ Have hbase require at least hadoop 1.0.0 in hbase 0.96.0?</link></para>
+ </footnote></para>
</section>
- <section xml:id="distributed">
- <title>Distributed</title>
-
- <para>Distributed mode can be subdivided into distributed but all
- daemons run on a single node -- a.k.a
- <emphasis>pseudo-distributed</emphasis>-- and
- <emphasis>fully-distributed</emphasis> where the daemons are spread
- across all nodes in the cluster <footnote>
- <para>The pseudo-distributed vs fully-distributed nomenclature
- comes from Hadoop.</para>
- </footnote>.</para>
-
- <para>Pseudo-distributed mode can run against the local filesystem or
- it can run against an instance of the <emphasis>Hadoop
- Distributed File System</emphasis> (HDFS). Fully-distributed mode can
- ONLY run on HDFS. See the Hadoop <link
- xlink:href="http://hadoop.apache.org/common/docs/r1.1.1/api/overview-summary.html#overview_description">
- requirements and instructions</link> for how to set up HDFS.</para>
-
- <para>Below we describe the different distributed setups. Starting,
- verification and exploration of your install, whether a
- <emphasis>pseudo-distributed</emphasis> or
- <emphasis>fully-distributed</emphasis> configuration is described in a
- section that follows, <xref linkend="confirm" />. The same verification script applies to both
- deploy types.</para>
-
-
-
+ <section
+ xml:id="hadoop.older.versions">
+ <title>Hadoop versions 0.20.x - 1.x</title>
+ <para> HBase will lose data unless it is running on an HDFS that has a durable
+ <code>sync</code> implementation. DO NOT use Hadoop 0.20.2, Hadoop 0.20.203.0, and
+ Hadoop 0.20.204.0 which DO NOT have this attribute. Currently only Hadoop versions
+ 0.20.205.x or any release in excess of this version -- this includes hadoop-1.0.0 -- have
+ a working, durable sync <footnote>
+ <para>The Cloudera blog post <link
+ xlink:href="http://www.cloudera.com/blog/2012/01/an-update-on-apache-hadoop-1-0/">An
+ update on Apache Hadoop 1.0</link> by Charles Zedlweski has a nice exposition on how
+ all the Hadoop versions relate. Its worth checking out if you are having trouble
+ making sense of the Hadoop version morass. </para>
+ </footnote>. Sync has to be explicitly enabled by setting
+ <varname>dfs.support.append</varname> equal to true on both the client side -- in
+ <filename>hbase-site.xml</filename> -- and on the serverside in
+ <filename>hdfs-site.xml</filename> (The sync facility HBase needs is a subset of the
+ append code path).</para>
+ <programlisting><![CDATA[
+<property>
+ <name>dfs.support.append</name>
+ <value>true</value>
+</property>]]></programlisting>
+ <para> You will have to restart your cluster after making this edit. Ignore the
+ chicken-little comment you'll find in the <filename>hdfs-default.xml</filename> in the
+ description for the <varname>dfs.support.append</varname> configuration. </para>
+ </section>
+ <section
+ xml:id="hadoop.security">
+ <title>Apache HBase on Secure Hadoop</title>
+ <para>Apache HBase will run on any Hadoop 0.20.x that incorporates Hadoop security features
+ as long as you do as suggested above and replace the Hadoop jar that ships with HBase with
+ the secure version. If you want to read more about how to setup Secure HBase, see <xref
+ linkend="hbase.secure.configuration" />.</para>
+ </section>
+ <section
+ xml:id="dfs.datanode.max.transfer.threads">
+ <title><varname>dfs.datanode.max.transfer.threads</varname><indexterm>
+ <primary>xcievers</primary>
+ </indexterm></title>
+ <para>An Hadoop HDFS datanode has an upper bound on the number of files that it will serve
+ at any one time. The upper bound parameter is called <varname>xcievers</varname> (yes,
+ this is misspelled). Again, before doing any loading, make sure you have configured
+ Hadoop's <filename>conf/hdfs-site.xml</filename> setting the <varname>xceivers</varname>
+ value to at least the following:</para>
+ <programlisting><![CDATA[
+<property>
+ <name>dfs.datanode.max.transfer.threads</name>
+ <value>4096</value>
+</property>
+ ]]></programlisting>
- <section xml:id="pseudo">
- <title>Pseudo-distributed</title>
+ <para>Be sure to restart your HDFS after making the above configuration.</para>
- <para>A pseudo-distributed mode is simply a fully-distributed mode run on
- a single host. Use this configuration testing and prototyping on
- HBase. Do not use this configuration for production nor for
- evaluating HBase performance.</para>
+ <para>Not having this configuration in place makes for strange looking failures. Eventually
+ you'll see a complain in the datanode logs complaining about the xcievers exceeded, but on
+ the run up to this one manifestation is complaint about missing blocks. For example:<footnote>
+ <para>See <link
+ xlink:href="http://ccgtech.blogspot.com/2010/02/hadoop-hdfs-deceived-by-xciever.html">Hadoop
+ HDFS: Deceived by Xciever</link> for an informative rant on xceivering.</para>
+ </footnote></para>
+ <para>See also <xref
+ linkend="casestudies.xceivers" />
+ </para>
+ <screen>10/12/08 20:10:31 INFO hdfs.DFSClient: Could not obtain block
+ blk_XXXXXXXXXXXXXXXXXXXXXX_YYYYYYYY from any node: java.io.IOException: No live nodes
+ contain current block. Will get new block locations from namenode and retry...</screen>
- <para>First, if you want to run on HDFS rather than on the local filesystem,
- setup your HDFS. You can set up HDFS also in pseudo-distributed mode
- (TODO: Add pointer to HOWTO doc; the hadoop site doesn't have any any more).
- Ensure you have a working HDFS before proceeding.
- </para>
+ </section>
+ </section>
+ <!-- hadoop -->
+ </section>
+
+ <section
+ xml:id="standalone_dist">
+ <title>HBase run modes: Standalone and Distributed</title>
+
+ <para>HBase has two run modes: <xref
+ linkend="standalone" /> and <xref
+ linkend="distributed" />. Out of the box, HBase runs in standalone mode. Whatever your mode,
+ you will need to configure HBase by editing files in the HBase <filename>conf</filename>
+ directory. At a minimum, you must edit <code>conf/hbase-env.sh</code> to tell HBase which
+ <command>java</command> to use. In this file you set HBase environment variables such as the
+ heapsize and other options for the <application>JVM</application>, the preferred location for
+ log files, etc. Set <varname>JAVA_HOME</varname> to point at the root of your
+ <command>java</command> install.</para>
+
+ <section
+ xml:id="standalone">
+ <title>Standalone HBase</title>
+
+ <para>This is the default mode. Standalone mode is what is described in the <xref
+ linkend="quickstart" /> section. In standalone mode, HBase does not use HDFS -- it uses
+ the local filesystem instead -- and it runs all HBase daemons and a local ZooKeeper all up
+ in the same JVM. Zookeeper binds to a well known port so clients may talk to HBase.</para>
+ </section>
- <para>Next, configure HBase. Edit <filename>conf/hbase-site.xml</filename>.
- This is the file into which you add local customizations and overrides.
- At a minimum, you must tell HBase to run in (pseudo-)distributed mode rather than
- in default standalone mode. To do this, set the <varname>hbase.cluster.distributed</varname>
- property to true (Its default is <varname>false</varname>). The absolute bare-minimum
- <filename>hbase-site.xml</filename> is therefore as follows:
-<programlisting>
-<configuration>
- <property>
- <name>hbase.cluster.distributed</name>
- <value>true</value>
- </property>
-</configuration>
-</programlisting>
-With this configuration, HBase will start up an HBase Master process, a ZooKeeper server,
-and a RegionServer process running against the
-local filesystem writing to wherever your operating system stores temporary files into a directory
-named <filename>hbase-YOUR_USER_NAME</filename>.</para>
-
-<para>Such a setup, using the local filesystem and
-writing to the operating systems's temporary directory is an ephemeral setup; the Hadoop
-local filesystem -- which is what HBase uses when it is writing the local filesytem does not
-support <command>sync</command> so unless the system is shutdown properly, the data will be lost. Writing to
-the operating system's temporary directory can also make for data loss when the machine
-is restarted as this directory is usually cleared on reboot. For a more permanent
-setup, see the next example where we make use of an instance of HDFS; HBase data will
-be written to the Hadoop distributed filesystem rather than to the local filesystem's
-tmp directory.</para>
-<para>In this <filename>conf/hbase-site.xml</filename> example, the
-<varname>hbase.rootdir</varname> property points to the local HDFS instance
-homed on the node <varname>h-24-30.example.com</varname>.
- <note>
- <title>Let HBase create <filename>${hbase.rootdir}</filename></title>
- <para>Let HBase create the <varname>hbase.rootdir</varname>
- directory. If you don't, you'll get warning saying HBase needs a
- migration run because the directory is missing files expected by
- HBase (it'll create them if you let it).</para>
- </note>
-<programlisting>
+ <section
+ xml:id="distributed">
+ <title>Distributed</title>
+
+ <para>Distributed mode can be subdivided into distributed but all daemons run on a single node
+ -- a.k.a <emphasis>pseudo-distributed</emphasis>-- and
+ <emphasis>fully-distributed</emphasis> where the daemons are spread across all nodes in
+ the cluster <footnote>
+ <para>The pseudo-distributed vs fully-distributed nomenclature comes from Hadoop.</para>
+ </footnote>.</para>
+
+ <para>Pseudo-distributed mode can run against the local filesystem or it can run against an
+ instance of the <emphasis>Hadoop Distributed File System</emphasis> (HDFS).
+ Fully-distributed mode can ONLY run on HDFS. See the Hadoop <link
+ xlink:href="http://hadoop.apache.org/common/docs/r1.1.1/api/overview-summary.html#overview_description">
+ requirements and instructions</link> for how to set up HDFS.</para>
+
+ <para>Below we describe the different distributed setups. Starting, verification and
+ exploration of your install, whether a <emphasis>pseudo-distributed</emphasis> or
+ <emphasis>fully-distributed</emphasis> configuration is described in a section that
+ follows, <xref
+ linkend="confirm" />. The same verification script applies to both deploy types.</para>
+ <section
+ xml:id="pseudo">
+ <title>Pseudo-distributed</title>
+
+ <para>A pseudo-distributed mode is simply a fully-distributed mode run on a single host. Use
+ this configuration testing and prototyping on HBase. Do not use this configuration for
+ production nor for evaluating HBase performance.</para>
+
+ <para>First, if you want to run on HDFS rather than on the local filesystem, setup your
+ HDFS. You can set up HDFS also in pseudo-distributed mode (TODO: Add pointer to HOWTO doc;
+ the hadoop site doesn't have any any more). Ensure you have a working HDFS before
+ proceeding. </para>
+
+ <para>Next, configure HBase. Edit <filename>conf/hbase-site.xml</filename>. This is the file
+ into which you add local customizations and overrides. At a minimum, you must tell HBase
+ to run in (pseudo-)distributed mode rather than in default standalone mode. To do this,
+ set the <varname>hbase.cluster.distributed</varname> property to true (Its default is
+ <varname>false</varname>). The absolute bare-minimum <filename>hbase-site.xml</filename>
+ is therefore as follows:</para>
+ <programlisting><![CDATA[
+<configuration>
+ <property>
+ <name>hbase.cluster.distributed</name>
+ <value>true</value>
+ </property>
+</configuration>
+]]>
+ </programlisting>
+ <para>With this configuration, HBase will start up an HBase Master process, a ZooKeeper
+ server, and a RegionServer process running against the local filesystem writing to
+ wherever your operating system stores temporary files into a directory named
+ <filename>hbase-YOUR_USER_NAME</filename>.</para>
+
+ <para>Such a setup, using the local filesystem and writing to the operating systems's
+ temporary directory is an ephemeral setup; the Hadoop local filesystem -- which is what
+ HBase uses when it is writing the local filesytem does not support <command>sync</command>
+ so unless the system is shutdown properly, the data will be lost. Writing to the operating
+ system's temporary directory can also make for data loss when the machine is restarted as
+ this directory is usually cleared on reboot. For a more permanent setup, see the next
+ example where we make use of an instance of HDFS; HBase data will be written to the Hadoop
+ distributed filesystem rather than to the local filesystem's tmp directory.</para>
+ <para>In this <filename>conf/hbase-site.xml</filename> example, the
+ <varname>hbase.rootdir</varname> property points to the local HDFS instance homed on the
+ node <varname>h-24-30.example.com</varname>.</para>
+ <note>
+ <title>Let HBase create <filename>${hbase.rootdir}</filename></title>
+ <para>Let HBase create the <varname>hbase.rootdir</varname> directory. If you don't,
+ you'll get warning saying HBase needs a migration run because the directory is missing
+ files expected by HBase (it'll create them if you let it).</para>
+ </note>
+ <programlisting>
<configuration>
<property>
<name>hbase.rootdir</name>
@@ -523,315 +644,296 @@ homed on the node <varname>h-24-30.example.com</varname>.
<value>true</value>
</property>
</configuration>
-</programlisting>
-</para>
- <para>Now skip to <xref linkend="confirm" /> for how to start and verify your
- pseudo-distributed install. <footnote>
- <para>See <xref linkend="pseudo.extras"/> for notes on how to start extra Masters and
- RegionServers when running pseudo-distributed.</para>
- </footnote></para>
-
- <section xml:id="pseudo.extras">
- <title>Pseudo-distributed Extras</title>
-
- <section xml:id="pseudo.extras.start">
- <title>Startup</title>
- <para>To start up the initial HBase cluster...
- <programlisting>% bin/start-hbase.sh</programlisting>
- </para>
- <para>To start up an extra backup master(s) on the same server run...
- <programlisting>% bin/local-master-backup.sh start 1</programlisting>
- ... the '1' means use ports 16001 & 16011, and this backup master's
- logfile will be at
- <filename>logs/hbase-${USER}-1-master-${HOSTNAME}.log</filename>.
- </para>
- <para>To startup multiple backup masters run... <programlisting>% bin/local-master-backup.sh start 2 3</programlisting> You can start up to 9 backup masters (10 total).
- </para>
- <para>To start up more regionservers...
- <programlisting>% bin/local-regionservers.sh start 1</programlisting>
- ... where '1' means use ports 16201 & 16301 and its logfile will be at
- `<filename>logs/hbase-${USER}-1-regionserver-${HOSTNAME}.log</filename>.
- </para>
- <para>To add 4 more regionservers in addition to the one you just started by running... <programlisting>% bin/local-regionservers.sh start 2 3 4 5</programlisting>
- This supports up to 99 extra regionservers (100 total).
- </para>
- </section>
- <section xml:id="pseudo.options.stop">
- <title>Stop</title>
- <para>Assuming you want to stop master backup # 1, run...
- <programlisting>% cat /tmp/hbase-${USER}-1-master.pid |xargs kill -9</programlisting>
- Note that bin/local-master-backup.sh stop 1 will try to stop the cluster along with the master.
- </para>
- <para>To stop an individual regionserver, run...
- <programlisting>% bin/local-regionservers.sh stop 1
- </programlisting>
- </para>
- </section>
-
- </section>
-
- </section>
-
-
-
-
-
- <section xml:id="fully_dist">
- <title>Fully-distributed</title>
+ </programlisting>
- <para>For running a fully-distributed operation on more than one
- host, make the following configurations. In
- <filename>hbase-site.xml</filename>, add the property
- <varname>hbase.cluster.distributed</varname> and set it to
- <varname>true</varname> and point the HBase
- <varname>hbase.rootdir</varname> at the appropriate HDFS NameNode
- and location in HDFS where you would like HBase to write data. For
- example, if you namenode were running at namenode.example.org on
- port 8020 and you wanted to home your HBase in HDFS at
- <filename>/hbase</filename>, make the following
- configuration.</para>
+ <para>Now skip to <xref
+ linkend="confirm" /> for how to start and verify your pseudo-distributed install. <footnote>
+ <para>See <xref
+ linkend="pseudo.extras" /> for notes on how to start extra Masters and RegionServers
+ when running pseudo-distributed.</para>
+ </footnote></para>
- <programlisting>
-<configuration>
- ...
- <property>
- <name>hbase.rootdir</name>
- <value>hdfs://namenode.example.org:8020/hbase</value>
- <description>The directory shared by RegionServers.
- </description>
- </property>
- <property>
- <name>hbase.cluster.distributed</name>
- <value>true</value>
- <description>The mode the cluster will be in. Possible values are
- false: standalone and pseudo-distributed setups with managed Zookeeper
- true: fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh)
- </description>
- </property>
- ...
-</configuration>
-</programlisting>
-
- <section xml:id="regionserver">
- <title><filename>regionservers</filename></title>
-
- <para>In addition, a fully-distributed mode requires that you
- modify <filename>conf/regionservers</filename>. The
- <xref linkend="regionservers" /> file
- lists all hosts that you would have running
- <application>HRegionServer</application>s, one host per line (This
- file in HBase is like the Hadoop <filename>slaves</filename>
- file). All servers listed in this file will be started and stopped
- when HBase cluster start or stop is run.</para>
+ <section
+ xml:id="pseudo.extras">
+ <title>Pseudo-distributed Extras</title>
+
+ <section
+ xml:id="pseudo.extras.start">
+ <title>Startup</title>
+ <para>To start up the initial HBase cluster...</para>
+ <screen>% bin/start-hbase.sh</screen>
+ <para>To start up an extra backup master(s) on the same server run...</para>
+ <screen>% bin/local-master-backup.sh start 1</screen>
+ <para>... the '1' means use ports 16001 & 16011, and this backup master's logfile
+ will be at <filename>logs/hbase-${USER}-1-master-${HOSTNAME}.log</filename>. </para>
+ <para>To startup multiple backup masters run...</para>
+ <screen>% bin/local-master-backup.sh start 2 3</screen>
+ <para>You can start up to 9 backup masters (10 total). </para>
+ <para>To start up more regionservers...</para>
+ <screen>% bin/local-regionservers.sh start 1</screen>
+ <para>... where '1' means use ports 16201 & 16301 and its logfile will be at
+ `<filename>logs/hbase-${USER}-1-regionserver-${HOSTNAME}.log</filename>. </para>
+ <para>To add 4 more regionservers in addition to the one you just started by
+ running...</para>
+ <screen>% bin/local-regionservers.sh start 2 3 4 5</screen>
+ <para>This supports up to 99 extra regionservers (100 total). </para>
</section>
-
- <section xml:id="hbase.zookeeper">
- <title>ZooKeeper and HBase</title>
- <para>See section <xref linkend="zookeeper"/> for ZooKeeper setup for HBase.</para>
+ <section
+ xml:id="pseudo.options.stop">
+ <title>Stop</title>
+ <para>Assuming you want to stop master backup # 1, run...</para>
+ <screen>% cat /tmp/hbase-${USER}-1-master.pid |xargs kill -9</screen>
+ <para>Note that bin/local-master-backup.sh stop 1 will try to stop the cluster along
+ with the master. </para>
+ <para>To stop an individual regionserver, run...</para>
+ <screen>% bin/local-regionservers.sh stop 1</screen>
</section>
- <section xml:id="hdfs_client_conf">
- <title>HDFS Client Configuration</title>
-
- <para>Of note, if you have made <emphasis>HDFS client
- configuration</emphasis> on your Hadoop cluster -- i.e.
- configuration you want HDFS clients to use as opposed to
- server-side configurations -- HBase will not see this
- configuration unless you do one of the following:</para>
-
- <itemizedlist>
- <listitem>
- <para>Add a pointer to your <varname>HADOOP_CONF_DIR</varname>
- to the <varname>HBASE_CLASSPATH</varname> environment variable
- in <filename>hbase-env.sh</filename>.</para>
- </listitem>
-
- <listitem>
- <para>Add a copy of <filename>hdfs-site.xml</filename> (or
- <filename>hadoop-site.xml</filename>) or, better, symlinks,
- under <filename>${HBASE_HOME}/conf</filename>, or</para>
- </listitem>
-
- <listitem>
- <para>if only a small set of HDFS client configurations, add
- them to <filename>hbase-site.xml</filename>.</para>
- </listitem>
- </itemizedlist>
-
- <para>An example of such an HDFS client configuration is
- <varname>dfs.replication</varname>. If for example, you want to
- run with a replication factor of 5, hbase will create files with
- the default of 3 unless you do the above to make the configuration
- available to HBase.</para>
- </section>
</section>
- </section>
-
- <section xml:id="confirm">
- <title>Running and Confirming Your Installation</title>
-
-
-
- <para>Make sure HDFS is running first. Start and stop the Hadoop HDFS
- daemons by running <filename>bin/start-hdfs.sh</filename> over in the
- <varname>HADOOP_HOME</varname> directory. You can ensure it started
- properly by testing the <command>put</command> and
- <command>get</command> of files into the Hadoop filesystem. HBase does
- not normally use the mapreduce daemons. These do not need to be
- started.</para>
-
-
-
- <para><emphasis>If</emphasis> you are managing your own ZooKeeper,
- start it and confirm its running else, HBase will start up ZooKeeper
- for you as part of its start process.</para>
-
+ </section>
- <para>Start HBase with the following command:</para>
- <programlisting>bin/start-hbase.sh</programlisting>
- <para>Run the above from the <varname>HBASE_HOME</varname> directory.</para>
+ <section
+ xml:id="fully_dist">
+ <title>Fully-distributed</title>
- <para>You should now have a running HBase instance. HBase logs can be
- found in the <filename>logs</filename> subdirectory. Check them out
- especially if HBase had trouble starting.</para>
+ <para>For running a fully-distributed operation on more than one host, make the following
+ configurations. In <filename>hbase-site.xml</filename>, add the property
+ <varname>hbase.cluster.distributed</varname> and set it to <varname>true</varname> and
+ point the HBase <varname>hbase.rootdir</varname> at the appropriate HDFS NameNode and
+ location in HDFS where you would like HBase to write data. For example, if you namenode
+ were running at namenode.example.org on port 8020 and you wanted to home your HBase in
+ HDFS at <filename>/hbase</filename>, make the following configuration.</para>
+ <programlisting><![CDATA[
+<configuration>
+ ...
+ <property>
+ <name>hbase.rootdir</name>
+ <value>hdfs://namenode.example.org:8020/hbase</value>
+ <description>The directory shared by RegionServers.
+ </description>
+ </property>
+ <property>
+ <name>hbase.cluster.distributed</name>
+ <value>true</value>
+ <description>The mode the cluster will be in. Possible values are
+ false: standalone and pseudo-distributed setups with managed Zookeeper
+ true: fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh)
+ </description>
+ </property>
+ ...
+</configuration>
+]]>
+ </programlisting>
+ <section
+ xml:id="regionserver">
+ <title><filename>regionservers</filename></title>
- <para>HBase also puts up a UI listing vital attributes. By default its
- deployed on the Master host at port 16010 (HBase RegionServers listen
- on port 16020 by default and put up an informational http server at
- 16030). If the Master were running on a host named
- <varname>master.example.org</varname> on the default port, to see the
- Master's homepage you'd point your browser at
- <filename>http://master.example.org:16010</filename>.</para>
+ <para>In addition, a fully-distributed mode requires that you modify
+ <filename>conf/regionservers</filename>. The <xref
+ linkend="regionservers" /> file lists all hosts that you would have running
+ <application>HRegionServer</application>s, one host per line (This file in HBase is
+ like the Hadoop <filename>slaves</filename> file). All servers listed in this file will
+ be started and stopped when HBase cluster start or stop is run.</para>
+ </section>
- <para>Prior to HBase 0.98, the default ports the master ui was deployed
- on port 16010, and the HBase RegionServers would listen
- on port 16020 by default and put up an informational http server at
- 16030.
- </para>
+ <section
+ xml:id="hbase.zookeeper">
+ <title>ZooKeeper and HBase</title>
+ <para>See section <xref
+ linkend="zookeeper" /> for ZooKeeper setup for HBase.</para>
+ </section>
- <para>Once HBase has started, see the <xref linkend="shell_exercises" /> for how to
- create tables, add data, scan your insertions, and finally disable and
- drop your tables.</para>
+ <section
+ xml:id="hdfs_client_conf">
+ <title>HDFS Client Configuration</title>
+ <para>Of note, if you have made <emphasis>HDFS client configuration</emphasis> on your
+ Hadoop cluster -- i.e. configuration you want HDFS clients to use as opposed to
+ server-side configurations -- HBase will not see this configuration unless you do one of
+ the following:</para>
+ <itemizedlist>
+ <listitem>
+ <para>Add a pointer to your <varname>HADOOP_CONF_DIR</varname> to the
+ <varname>HBASE_CLASSPATH</varname> environment variable in
+ <filename>hbase-env.sh</filename>.</para>
+ </listitem>
- <para>To stop HBase after exiting the HBase shell enter
- <programlisting>$ ./bin/stop-hbase.sh
-stopping hbase...............</programlisting> Shutdown can take a moment to
- complete. It can take longer if your cluster is comprised of many
- machines. If you are running a distributed operation, be sure to wait
- until HBase has shut down completely before stopping the Hadoop
- daemons.</para>
+ <listitem>
+ <para>Add a copy of <filename>hdfs-site.xml</filename> (or
+ <filename>hadoop-site.xml</filename>) or, better, symlinks, under
+ <filename>${HBASE_HOME}/conf</filename>, or</para>
+ </listitem>
+ <listitem>
+ <para>if only a small set of HDFS client configurations, add them to
+ <filename>hbase-site.xml</filename>.</para>
+ </listitem>
+ </itemizedlist>
+ <para>An example of such an HDFS client configuration is
+ <varname>dfs.replication</varname>. If for example, you want to run with a replication
+ factor of 5, hbase will create files with the default of 3 unless you do the above to
+ make the configuration available to HBase.</para>
+ </section>
</section>
- </section> <!-- run modes -->
-
-
-
- <section xml:id="config.files">
- <title>Configuration Files</title>
-
- <section xml:id="hbase.site">
- <title><filename>hbase-site.xml</filename> and <filename>hbase-default.xml</filename></title>
- <para>Just as in Hadoop where you add site-specific HDFS configuration
- to the <filename>hdfs-site.xml</filename> file,
- for HBase, site specific customizations go into
- the file <filename>conf/hbase-site.xml</filename>.
- For the list of configurable properties, see
- <xref linkend="hbase_default_configurations" />
- below or view the raw <filename>hbase-default.xml</filename>
- source file in the HBase source code at
- <filename>src/main/resources</filename>.
- </para>
- <para>
- Not all configuration options make it out to
- <filename>hbase-default.xml</filename>. Configuration
- that it is thought rare anyone would change can exist only
- in code; the only way to turn up such configurations is
- via a reading of the source code itself.
- </para>
- <para>
- Currently, changes here will require a cluster restart for HBase to notice the change.
- </para>
- <!--The file hbase-default.xml is generated as part of
+ </section>
+
+ <section
+ xml:id="confirm">
+ <title>Running and Confirming Your Installation</title>
+
+
+
+ <para>Make sure HDFS is running first. Start and stop the Hadoop HDFS daemons by running
+ <filename>bin/start-hdfs.sh</filename> over in the <varname>HADOOP_HOME</varname>
+ directory. You can ensure it started properly by testing the <command>put</command> and
+ <command>get</command> of files into the Hadoop filesystem. HBase does not normally use
+ the mapreduce daemons. These do not need to be started.</para>
+ <para><emphasis>If</emphasis> you are managing your own ZooKeeper, start it and confirm its
+ running else, HBase will start up ZooKeeper for you as part of its start process.</para>
+ <para>Start HBase with the following command:</para>
+ <screen>bin/start-hbase.sh</screen>
+ <para>Run the above from the <varname>HBASE_HOME</varname> directory.</para>
+ <para>You should now have a running HBase instance. HBase logs can be found in the
+ <filename>logs</filename> subdirectory. Check them out especially if HBase had trouble
+ starting.</para>
+
+ <para>HBase also puts up a UI listing vital attributes. By default its deployed on the Master
+ host at port 16010 (HBase RegionServers listen on port 16020 by default and put up an
+ informational http server at 16030). If the Master were running on a host named
+ <varname>master.example.org</varname> on the default port, to see the Master's homepage
+ you'd point your browser at <filename>http://master.example.org:16010</filename>.</para>
+
+ <para>Prior to HBase 0.98, the default ports the master ui was deployed on port 16010, and the
+ HBase RegionServers would listen on port 16020 by default and put up an informational http
+ server at 16030. </para>
+
+ <para>Once HBase has started, see the <xref
+ linkend="shell_exercises" /> for how to create tables, add data, scan your insertions, and
+ finally disable and drop your tables.</para>
+
+ <para>To stop HBase after exiting the HBase shell enter</para>
+ <screen>$ ./bin/stop-hbase.sh
+stopping hbase...............</screen>
+ <para>Shutdown can take a moment to complete. It can take longer if your cluster is comprised
+ of many machines. If you are running a distributed operation, be sure to wait until HBase
+ has shut down completely before stopping the Hadoop daemons.</para>
+ </section>
+ </section>
+ <!-- run modes -->
+
+
+
+ <section
+ xml:id="config.files">
+ <title>Configuration Files</title>
+
+ <section
+ xml:id="hbase.site">
+ <title><filename>hbase-site.xml</filename> and <filename>hbase-default.xml</filename></title>
+ <para>Just as in Hadoop where you add site-specific HDFS configuration to the
+ <filename>hdfs-site.xml</filename> file, for HBase, site specific customizations go into
+ the file <filename>conf/hbase-site.xml</filename>. For the list of configurable properties,
+ see <xref
+ linkend="hbase_default_configurations" /> below or view the raw
+ <filename>hbase-default.xml</filename> source file in the HBase source code at
+ <filename>src/main/resources</filename>. </para>
+ <para> Not all configuration options make it out to <filename>hbase-default.xml</filename>.
+ Configuration that it is thought rare anyone would change can exist only in code; the only
+ way to turn up such configurations is via a reading of the source code itself. </para>
+ <para> Currently, changes here will require a cluster restart for HBase to notice the change. </para>
+ <!--The file hbase-default.xml is generated as part of
the build of the hbase site. See the hbase pom.xml.
The generated file is a docbook section with a glossary
in it-->
- <!--presumes the pre-site target has put the hbase-default.xml at this location-->
- <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="../../../target/docbkx/hbase-default.xml">
+ <!--presumes the pre-site target has put the hbase-default.xml at this location-->
+ <xi:include
+ xmlns:xi="http://www.w3.org/2001/XInclude"
+ href="../../../target/docbkx/hbase-default.xml">
<xi:fallback>
- <section xml:id="hbase_default_configurations">
- <title></title>
+ <section
+ xml:id="hbase_default_configurations">
+ <title />
<para>
- <emphasis>This file is fallback content</emphasis>. If you are seeing this, something is wrong with the build of the HBase documentation or you are doing pre-build verification.
- </para>
- <para>
- The file hbase-default.xml is generated as part of
- the build of the hbase site. See the hbase <filename>pom.xml</filename>.
- The generated file is a docbook glossary.
- </para>
+ <emphasis>This file is fallback content</emphasis>. If you are seeing this, something
+ is wrong with the build of the HBase documentation or you are doing pre-build
+ verification. </para>
+ <para> The file hbase-default.xml is generated as part of the build of the hbase site.
+ See the hbase <filename>pom.xml</filename>. The generated file is a docbook glossary. </para>
<section>
<title>IDs that are auto-generated and cause validation errors if not present</title>
- <para>
- Each of these is a reference to a configuration file parameter which will cause an error if you are using the fallback content here. This is a dirty dirty hack.
- </para>
- <section xml:id="fail.fast.expired.active.master">
+ <para> Each of these is a reference to a configuration file parameter which will cause
+ an error if you are using the fallback content here. This is a dirty dirty hack. </para>
+ <section
+ xml:id="fail.fast.expired.active.master">
<title>fail.fast.expired.active.master</title>
<para />
</section>
- <section xml:id="hbase.hregion.memstore.flush.size">
+ <section
+ xml:id="hbase.hregion.memstore.flush.size">
<title>"hbase.hregion.memstore.flush.size"</title>
<para />
</section>
- <section xml:id="hbase.hstore.bytes.per.checksum">
+ <section
+ xml:id="hbase.hstore.bytes.per.checksum">
<title>hbase.hstore.bytes.per.checksum</title>
<para />
</section>
- <section xml:id="hbase.online.schema.update.enable">
+ <section
+ xml:id="hbase.online.schema.update.enable">
<title>hbase.online.schema.update.enable</title>
<para />
</section>
- <section xml:id="hbase.regionserver.global.memstore.size">
+ <section
+ xml:id="hbase.regionserver.global.memstore.size">
<title>hbase.regionserver.global.memstore.size</title>
<para />
</section>
- <section xml:id="hbase.hregion.max.filesize">
+ <section
+ xml:id="hbase.hregion.max.filesize">
<title>hbase.hregion.max.filesize</title>
<para />
</section>
- <section xml:id="hbase.hstore.blockingStoreFiles">
+ <section
+ xml:id="hbase.hstore.blockingStoreFiles">
<title>hbase.hstore.BlockingStoreFiles</title>
<para />
</section>
- <section xml:id="hfile.block.cache.size">
+ <section
+ xml:id="hfile.block.cache.size">
<title>hfile.block.cache.size</title>
<para />
</section>
- <section xml:id="copy.table">
+ <section
+ xml:id="copy.table">
<title>copy.table</title>
<para />
</section>
- <section xml:id="hbase.hstore.checksum.algorithm">
+ <section
+ xml:id="hbase.hstore.checksum.algorithm">
<title>hbase.hstore.checksum.algorithm</title>
<para />
</section>
- <section xml:id="hbase.zookeeper.useMulti">
+ <section
+ xml:id="hbase.zookeeper.useMulti">
<title>hbase.zookeeper.useMulti</title>
<para />
</section>
- <section xml:id="hbase.hregion.memstore.block.multiplier">
+ <section
+ xml:id="hbase.hregion.memstore.block.multiplier">
<title>hbase.hregion.memstore.block.multiplier</title>
<para />
</section>
- <section xml:id="hbase.regionserver.global.memstore.size.lower.limit">
+ <section
+ xml:id="hbase.regionserver.global.memstore.size.lower.limit">
<title>hbase.regionserver.global.memstore.size.lower.limit</title>
<para />
</section>
@@ -841,52 +943,45 @@ stopping hbase...............</programlisting> Shutdown can take a moment to
</xi:include>
</section>
- <section xml:id="hbase.env.sh">
+ <section
+ xml:id="hbase.env.sh">
<title><filename>hbase-env.sh</filename></title>
- <para>Set HBase environment variables in this file.
- Examples include options to pass the JVM on start of
- an HBase daemon such as heap size and garbage collector configs.
- You can also set configurations for HBase configuration, log directories,
- niceness, ssh options, where to locate process pid files,
- etc. Open the file at
- <filename>conf/hbase-env.sh</filename> and peruse its content.
- Each option is fairly well documented. Add your own environment
- variables here if you want them read by HBase daemons on startup.</para>
- <para>
- Changes here will require a cluster restart for HBase to notice the change.
- </para>
- </section>
+ <para>Set HBase environment variables in this file. Examples include options to pass the JVM
+ on start of an HBase daemon such as heap size and garbage collector configs. You can also
+ set configurations for HBase configuration, log directories, niceness, ssh options, where to
+ locate process pid files, etc. Open the file at <filename>conf/hbase-env.sh</filename> and
+ peruse its content. Each option is fairly well documented. Add your own environment
+ variables here if you want them read by HBase daemons on startup.</para>
+ <para> Changes here will require a cluster restart for HBase to notice the change. </para>
+ </section>
- <section xml:id="log4j">
+ <section
+ xml:id="log4j">
<title><filename>log4j.properties</filename></title>
- <para>Edit this file to change rate at which HBase files
- are rolled and to change the level at which HBase logs messages.
- </para>
- <para>
- Changes here will require a cluster restart for HBase to notice the change
- though log levels can be changed for particular daemons via the HBase UI.
- </para>
- </section>
+ <para>Edit this file to change rate at which HBase files are rolled and to change the level at
+ which HBase logs messages. </para>
+ <para> Changes here will require a cluster restart for HBase to notice the change though log
+ levels can be changed for particular daemons via the HBase UI. </para>
+ </section>
- <section xml:id="client_dependencies"><title>Client configuration and dependencies connecting to an HBase cluster</title>
- <para>If you are running HBase in standalone mode, you don't need to configure anything for your client to work
- provided that they are all on the same machine.</para>
- <para>
- Since the HBase Master may move around, clients bootstrap by looking to ZooKeeper for
- current critical locations. ZooKeeper is where all these values are kept. Thus clients
- require the location of the ZooKeeper ensemble information before they can do anything else.
- Usually this the ensemble location is kept out in the <filename>hbase-site.xml</filename> and
- is picked up by the client from the <varname>CLASSPATH</varname>.</para>
-
- <para>If you are configuring an IDE to run a HBase client, you should
- include the <filename>conf/</filename> directory on your classpath so
- <filename>hbase-site.xml</filename> settings can be found (or
- add <filename>src/test/resources</filename> to pick up the hbase-site.xml
- used by tests).
- </para>
- <para>
- Minimally, a client of HBase needs several libraries in its <varname>CLASSPATH</varname> when connecting to a cluster, including:
- <programlisting>
+ <section
+ xml:id="client_dependencies">
+ <title>Client configuration and dependencies connecting to an HBase cluster</title>
+ <para>If you are running HBase in standalone mode, you don't need to configure anything for
+ your client to work provided that they are all on the same machine.</para>
+ <para> Since the HBase Master may move around, clients bootstrap by looking to ZooKeeper for
+ current critical locations. ZooKeeper is where all these values are kept.
<TRUNCATED>