You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hbase.apache.org by st...@apache.org on 2013/03/13 16:20:20 UTC
svn commit: r1455996 [5/7] - in /hbase/branches/0.94/src: docbkx/ site/
site/resources/css/ site/resources/images/ site/xdoc/
Modified: hbase/branches/0.94/src/docbkx/getting_started.xml
URL: http://svn.apache.org/viewvc/hbase/branches/0.94/src/docbkx/getting_started.xml?rev=1455996&r1=1455995&r2=1455996&view=diff
==============================================================================
--- hbase/branches/0.94/src/docbkx/getting_started.xml (original)
+++ hbase/branches/0.94/src/docbkx/getting_started.xml Wed Mar 13 15:20:19 2013
@@ -32,9 +32,8 @@
<title>Introduction</title>
<para><xref linkend="quickstart" /> will get you up and
- running on a single-node instance of HBase using the local filesystem.
- <xref linkend="configuration" /> describes setup
- of HBase in distributed mode running on top of HDFS.</para>
+ running on a single-node instance of HBase using the local filesystem.
+ </para>
</section>
<section xml:id="quickstart">
@@ -45,17 +44,31 @@
rows via the HBase <command>shell</command>, and then cleaning
up and shutting down your standalone HBase instance. The below exercise
should take no more than ten minutes (not including download time).</para>
+ <para>Before we proceed, make sure you are good on the below loopback prerequisite.</para>
+ <note xml:id="loopback.ip.getting.started">
+ <title>Loopback IP</title>
+ <para>HBase expects the loopback IP address to be 127.0.0.1. Ubuntu and some other distributions,
+ for example, will default to 127.0.1.1 and this will cause problems for you.
+ </para>
+ <para><filename>/etc/hosts</filename> should look something like this:
+<programlisting>
+ 127.0.0.1 localhost
+ 127.0.0.1 ubuntu.ubuntu-domain ubuntu
+</programlisting>
+ </para>
+ </note>
+
<section>
<title>Download and unpack the latest stable release.</title>
<para>Choose a download site from this list of <link
xlink:href="http://www.apache.org/dyn/closer.cgi/hbase/">Apache Download
- Mirrors</link>. Click on suggested top link. This will take you to a
+ Mirrors</link>. Click on the suggested top link. This will take you to a
mirror of <emphasis>HBase Releases</emphasis>. Click on the folder named
<filename>stable</filename> and then download the file that ends in
<filename>.tar.gz</filename> to your local filesystem; e.g.
- <filename>hbase-<?eval ${project.version}?>.tar.gz</filename>.</para>
+ <filename>hbase-0.94.2.tar.gz</filename>.</para>
<para>Decompress and untar your download and then change into the
unpacked directory.</para>
@@ -65,24 +78,27 @@ $ cd hbase-<?eval ${project.version}?>
</programlisting></para>
<para>At this point, you are ready to start HBase. But before starting
- it, you might want to edit <filename>conf/hbase-site.xml</filename> and
- set the directory you want HBase to write to,
- <varname>hbase.rootdir</varname>. <programlisting>
-
-<?xml version="1.0"?>
+ it, edit <filename>conf/hbase-site.xml</filename>, the file you write
+ your site-specific configurations into. Set
+ <varname>hbase.rootdir</varname>, the directory HBase writes data to,
+ and <varname>hbase.zookeeper.property.dataDir</varname>, the director
+ ZooKeeper writes its data too:
+<programlisting><?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>hbase.rootdir</name>
<value>file:///DIRECTORY/hbase</value>
</property>
-</configuration>
-
-</programlisting> Replace <varname>DIRECTORY</varname> in the above with a
- path to a directory where you want HBase to store its data. By default,
- <varname>hbase.rootdir</varname> is set to
- <filename>/tmp/hbase-${user.name}</filename> which means you'll lose all
- your data whenever your server reboots (Most operating systems clear
+ <property>
+ <name>hbase.zookeeper.property.dataDir</name>
+ <value>/DIRECTORY/zookeeper</value>
+ </property>
+</configuration></programlisting> Replace <varname>DIRECTORY</varname> in the above with the
+ path to the directory you would have HBase and ZooKeeper write their data. By default,
+ <varname>hbase.rootdir</varname> is set to <filename>/tmp/hbase-${user.name}</filename>
+ and similarly so for the default ZooKeeper data location which means you'll lose all
+ your data whenever your server reboots unless you change it (Most operating systems clear
<filename>/tmp</filename> on restart).</para>
</section>
@@ -96,19 +112,19 @@ starting Master, logging to logs/hbase-u
standalone mode, HBase runs all daemons in the the one JVM; i.e. both
the HBase and ZooKeeper daemons. HBase logs can be found in the
<filename>logs</filename> subdirectory. Check them out especially if
- HBase had trouble starting.</para>
+ it seems HBase had trouble starting.</para>
<note>
<title>Is <application>java</application> installed?</title>
<para>All of the above presumes a 1.6 version of Oracle
<application>java</application> is installed on your machine and
- available on your path; i.e. when you type
+ available on your path (See <xref linkend="java" />); i.e. when you type
<application>java</application>, you see output that describes the
options the java program takes (HBase requires java 6). If this is not
the case, HBase will not start. Install java, edit
<filename>conf/hbase-env.sh</filename>, uncommenting the
- <envar>JAVA_HOME</envar> line pointing it to your java install. Then,
+ <envar>JAVA_HOME</envar> line pointing it to your java install, then,
retry the steps above.</para>
</note>
</section>
@@ -154,9 +170,7 @@ hbase(main):006:0> put 'test', 'row3'
<varname>cf</varname> in this example -- followed by a colon and then a
column qualifier suffix (<varname>a</varname> in this case).</para>
- <para>Verify the data insert.</para>
-
- <para>Run a scan of the table by doing the following</para>
+ <para>Verify the data insert by running a scan of the table as follows</para>
<para><programlisting>hbase(main):007:0> scan 'test'
ROW COLUMN+CELL
@@ -165,7 +179,7 @@ row2 column=cf:b, timestamp=128838
row3 column=cf:c, timestamp=1288380747365, value=value3
3 row(s) in 0.0590 seconds</programlisting></para>
- <para>Get a single row as follows</para>
+ <para>Get a single row</para>
<para><programlisting>hbase(main):008:0> get 'test', 'row1'
COLUMN CELL
@@ -198,9 +212,9 @@ stopping hbase...............</programli
<title>Where to go next</title>
<para>The above described standalone setup is good for testing and
- experiments only. Next move on to <xref linkend="configuration" /> where we'll go into
- depth on the different HBase run modes, requirements and critical
- configurations needed setting up a distributed HBase deploy.</para>
+ experiments only. In the next chapter, <xref linkend="configuration" />,
+ we'll go into depth on the different HBase run modes, system requirements
+ running HBase, and critical configurations setting up a distributed HBase deploy.</para>
</section>
</section>
Modified: hbase/branches/0.94/src/docbkx/ops_mgt.xml
URL: http://svn.apache.org/viewvc/hbase/branches/0.94/src/docbkx/ops_mgt.xml?rev=1455996&r1=1455995&r2=1455996&view=diff
==============================================================================
--- hbase/branches/0.94/src/docbkx/ops_mgt.xml (original)
+++ hbase/branches/0.94/src/docbkx/ops_mgt.xml Wed Mar 13 15:20:19 2013
@@ -26,16 +26,35 @@
* limitations under the License.
*/
-->
- <title>HBase Operational Management</title>
- This chapter will cover operational tools and practices required of a running HBase cluster.
+ <title>Apache HBase (TM) Operational Management</title>
+ This chapter will cover operational tools and practices required of a running Apache HBase cluster.
The subject of operations is related to the topics of <xref linkend="trouble" />, <xref linkend="performance"/>,
- and <xref linkend="configuration" /> but is a distinct topic in itself.
-
+ and <xref linkend="configuration" /> but is a distinct topic in itself.
+
<section xml:id="tools">
<title >HBase Tools and Utilities</title>
<para>Here we list HBase tools for administration, analysis, fixup, and
debugging.</para>
+ <section xml:id="driver"><title>Driver</title>
+ <para>There is a <code>Driver</code> class that is executed by the HBase jar can be used to invoke frequently accessed utilities. For example,
+<programlisting>HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath` ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/hbase-VERSION.jar
+</programlisting>
+... will return...
+<programlisting>
+An example program must be given as the first argument.
+Valid program names are:
+ completebulkload: Complete a bulk data load.
+ copytable: Export a table from local cluster to peer cluster
+ export: Write table data to HDFS.
+ import: Import data written by Export.
+ importtsv: Import data in TSV format.
+ rowcounter: Count rows in HBase table
+ verifyrep: Compare the data from tables in two different clusters. WARNING: It doesn't work for incrementColumnValues'd cells since the timestamp is chan
+</programlisting>
+... for allowable program names.
+ </para>
+ </section>
<section xml:id="hbck">
<title>HBase <application>hbck</application></title>
<subtitle>An <emphasis>fsck</emphasis> for your HBase install</subtitle>
@@ -50,6 +69,8 @@
Passing <command>-fix</command> may correct the inconsistency (This latter
is an experimental feature).
</para>
+ <para>For more information, see <xref linkend="hbck.in.depth"/>.
+ </para>
</section>
<section xml:id="hfile_tool2"><title>HFile Tool</title>
<para>See <xref linkend="hfile_tool" />.</para>
@@ -72,10 +93,17 @@
<para>Similarly you can force a split of a log file directory by
doing:<programlisting> $ ./<code>bin/hbase org.apache.hadoop.hbase.regionserver.wal.HLog --split hdfs://example.org:8020/hbase/.logs/example.org,60020,1283516293161/</code></programlisting></para>
+
+ <section xml:id="hlog_tool.prettyprint">
+ <title><classname>HLogPrettyPrinter</classname></title>
+ <para><classname>HLogPrettyPrinter</classname> is a tool with configurable options to print the contents of an HLog.
+ </para>
+ </section>
+
</section>
</section>
<section xml:id="compression.tool"><title>Compression Tool</title>
- <para>See <xref linkend="compression.tool" />.</para>
+ <para>See <xref linkend="compression.test" />.</para>
</section>
<section xml:id="copytable">
<title>CopyTable</title>
@@ -105,7 +133,12 @@
--starttime=1265875194289 --endtime=1265878794289
--peer.adr=server1,server2,server3:2181:/hbase TestTable</programlisting>
</para>
- <para>Note: caching for the input Scan is configured via <code>hbase.client.scanner.caching</code> in the job configuration.
+ <note><title>Scanner Caching</title>
+ <para>Caching for the input Scan is configured via <code>hbase.client.scanner.caching</code> in the job configuration.
+ </para>
+ </note>
+ <para>
+ See Jonathan Hsieh's <link xlink:href="http://www.cloudera.com/blog/2012/06/online-hbase-backups-with-copytable-2/">Online HBase Backups with CopyTable</link> blog post for more on <command>CopyTable</command>.
</para>
</section>
<section xml:id="export">
@@ -124,13 +157,110 @@
</programlisting>
</para>
</section>
+ <section xml:id="importtsv">
+ <title>ImportTsv</title>
+ <para>ImportTsv is a utility that will load data in TSV format into HBase. It has two distinct usages: loading data from TSV format in HDFS
+ into HBase via Puts, and preparing StoreFiles to be loaded via the <code>completebulkload</code>.
+ </para>
+ <para>To load data via Puts (i.e., non-bulk loading):
+<programlisting>$ bin/hbase org.apache.hadoop.hbase.mapreduce.ImportTsv -Dimporttsv.columns=a,b,c <tablename> <hdfs-inputdir>
+</programlisting>
+ </para>
+ <para>To generate StoreFiles for bulk-loading:
+<programlisting>$ bin/hbase org.apache.hadoop.hbase.mapreduce.ImportTsv -Dimporttsv.columns=a,b,c -Dimporttsv.bulk.output=hdfs://storefile-outputdir <tablename> <hdfs-data-inputdir>
+</programlisting>
+ </para>
+ <para>These generated StoreFiles can be loaded into HBase via <xref linkend="completebulkload"/>.
+ </para>
+ <section xml:id="importtsv.options"><title>ImportTsv Options</title>
+ Running ImportTsv with no arguments prints brief usage information:
+<programlisting>
+Usage: importtsv -Dimporttsv.columns=a,b,c <tablename> <inputdir>
+
+Imports the given input directory of TSV data into the specified table.
+
+The column names of the TSV data must be specified using the -Dimporttsv.columns
+option. This option takes the form of comma-separated column names, where each
+column name is either a simple column family, or a columnfamily:qualifier. The special
+column name HBASE_ROW_KEY is used to designate that this column should be used
+as the row key for each imported record. You must specify exactly one column
+to be the row key, and you must specify a column name for every column that exists in the
+input data.
+
+By default importtsv will load data directly into HBase. To instead generate
+HFiles of data to prepare for a bulk data load, pass the option:
+ -Dimporttsv.bulk.output=/path/for/output
+ Note: the target table will be created with default column family descriptors if it does not already exist.
+
+Other options that may be specified with -D include:
+ -Dimporttsv.skip.bad.lines=false - fail if encountering an invalid line
+ '-Dimporttsv.separator=|' - eg separate on pipes instead of tabs
+ -Dimporttsv.timestamp=currentTimeAsLong - use the specified timestamp for the import
+ -Dimporttsv.mapper.class=my.Mapper - A user-defined Mapper to use instead of org.apache.hadoop.hbase.mapreduce.TsvImporterMapper
+</programlisting>
+ </section>
+ <section xml:id="importtsv.example"><title>ImportTsv Example</title>
+ <para>For example, assume that we are loading data into a table called 'datatsv' with a ColumnFamily called 'd' with two columns "c1" and "c2".
+ </para>
+ <para>Assume that an input file exists as follows:
+<programlisting>
+row1 c1 c2
+row2 c1 c2
+row3 c1 c2
+row4 c1 c2
+row5 c1 c2
+row6 c1 c2
+row7 c1 c2
+row8 c1 c2
+row9 c1 c2
+row10 c1 c2
+</programlisting>
+ </para>
+ <para>For ImportTsv to use this imput file, the command line needs to look like this:
+ <programlisting>
+ HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath` ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/hbase-VERSION.jar importtsv -Dimporttsv.columns=HBASE_ROW_KEY,d:c1,d:c2 -Dimporttsv.bulk.output=hdfs://storefileoutput datatsv hdfs://inputfile
+ </programlisting>
+ ... and in this example the first column is the rowkey, which is why the HBASE_ROW_KEY is used. The second and third columns in the file will be imported as "d:c1" and "d:c2", respectively.
+ </para>
+ </section>
+ <section xml:id="importtsv.warning"><title>ImportTsv Warning</title>
+ <para>If you have preparing a lot of data for bulk loading, make sure the target HBase table is pre-split appropriately.
+ </para>
+ </section>
+ <section xml:id="importtsv.also"><title>See Also</title>
+ For more information about bulk-loading HFiles into HBase, see <xref linkend="arch.bulk.load"/>
+ </section>
+ </section>
+
+ <section xml:id="completebulkload">
+ <title>CompleteBulkLoad</title>
+ <para>The <code>completebulkload</code> utility will move generated StoreFiles into an HBase table. This utility is often used
+ in conjunction with output from <xref linkend="importtsv"/>.
+ </para>
+ <para>There are two ways to invoke this utility, with explicit classname and via the driver:
+<programlisting>$ bin/hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles <hdfs://storefileoutput> <tablename>
+</programlisting>
+.. and via the Driver..
+<programlisting>HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath` ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/hbase-VERSION.jar completebulkload <hdfs://storefileoutput> <tablename>
+</programlisting>
+ </para>
+ <section xml:id="completebulkload.warning"><title>CompleteBulkLoad Warning</title>
+ <para>Data generated via MapReduce is often created with file permissions that are not compatible with the running HBase process. Assuming you're running HDFS with permissions enabled, those permissions will need to be updated before you run CompleteBulkLoad.
+ </para>
+ </section>
+ <para>For more information about bulk-loading HFiles into HBase, see <xref linkend="arch.bulk.load"/>.
+ </para>
+ </section>
<section xml:id="walplayer">
<title>WALPlayer</title>
<para>WALPlayer is a utility to replay WAL files into HBase.
</para>
- <para>The WAL can be replayed for a set of tables or all tables, and a timerange can be provided (in milliseconds). The WAL is filtered to this set of tables. The output can optionally be mapped to another set of tables.
+ <para>The WAL can be replayed for a set of tables or all tables, and a
+ timerange can be provided (in milliseconds). The WAL is filtered to
+ this set of tables. The output can optionally be mapped to another set of tables.
</para>
- <para>WALPlayer can also generate HFiles for later bulk importing, in that case only a single table and no mapping can be specified.
+ <para>WALPlayer can also generate HFiles for later bulk importing, in that case
+ only a single table and no mapping can be specified.
</para>
<para>Invoke via:
<programlisting>$ bin/hbase org.apache.hadoop.hbase.mapreduce.WALPlayer [options] <wal inputdir> <tables> [<tableMappings>]>
@@ -140,18 +270,43 @@
<programlisting>$ bin/hbase org.apache.hadoop.hbase.mapreduce.WALPlayer /backuplogdir oldTable1,oldTable2 newTable1,newTable2
</programlisting>
</para>
+ <para>
+ WALPlayer, by default, runs as a mapreduce job. To NOT run WALPlayer as a mapreduce job on your cluster,
+ force it to run all in the local process by adding the flags <code>-Dmapred.job.tracker=local</code> on the command line.
+ </para>
</section>
<section xml:id="rowcounter">
- <title>RowCounter</title>
- <para>RowCounter is a utility that will count all the rows of a table. This is a good utility to use
- as a sanity check to ensure that HBase can read all the blocks of a table if there are any concerns of metadata inconsistency.
+ <title>RowCounter and CellCounter</title>
+ <para><ulink url="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/RowCounter.html">RowCounter</ulink> is a
+ mapreduce job to count all the rows of a table. This is a good utility to use as a sanity check to ensure that HBase can read
+ all the blocks of a table if there are any concerns of metadata inconsistency. It will run the mapreduce all in a single
+ process but it will run faster if you have a MapReduce cluster in place for it to exploit.
<programlisting>$ bin/hbase org.apache.hadoop.hbase.mapreduce.RowCounter <tablename> [<column1> <column2>...]
</programlisting>
</para>
- <para>Note: caching for the input Scan is configured via <code>hbase.client.scanner.caching</code> in the job configuration.
+ <para>Note: caching for the input Scan is configured via <code>hbase.client.scanner.caching</code> in the job configuration.
+ </para>
+ <para>HBase ships another diagnostic mapreduce job called
+ <ulink url="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/CellCounter.html">CellCounter</ulink>. Like
+ RowCounter, it gathers more fine-grained statistics about your table. The statistics gathered by RowCounter are more fine-grained
+ and include:
+ <itemizedlist>
+ <listitem>Total number of rows in the table.</listitem>
+ <listitem>Total number of CFs across all rows.</listitem>
+ <listitem>Total qualifiers across all rows.</listitem>
+ <listitem>Total occurrence of each CF.</listitem>
+ <listitem>Total occurrence of each qualifier.</listitem>
+ <listitem>Total number of versions of each qualifier.</listitem>
+ </itemizedlist>
+ </para>
+ <para>The program allows you to limit the scope of the run. Provide a row regex or prefix to limit the rows to analyze. Use
+ <code>hbase.mapreduce.scan.column.family</code> to specify scanning a single column family.
+ <programlisting>$ bin/hbase org.apache.hadoop.hbase.mapreduce.CellCounter <tablename> <outputDir> [regex or prefix]</programlisting>
</para>
+ <para>Note: just like RowCounter, caching for the input Scan is configured via <code>hbase.client.scanner.caching</code> in the
+ job configuration. </para>
</section>
-
+
</section> <!-- tools -->
<section xml:id="ops.regionmgt">
@@ -161,7 +316,7 @@
<para>Major compactions can be requested via the HBase shell or <link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HBaseAdmin.html#majorCompact%28java.lang.String%29">HBaseAdmin.majorCompact</link>.
</para>
<para>Note: major compactions do NOT do region merges. See <xref linkend="compaction"/> for more information about compactions.
-
+
</para>
</section>
<section xml:id="ops.regionmgt.merge">
@@ -170,16 +325,16 @@
<programlisting>$ bin/hbase org.apache.hbase.util.Merge <tablename> <region1> <region2>
</programlisting>
<para>If you feel you have too many regions and want to consolidate them, Merge is the utility you need. Merge must
- run be done when the cluster is down.
+ run be done when the cluster is down.
See the <link xlink:href="http://ofps.oreilly.com/titles/9781449396107/performance.html">O'Reilly HBase Book</link> for
an example of usage.
</para>
- <para>Additionally, there is a Ruby script attached to <link xlink:href="https://issues.apache.org/jira/browse/HBASE-1621">HBASE-1621</link>
+ <para>Additionally, there is a Ruby script attached to <link xlink:href="https://issues.apache.org/jira/browse/HBASE-1621">HBASE-1621</link>
for region merging.
</para>
</section>
</section>
-
+
<section xml:id="node.management"><title>Node Management</title>
<section xml:id="decommission"><title>Node Decommission</title>
<para>You can stop an individual RegionServer by running the following
@@ -202,10 +357,10 @@
A downside to the above stop of a RegionServer is that regions could be offline for
a good period of time. Regions are closed in order. If many regions on the server, the
first region to close may not be back online until all regions close and after the master
- notices the RegionServer's znode gone. In HBase 0.90.2, we added facility for having
- a node gradually shed its load and then shutdown itself down. HBase 0.90.2 added the
+ notices the RegionServer's znode gone. In Apache HBase 0.90.2, we added facility for having
+ a node gradually shed its load and then shutdown itself down. Apache HBase 0.90.2 added the
<filename>graceful_stop.sh</filename> script. Here is its usage:
- <programlisting>$ ./bin/graceful_stop.sh
+ <programlisting>$ ./bin/graceful_stop.sh
Usage: graceful_stop.sh [--config &conf-dir>] [--restart] [--reload] [--thrift] [--rest] &hostname>
thrift If we should stop/start thrift before/after the hbase stop/start
rest If we should stop/start rest before/after the hbase stop/start
@@ -218,7 +373,7 @@ Usage: graceful_stop.sh [--config &c
To decommission a loaded RegionServer, run the following:
<programlisting>$ ./bin/graceful_stop.sh HOSTNAME</programlisting>
where <varname>HOSTNAME</varname> is the host carrying the RegionServer
- you would decommission.
+ you would decommission.
<note><title>On <varname>HOSTNAME</varname></title>
<para>The <varname>HOSTNAME</varname> passed to <filename>graceful_stop.sh</filename>
must match the hostname that hbase is using to identify RegionServers.
@@ -240,7 +395,7 @@ Usage: graceful_stop.sh [--config &c
and because the RegionServer went down cleanly, there will be no
WAL logs to split.
<note xml:id="lb"><title>Load Balancer</title>
- <para>
+ <para>
It is assumed that the Region Load Balancer is disabled while the
<command>graceful_stop</command> script runs (otherwise the balancer
and the decommission script will end up fighting over region deployments).
@@ -252,10 +407,31 @@ This turns the balancer OFF. To reenabl
<programlisting>hbase(main):001:0> balance_switch true
false
0 row(s) in 0.3590 seconds</programlisting>
- </para>
+ </para>
</note>
</para>
- </section>
+ <section xml:id="bad.disk">
+ <title>Bad or Failing Disk</title>
+ <para>It is good having <xref linkend="dfs.datanode.failed.volumes.tolerated" /> set if you have a decent number of disks
+ per machine for the case where a disk plain dies. But usually disks do the "John Wayne" -- i.e. take a while
+ to go down spewing errors in <filename>dmesg</filename> -- or for some reason, run much slower than their
+ companions. In this case you want to decommission the disk. You have two options. You can
+ <xlink href="http://wiki.apache.org/hadoop/FAQ#I_want_to_make_a_large_cluster_smaller_by_taking_out_a_bunch_of_nodes_simultaneously._How_can_this_be_done.3F">decommission the datanode</xlink>
+ or, less disruptive in that only the bad disks data will be rereplicated, can stop the datanode,
+ unmount the bad volume (You can't umount a volume while the datanode is using it), and then restart the
+ datanode (presuming you have set dfs.datanode.failed.volumes.tolerated > 0). The regionserver will
+ throw some errors in its logs as it recalibrates where to get its data from -- it will likely
+ roll its WAL log too -- but in general but for some latency spikes, it should keep on chugging.
+ <note>
+ <para>If you are doing short-circuit reads, you will have to move the regions off the regionserver
+ before you stop the datanode; when short-circuiting reading, though chmod'd so regionserver cannot
+ have access, because it already has the files open, it will be able to keep reading the file blocks
+ from the bad disk even though the datanode is down. Move the regions back after you restart the
+ datanode.</para>
+ </note>
+ </para>
+ </section>
+ </section>
<section xml:id="rolling">
<title>Rolling Restart</title>
<para>
@@ -313,7 +489,7 @@ false
</section> <!-- node mgt -->
<section xml:id="hbase_metrics">
- <title>Metrics</title>
+ <title>HBase Metrics</title>
<section xml:id="metric_setup">
<title>Metric Setup</title>
<para>See <link xlink:href="http://hbase.apache.org/metrics.html">Metrics</link> for
@@ -394,8 +570,37 @@ false
<section xml:id="ops.monitoring">
<title >HBase Monitoring</title>
- <para>TODO
- </para>
+ <section xml:id="ops.monitoring.overview">
+ <title>Overview</title>
+ <para>The following metrics are arguably the most important to monitor for each RegionServer for
+ "macro monitoring", preferably with a system like <link xlink:href="http://opentsdb.net/">OpenTSDB</link>.
+ If your cluster is having performance issues it's likely that you'll see something unusual with
+ this group.
+ </para>
+ <para>HBase:
+ <itemizedlist>
+ <listitem>Requests</listitem>
+ <listitem>Compactions queue</listitem>
+ </itemizedlist>
+ </para>
+ <para>OS:
+ <itemizedlist>
+ <listitem>IO Wait</listitem>
+ <listitem>User CPU</listitem>
+ </itemizedlist>
+ </para>
+ <para>Java:
+ <itemizedlist>
+ <listitem>GC</listitem>
+ </itemizedlist>
+ </para>
+ <para>
+ </para>
+ <para>
+ For more information on HBase metrics, see <xref linkend="hbase_metrics"/>.
+ </para>
+ </section>
+
<section xml:id="ops.slow.query">
<title>Slow Query Log</title>
<para>The HBase slow query log consists of parseable JSON structures describing the properties of those client operations (Gets, Puts, Deletes, etc.) that either took too long to run, or produced too much output. The thresholds for "too long to run" and "too much output" are configurable, as described below. The output is produced inline in the main region server logs so that it is easy to discover further details from context with other logged events. It is also prepended with identifying tags <constant>(responseTooSlow)</constant>, <constant>(responseTooLarge)</constant>, <constant>(operationTooSlow)</constant>, and <constant>(operationTooLarge)</constant> in order to enable easy filtering with grep, in case the user desires to see only slow queries.
@@ -442,7 +647,7 @@ false
</section>
-
+
<section xml:id="cluster_replication">
<title>Cluster Replication</title>
<para>See <link xlink:href="http://hbase.apache.org/replication.html">Cluster Replication</link>.
@@ -450,8 +655,8 @@ false
</section>
<section xml:id="ops.backup">
<title >HBase Backup</title>
- <para>There are two broad strategies for performing HBase backups: backing up with a full cluster shutdown, and backing up on a live cluster.
- Each approach has pros and cons.
+ <para>There are two broad strategies for performing HBase backups: backing up with a full cluster shutdown, and backing up on a live cluster.
+ Each approach has pros and cons.
</para>
<para>For additional information, see <link xlink:href="http://blog.sematext.com/2011/03/11/hbase-backup-options/">HBase Backup Options</link> over on the Sematext Blog.
</para>
@@ -465,27 +670,27 @@ false
</para>
</section>
<section xml:id="ops.backup.fullshutdown.distcp"><title>Distcp</title>
- <para>Distcp could be used to either copy the contents of the HBase directory in HDFS to either the same cluster in another directory, or
+ <para>Distcp could be used to either copy the contents of the HBase directory in HDFS to either the same cluster in another directory, or
to a different cluster.
</para>
- <para>Note: Distcp works in this situation because the cluster is down and there are no in-flight edits to files.
+ <para>Note: Distcp works in this situation because the cluster is down and there are no in-flight edits to files.
Distcp-ing of files in the HBase directory is not generally recommended on a live cluster.
</para>
</section>
<section xml:id="ops.backup.fullshutdown.restore"><title>Restore (if needed)</title>
- <para>The backup of the hbase directory from HDFS is copied onto the 'real' hbase directory via distcp. The act of copying these files
+ <para>The backup of the hbase directory from HDFS is copied onto the 'real' hbase directory via distcp. The act of copying these files
creates new HDFS metadata, which is why a restore of the NameNode edits from the time of the HBase backup isn't required for this kind of
restore, because it's a restore (via distcp) of a specific HDFS directory (i.e., the HBase part) not the entire HDFS file-system.
</para>
</section>
</section>
<section xml:id="ops.backup.live.replication"><title>Live Cluster Backup - Replication</title>
- <para>This approach assumes that there is a second cluster.
+ <para>This approach assumes that there is a second cluster.
See the HBase page on <link xlink:href="http://hbase.apache.org/replication.html">replication</link> for more information.
</para>
</section>
<section xml:id="ops.backup.live.copytable"><title>Live Cluster Backup - CopyTable</title>
- <para>The <xref linkend="copytable" /> utility could either be used to copy data from one table to another on the
+ <para>The <xref linkend="copytable" /> utility could either be used to copy data from one table to another on the
same cluster, or to copy data to another table on another cluster.
</para>
<para>Since the cluster is up, there is a risk that edits could be missed in the copy process.
@@ -506,10 +711,10 @@ false
with a solid understanding of how HBase handles data internally (KeyValue).
</para>
<section xml:id="ops.capacity.storage.kv"><title>KeyValue</title>
- <para>HBase storage will be dominated by KeyValues. See <xref linkend="keyvalue" /> and <xref linkend="keysize" /> for
- how HBase stores data internally.
+ <para>HBase storage will be dominated by KeyValues. See <xref linkend="keyvalue" /> and <xref linkend="keysize" /> for
+ how HBase stores data internally.
</para>
- <para>It is critical to understand that there is a KeyValue instance for every attribute stored in a row, and the
+ <para>It is critical to understand that there is a KeyValue instance for every attribute stored in a row, and the
rowkey-length, ColumnFamily name-length and attribute lengths will drive the size of the database more than any other
factor.
</para>
Modified: hbase/branches/0.94/src/docbkx/performance.xml
URL: http://svn.apache.org/viewvc/hbase/branches/0.94/src/docbkx/performance.xml?rev=1455996&r1=1455995&r2=1455996&view=diff
==============================================================================
--- hbase/branches/0.94/src/docbkx/performance.xml (original)
+++ hbase/branches/0.94/src/docbkx/performance.xml Wed Mar 13 15:20:19 2013
@@ -26,7 +26,7 @@
* limitations under the License.
*/
-->
- <title>Performance Tuning</title>
+ <title>Apache HBase (TM) Performance Tuning</title>
<section xml:id="perf.os">
<title>Operating System</title>
@@ -47,7 +47,7 @@
<title>Network</title>
<para>
Perhaps the most important factor in avoiding network issues degrading Hadoop and HBbase performance is the switching hardware
- that is used, decisions made early in the scope of the project can cause major problems when you double or triple the size of your cluster (or more).
+ that is used, decisions made early in the scope of the project can cause major problems when you double or triple the size of your cluster (or more).
</para>
<para>
Important items to consider:
@@ -59,15 +59,15 @@
</para>
<section xml:id="perf.network.1switch">
<title>Single Switch</title>
- <para>The single most important factor in this configuration is that the switching capacity of the hardware is capable of
+ <para>The single most important factor in this configuration is that the switching capacity of the hardware is capable of
handling the traffic which can be generated by all systems connected to the switch. Some lower priced commodity hardware
- can have a slower switching capacity than could be utilized by a full switch.
+ can have a slower switching capacity than could be utilized by a full switch.
</para>
</section>
<section xml:id="perf.network.2switch">
<title>Multiple Switches</title>
<para>Multiple switches are a potential pitfall in the architecture. The most common configuration of lower priced hardware is a
- simple 1Gbps uplink from one switch to another. This often overlooked pinch point can easily become a bottleneck for cluster communication.
+ simple 1Gbps uplink from one switch to another. This often overlooked pinch point can easily become a bottleneck for cluster communication.
Especially with MapReduce jobs that are both reading and writing a lot of data the communication across this uplink could be saturated.
</para>
<para>Mitigation of this issue is fairly simple and can be accomplished in multiple ways:
@@ -85,22 +85,27 @@
<listitem>Poor switch capacity performance</listitem>
<listitem>Insufficient uplink to another rack</listitem>
</itemizedlist>
- If the the switches in your rack have appropriate switching capacity to handle all the hosts at full speed, the next most likely issue will be caused by homing
+ If the the switches in your rack have appropriate switching capacity to handle all the hosts at full speed, the next most likely issue will be caused by homing
more of your cluster across racks. The easiest way to avoid issues when spanning multiple racks is to use port trunking to create a bonded uplink to other racks.
The downside of this method however, is in the overhead of ports that could potentially be used. An example of this is, creating an 8Gbps port channel from rack
- A to rack B, using 8 of your 24 ports to communicate between racks gives you a poor ROI, using too few however can mean you're not getting the most out of your cluster.
+ A to rack B, using 8 of your 24 ports to communicate between racks gives you a poor ROI, using too few however can mean you're not getting the most out of your cluster.
</para>
<para>Using 10Gbe links between racks will greatly increase performance, and assuming your switches support a 10Gbe uplink or allow for an expansion card will allow you to
save your ports for machines as opposed to uplinks.
</para>
-
+ </section>
+ <section xml:id="perf.network.ints">
+ <title>Network Interfaces</title>
+ <para>Are all the network interfaces functioning correctly? Are you sure? See the Troubleshooting Case Study in <xref linkend="casestudies.slownode"/>.
+ </para>
</section>
</section> <!-- network -->
+
<section xml:id="jvm">
<title>Java</title>
<section xml:id="gc">
- <title>The Garbage Collector and HBase</title>
+ <title>The Garbage Collector and Apache HBase</title>
<section xml:id="gcpause">
<title>Long GC pauses</title>
@@ -117,13 +122,20 @@
threshold, the more GCing is done, the more CPU used). To address the
second fragmentation issue, Todd added an experimental facility,
<indexterm><primary>MSLAB</primary></indexterm>, that
- must be explicitly enabled in HBase 0.90.x (Its defaulted to be on in
- 0.92.x HBase). See <code>hbase.hregion.memstore.mslab.enabled</code>
+ must be explicitly enabled in Apache HBase 0.90.x (Its defaulted to be on in
+ Apache 0.92.x HBase). See <code>hbase.hregion.memstore.mslab.enabled</code>
to true in your <classname>Configuration</classname>. See the cited
slides for background and detail<footnote><para>The latest jvms do better
regards fragmentation so make sure you are running a recent release.
Read down in the message,
- <link xlink:href="http://osdir.com/ml/hotspot-gc-use/2011-11/msg00002.html">Identifying concurrent mode failures caused by fragmentation</link>.</para></footnote>.</para>
+ <link xlink:href="http://osdir.com/ml/hotspot-gc-use/2011-11/msg00002.html">Identifying concurrent mode failures caused by fragmentation</link>.</para></footnote>.
+ Be aware that when enabled, each MemStore instance will occupy at least
+ an MSLAB instance of memory. If you have thousands of regions or lots
+ of regions each with many column families, this allocation of MSLAB
+ may be responsible for a good portion of your heap allocation and in
+ an extreme case cause you to OOME. Disable MSLAB in this case, or
+ lower the amount of memory it uses or float less regions per server.
+ </para>
<para>For more information about GC logs, see <xref linkend="trouble.log.gc" />.
</para>
</section>
@@ -135,6 +147,7 @@
<para>See <xref linkend="recommended_configurations" />.</para>
+
<section xml:id="perf.number.of.regions">
<title>Number of Regions</title>
@@ -153,41 +166,52 @@
<section xml:id="perf.handlers">
<title><varname>hbase.regionserver.handler.count</varname></title>
- <para>See <xref linkend="hbase.regionserver.handler.count"/>.
+ <para>See <xref linkend="hbase.regionserver.handler.count"/>.
</para>
</section>
<section xml:id="perf.hfile.block.cache.size">
<title><varname>hfile.block.cache.size</varname></title>
- <para>See <xref linkend="hfile.block.cache.size"/>.
+ <para>See <xref linkend="hfile.block.cache.size"/>.
A memory setting for the RegionServer process.
</para>
- </section>
+ </section>
<section xml:id="perf.rs.memstore.upperlimit">
<title><varname>hbase.regionserver.global.memstore.upperLimit</varname></title>
- <para>See <xref linkend="hbase.regionserver.global.memstore.upperLimit"/>.
+ <para>See <xref linkend="hbase.regionserver.global.memstore.upperLimit"/>.
This memory setting is often adjusted for the RegionServer process depending on needs.
</para>
- </section>
+ </section>
<section xml:id="perf.rs.memstore.lowerlimit">
<title><varname>hbase.regionserver.global.memstore.lowerLimit</varname></title>
- <para>See <xref linkend="hbase.regionserver.global.memstore.lowerLimit"/>.
+ <para>See <xref linkend="hbase.regionserver.global.memstore.lowerLimit"/>.
This memory setting is often adjusted for the RegionServer process depending on needs.
</para>
</section>
<section xml:id="perf.hstore.blockingstorefiles">
<title><varname>hbase.hstore.blockingStoreFiles</varname></title>
- <para>See <xref linkend="hbase.hstore.blockingStoreFiles"/>.
+ <para>See <xref linkend="hbase.hstore.blockingStoreFiles"/>.
If there is blocking in the RegionServer logs, increasing this can help.
</para>
</section>
<section xml:id="perf.hregion.memstore.block.multiplier">
<title><varname>hbase.hregion.memstore.block.multiplier</varname></title>
- <para>See <xref linkend="hbase.hregion.memstore.block.multiplier"/>.
- If there is enough RAM, increasing this can help.
+ <para>See <xref linkend="hbase.hregion.memstore.block.multiplier"/>.
+ If there is enough RAM, increasing this can help.
+ </para>
+ </section>
+ <section xml:id="hbase.regionserver.checksum.verify">
+ <title><varname>hbase.regionserver.checksum.verify</varname></title>
+ <para>Have HBase write the checksum into the datablock and save
+ having to do the checksum seek whenever you read. See the
+ release note on <link xlink:href="https://issues.apache.org/jira/browse/HBASE-5074">HBASE-5074 support checksums in HBase block cache</link>.
</para>
</section>
</section>
+
+
+
+
<section xml:id="perf.zookeeper">
<title>ZooKeeper</title>
<para>See <xref linkend="zookeeper"/> for information on configuring ZooKeeper, and see the part
@@ -196,19 +220,19 @@
</section>
<section xml:id="perf.schema">
<title>Schema Design</title>
-
+
<section xml:id="perf.number.of.cfs">
<title>Number of Column Families</title>
<para>See <xref linkend="number.of.cfs" />.</para>
</section>
<section xml:id="perf.schema.keys">
<title>Key and Attribute Lengths</title>
- <para>See <xref linkend="keysize" />. See also <xref linkend="perf.compression.however" /> for
+ <para>See <xref linkend="keysize" />. See also <xref linkend="perf.compression.however" /> for
compression caveats.</para>
</section>
<section xml:id="schema.regionsize"><title>Table RegionSize</title>
<para>The regionsize can be set on a per-table basis via <code>setFileSize</code> on
- <link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/HTableDescriptor.html">HTableDescriptor</link> in the
+ <link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/HTableDescriptor.html">HTableDescriptor</link> in the
event where certain tables require different regionsizes than the configured default regionsize.
</para>
<para>See <xref linkend="perf.number.of.regions"/> for more information.
@@ -224,22 +248,23 @@
on each insert. If <varname>ROWCOL</varname>, the hash of the row +
column family + column family qualifier will be added to the bloom on
each key insert.</para>
- <para>See <link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/HColumnDescriptor.html">HColumnDescriptor</link> and
- <xref linkend="blooms"/> for more information.
+ <para>See <link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/HColumnDescriptor.html">HColumnDescriptor</link> and
+ <xref linkend="blooms"/> for more information or this answer up in quora,
+<link xlink:href="http://www.quora.com/How-are-bloom-filters-used-in-HBase">How are bloom filters used in HBase?</link>.
</para>
</section>
<section xml:id="schema.cf.blocksize"><title>ColumnFamily BlockSize</title>
- <para>The blocksize can be configured for each ColumnFamily in a table, and this defaults to 64k. Larger cell values require larger blocksizes.
+ <para>The blocksize can be configured for each ColumnFamily in a table, and this defaults to 64k. Larger cell values require larger blocksizes.
There is an inverse relationship between blocksize and the resulting StoreFile indexes (i.e., if the blocksize is doubled then the resulting
indexes should be roughly halved).
</para>
- <para>See <link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/HColumnDescriptor.html">HColumnDescriptor</link>
+ <para>See <link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/HColumnDescriptor.html">HColumnDescriptor</link>
and <xref linkend="store"/>for more information.
</para>
</section>
<section xml:id="cf.in.memory">
<title>In-Memory ColumnFamilies</title>
- <para>ColumnFamilies can optionally be defined as in-memory. Data is still persisted to disk, just like any other ColumnFamily.
+ <para>ColumnFamilies can optionally be defined as in-memory. Data is still persisted to disk, just like any other ColumnFamily.
In-memory blocks have the highest priority in the <xref linkend="block.cache" />, but it is not a guarantee that the entire table
will be in memory.
</para>
@@ -251,24 +276,24 @@
<para>Production systems should use compression with their ColumnFamily definitions. See <xref linkend="compression" /> for more information.
</para>
<section xml:id="perf.compression.however"><title>However...</title>
- <para>Compression deflates data <emphasis>on disk</emphasis>. When it's in-memory (e.g., in the
+ <para>Compression deflates data <emphasis>on disk</emphasis>. When it's in-memory (e.g., in the
MemStore) or on the wire (e.g., transferring between RegionServer and Client) it's inflated.
So while using ColumnFamily compression is a best practice, but it's not going to completely eliminate
- the impact of over-sized Keys, over-sized ColumnFamily names, or over-sized Column names.
+ the impact of over-sized Keys, over-sized ColumnFamily names, or over-sized Column names.
</para>
<para>See <xref linkend="keysize" /> on for schema design tips, and <xref linkend="keyvalue"/> for more information on HBase stores data internally.
- </para>
+ </para>
</section>
</section>
</section> <!-- perf schema -->
-
+
<section xml:id="perf.writing">
<title>Writing to HBase</title>
<section xml:id="perf.batch.loading">
<title>Batch Loading</title>
<para>Use the bulk load tool if you can. See
- <link xlink:href="http://hbase.apache.org/bulk-loads.html">Bulk Loads</link>.
+ <xref linkend="arch.bulk.load"/>.
Otherwise, pay attention to the below.
</para>
</section> <!-- batch loading -->
@@ -278,35 +303,27 @@
Table Creation: Pre-Creating Regions
</title>
<para>
-Tables in HBase are initially created with one region by default. For bulk imports, this means that all clients will write to the same region until it is large enough to split and become distributed across the cluster. A useful pattern to speed up the bulk import process is to pre-create empty regions. Be somewhat conservative in this, because too-many regions can actually degrade performance. An example of pre-creation using hex-keys is as follows (note: this example may need to be tweaked to the individual applications keys):
+Tables in HBase are initially created with one region by default. For bulk imports, this means that all clients will write to the same region
+until it is large enough to split and become distributed across the cluster. A useful pattern to speed up the bulk import process is to pre-create empty regions.
+ Be somewhat conservative in this, because too-many regions can actually degrade performance.
</para>
+ <para>There are two different approaches to pre-creating splits. The first approach is to rely on the default <code>HBaseAdmin</code> strategy
+ (which is implemented in <code>Bytes.split</code>)...
+ </para>
+<programlisting>
+byte[] startKey = ...; // your lowest keuy
+byte[] endKey = ...; // your highest key
+int numberOfRegions = ...; // # of regions to create
+admin.createTable(table, startKey, endKey, numberOfRegions);
+</programlisting>
+ <para>And the other approach is to define the splits yourself...
+ </para>
+<programlisting>
+byte[][] splits = ...; // create your own splits
+admin.createTable(table, splits);
+</programlisting>
<para>
-<programlisting>public static boolean createTable(HBaseAdmin admin, HTableDescriptor table, byte[][] splits)
-throws IOException {
- try {
- admin.createTable( table, splits );
- return true;
- } catch (TableExistsException e) {
- logger.info("table " + table.getNameAsString() + " already exists");
- // the table already exists...
- return false;
- }
-}
-
-public static byte[][] getHexSplits(String startKey, String endKey, int numRegions) {
- byte[][] splits = new byte[numRegions-1][];
- BigInteger lowestKey = new BigInteger(startKey, 16);
- BigInteger highestKey = new BigInteger(endKey, 16);
- BigInteger range = highestKey.subtract(lowestKey);
- BigInteger regionIncrement = range.divide(BigInteger.valueOf(numRegions));
- lowestKey = lowestKey.add(regionIncrement);
- for(int i=0; i < numRegions-1;i++) {
- BigInteger key = lowestKey.add(regionIncrement.multiply(BigInteger.valueOf(i)));
- byte[] b = String.format("%016x", key).getBytes();
- splits[i] = b;
- }
- return splits;
-}</programlisting>
+ See <xref linkend="rowkey.regionsplits"/> for issues related to understanding your keyspace and pre-creating regions.
</para>
</section>
<section xml:id="def.log.flush">
@@ -314,7 +331,7 @@ public static byte[][] getHexSplits(Stri
Table Creation: Deferred Log Flush
</title>
<para>
-The default behavior for Puts using the Write Ahead Log (WAL) is that <classname>HLog</classname> edits will be written immediately. If deferred log flush is used,
+The default behavior for Puts using the Write Ahead Log (WAL) is that <classname>HLog</classname> edits will be written immediately. If deferred log flush is used,
WAL edits are kept in memory until the flush period. The benefit is aggregated and asynchronous <classname>HLog</classname>- writes, but the potential downside is that if
the RegionServer goes down the yet-to-be-flushed edits are lost. This is safer, however, than not using WAL at all with Puts.
</para>
@@ -322,7 +339,7 @@ WAL edits are kept in memory until the f
Deferred log flush can be configured on tables via <link
xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/HTableDescriptor.html">HTableDescriptor</link>. The default value of <varname>hbase.regionserver.optionallogflushinterval</varname> is 1000ms.
</para>
- </section>
+ </section>
<section xml:id="perf.hbase.client.autoflush">
<title>HBase Client: AutoFlush</title>
@@ -348,25 +365,25 @@ Deferred log flush can be configured on
it makes little difference if your load is well distributed across the cluster.
</para>
<para>In general, it is best to use WAL for Puts, and where loading throughput
- is a concern to use <link linkend="perf.batch.loading">bulk loading</link> techniques instead.
+ is a concern to use <link linkend="perf.batch.loading">bulk loading</link> techniques instead.
</para>
</section>
<section xml:id="perf.hbase.client.regiongroup">
<title>HBase Client: Group Puts by RegionServer</title>
- <para>In addition to using the writeBuffer, grouping <classname>Put</classname>s by RegionServer can reduce the number of client RPC calls per writeBuffer flush.
+ <para>In addition to using the writeBuffer, grouping <classname>Put</classname>s by RegionServer can reduce the number of client RPC calls per writeBuffer flush.
There is a utility <classname>HTableUtil</classname> currently on TRUNK that does this, but you can either copy that or implement your own verison for
those still on 0.90.x or earlier.
</para>
- </section>
+ </section>
<section xml:id="perf.hbase.write.mr.reducer">
<title>MapReduce: Skip The Reducer</title>
<para>When writing a lot of data to an HBase table from a MR job (e.g., with <link
xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/TableOutputFormat.html">TableOutputFormat</link>), and specifically where Puts are being emitted
- from the Mapper, skip the Reducer step. When a Reducer step is used, all of the output (Puts) from the Mapper will get spooled to disk, then sorted/shuffled to other
- Reducers that will most likely be off-node. It's far more efficient to just write directly to HBase.
+ from the Mapper, skip the Reducer step. When a Reducer step is used, all of the output (Puts) from the Mapper will get spooled to disk, then sorted/shuffled to other
+ Reducers that will most likely be off-node. It's far more efficient to just write directly to HBase.
</para>
- <para>For summary jobs where HBase is used as a source and a sink, then writes will be coming from the Reducer step (e.g., summarize values then write out result).
- This is a different processing problem than from the the above case.
+ <para>For summary jobs where HBase is used as a source and a sink, then writes will be coming from the Reducer step (e.g., summarize values then write out result).
+ This is a different processing problem than from the the above case.
</para>
</section>
@@ -375,16 +392,16 @@ Deferred log flush can be configured on
<para>If all your data is being written to one region at a time, then re-read the
section on processing <link linkend="timeseries">timeseries</link> data.</para>
<para>Also, if you are pre-splitting regions and all your data is <emphasis>still</emphasis> winding up in a single region even though
- your keys aren't monotonically increasing, confirm that your keyspace actually works with the split strategy. There are a
+ your keys aren't monotonically increasing, confirm that your keyspace actually works with the split strategy. There are a
variety of reasons that regions may appear "well split" but won't work with your data. As
- the HBase client communicates directly with the RegionServers, this can be obtained via
+ the HBase client communicates directly with the RegionServers, this can be obtained via
<link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HTable.html#getRegionLocation%28byte[]%29">HTable.getRegionLocation</link>.
</para>
- <para>See <xref linkend="precreate.regions"/>, as well as <xref linkend="perf.configurations"/> </para>
+ <para>See <xref linkend="precreate.regions"/>, as well as <xref linkend="perf.configurations"/> </para>
</section>
</section> <!-- writing -->
-
+
<section xml:id="perf.reading">
<title>Reading from HBase</title>
@@ -406,7 +423,7 @@ Deferred log flush can be configured on
<para>Scan settings in MapReduce jobs deserve special attention. Timeouts can result (e.g., UnknownScannerException)
in Map tasks if it takes longer to process a batch of records before the client goes back to the RegionServer for the
next set of data. This problem can occur because there is non-trivial processing occuring per row. If you process
- rows quickly, set caching higher. If you process rows more slowly (e.g., lots of transformations per row, writes),
+ rows quickly, set caching higher. If you process rows more slowly (e.g., lots of transformations per row, writes),
then set caching lower.
</para>
<para>Timeouts can also happen in a non-MapReduce use case (i.e., single threaded HBase client doing a Scan), but the
@@ -424,6 +441,13 @@ Deferred log flush can be configured on
in the input scan because attribute over-selection is a non-trivial performance penalty over large datasets.
</para>
</section>
+ <section xml:id="perf.hbase.mr.input">
+ <title>MapReduce - Input Splits</title>
+ <para>For MapReduce jobs that use HBase tables as a source, if there a pattern where the "slow" map tasks seem to
+ have the same Input Split (i.e., the RegionServer serving the data), see the
+ Troubleshooting Case Study in <xref linkend="casestudies.slownode"/>.
+ </para>
+ </section>
<section xml:id="perf.hbase.client.scannerclose">
<title>Close ResultScanners</title>
@@ -469,13 +493,103 @@ htable.close();</programlisting></para>
</section>
<section xml:id="perf.hbase.read.dist">
<title>Concurrency: Monitor Data Spread</title>
- <para>When performing a high number of concurrent reads, monitor the data spread of the target tables. If the target table(s) have
+ <para>When performing a high number of concurrent reads, monitor the data spread of the target tables. If the target table(s) have
too few regions then the reads could likely be served from too few nodes. </para>
- <para>See <xref linkend="precreate.regions"/>, as well as <xref linkend="perf.configurations"/> </para>
+ <para>See <xref linkend="precreate.regions"/>, as well as <xref linkend="perf.configurations"/> </para>
</section>
-
+ <section xml:id="blooms">
+ <title>Bloom Filters</title>
+ <para>Enabling Bloom Filters can save your having to go to disk and
+ can help improve read latencys.</para>
+ <para><link xlink:href="http://en.wikipedia.org/wiki/Bloom_filter">Bloom filters</link> were developed over in <link
+ xlink:href="https://issues.apache.org/jira/browse/HBASE-1200">HBase-1200
+ Add bloomfilters</link>.<footnote>
+ <para>For description of the development process -- why static blooms
+ rather than dynamic -- and for an overview of the unique properties
+ that pertain to blooms in HBase, as well as possible future
+ directions, see the <emphasis>Development Process</emphasis> section
+ of the document <link
+ xlink:href="https://issues.apache.org/jira/secure/attachment/12444007/Bloom_Filters_in_HBase.pdf">BloomFilters
+ in HBase</link> attached to <link
+ xlink:href="https://issues.apache.org/jira/browse/HBASE-1200">HBase-1200</link>.</para>
+ </footnote><footnote>
+ <para>The bloom filters described here are actually version two of
+ blooms in HBase. In versions up to 0.19.x, HBase had a dynamic bloom
+ option based on work done by the <link
+ xlink:href="http://www.one-lab.org">European Commission One-Lab
+ Project 034819</link>. The core of the HBase bloom work was later
+ pulled up into Hadoop to implement org.apache.hadoop.io.BloomMapFile.
+ Version 1 of HBase blooms never worked that well. Version 2 is a
+ rewrite from scratch though again it starts with the one-lab
+ work.</para>
+ </footnote></para>
+ <para>See also <xref linkend="schema.bloom" />.
+ </para>
+
+ <section xml:id="bloom_footprint">
+ <title>Bloom StoreFile footprint</title>
+
+ <para>Bloom filters add an entry to the <classname>StoreFile</classname>
+ general <classname>FileInfo</classname> data structure and then two
+ extra entries to the <classname>StoreFile</classname> metadata
+ section.</para>
+
+ <section>
+ <title>BloomFilter in the <classname>StoreFile</classname>
+ <classname>FileInfo</classname> data structure</title>
+
+ <para><classname>FileInfo</classname> has a
+ <varname>BLOOM_FILTER_TYPE</varname> entry which is set to
+ <varname>NONE</varname>, <varname>ROW</varname> or
+ <varname>ROWCOL.</varname></para>
+ </section>
+
+ <section>
+ <title>BloomFilter entries in <classname>StoreFile</classname>
+ metadata</title>
+
+ <para><varname>BLOOM_FILTER_META</varname> holds Bloom Size, Hash
+ Function used, etc. Its small in size and is cached on
+ <classname>StoreFile.Reader</classname> load</para>
+ <para><varname>BLOOM_FILTER_DATA</varname> is the actual bloomfilter
+ data. Obtained on-demand. Stored in the LRU cache, if it is enabled
+ (Its enabled by default).</para>
+ </section>
+ </section>
+ <section xml:id="config.bloom">
+ <title>Bloom Filter Configuration</title>
+ <section>
+ <title><varname>io.hfile.bloom.enabled</varname> global kill
+ switch</title>
+
+ <para><code>io.hfile.bloom.enabled</code> in
+ <classname>Configuration</classname> serves as the kill switch in case
+ something goes wrong. Default = <varname>true</varname>.</para>
+ </section>
+
+ <section>
+ <title><varname>io.hfile.bloom.error.rate</varname></title>
+
+ <para><varname>io.hfile.bloom.error.rate</varname> = average false
+ positive rate. Default = 1%. Decrease rate by ½ (e.g. to .5%) == +1
+ bit per bloom entry.</para>
+ </section>
+
+ <section>
+ <title><varname>io.hfile.bloom.max.fold</varname></title>
+
+ <para><varname>io.hfile.bloom.max.fold</varname> = guaranteed minimum
+ fold rate. Most people should leave this alone. Default = 7, or can
+ collapse to at least 1/128th of original size. See the
+ <emphasis>Development Process</emphasis> section of the document <link
+ xlink:href="https://issues.apache.org/jira/secure/attachment/12444007/Bloom_Filters_in_HBase.pdf">BloomFilters
+ in HBase</link> for more on what this option means.</para>
+ </section>
+ </section>
+ </section> <!-- bloom -->
+
</section> <!-- reading -->
-
+
<section xml:id="perf.deleting">
<title>Deleting from HBase</title>
<section xml:id="perf.deleting.queue">
@@ -503,21 +617,54 @@ htable.close();</programlisting></para>
</para>
<section xml:id="perf.hdfs.curr"><title>Current Issues With Low-Latency Reads</title>
<para>The original use-case for HDFS was batch processing. As such, there low-latency reads were historically not a priority.
- With the increased adoption of HBase this is changing, and several improvements are already in development.
- See the
+ With the increased adoption of Apache HBase this is changing, and several improvements are already in development.
+ See the
<link xlink:href="https://issues.apache.org/jira/browse/HDFS-1599">Umbrella Jira Ticket for HDFS Improvements for HBase</link>.
</para>
</section>
+ <section xml:id="perf.hdfs.configs.localread">
+ <title>Leveraging local data</title>
+<para>Since Hadoop 1.0.0 (also 0.22.1, 0.23.1, CDH3u3 and HDP 1.0) via
+<link xlink:href="https://issues.apache.org/jira/browse/HDFS-2246">HDFS-2246</link>,
+it is possible for the DFSClient to take a "short circuit" and
+read directly from disk instead of going through the DataNode when the
+data is local. What this means for HBase is that the RegionServers can
+read directly off their machine's disks instead of having to open a
+socket to talk to the DataNode, the former being generally much
+faster<footnote><para>See JD's <link xlink:href="http://files.meetup.com/1350427/hug_ebay_jdcryans.pdf">Performance Talk</link></para></footnote>.
+Also see <link xlink:href="http://search-hadoop.com/m/zV6dKrLCVh1">HBase, mail # dev - read short circuit</link> thread for
+more discussion around short circuit reads.
+</para>
+<para>To enable "short circuit" reads, you must set two configurations.
+First, the hdfs-site.xml needs to be amended. Set
+the property <varname>dfs.block.local-path-access.user</varname>
+to be the <emphasis>only</emphasis> user that can use the shortcut.
+This has to be the user that started HBase. Then in hbase-site.xml,
+set <varname>dfs.client.read.shortcircuit</varname> to be <varname>true</varname>
+</para>
+<para>
+ For optimal performance when short-circuit reads are enabled, it is recommended that HDFS checksums are disabled.
+ To maintain data integrity with HDFS checksums disabled, HBase can be configured to write its own checksums into
+ its datablocks and verify against these. See <xref linkend="hbase.regionserver.checksum.verify" />.
+</para>
+<para>
+The DataNodes need to be restarted in order to pick up the new
+configuration. Be aware that if a process started under another
+username than the one configured here also has the shortcircuit
+enabled, it will get an Exception regarding an unauthorized access but
+the data will still be read.
+</para>
+ </section>
<section xml:id="perf.hdfs.comp"><title>Performance Comparisons of HBase vs. HDFS</title>
- <para>A fairly common question on the dist-list is why HBase isn't as performant as HDFS files in a batch context (e.g., as
- a MapReduce source or sink). The short answer is that HBase is doing a lot more than HDFS (e.g., reading the KeyValues,
- returning the most current row or specified timestamps, etc.), and as such HBase is 4-5 times slower than HDFS in this
+ <para>A fairly common question on the dist-list is why HBase isn't as performant as HDFS files in a batch context (e.g., as
+ a MapReduce source or sink). The short answer is that HBase is doing a lot more than HDFS (e.g., reading the KeyValues,
+ returning the most current row or specified timestamps, etc.), and as such HBase is 4-5 times slower than HDFS in this
processing context. Not that there isn't room for improvement (and this gap will, over time, be reduced), but HDFS
will always be faster in this use-case.
</para>
</section>
</section>
-
+
<section xml:id="perf.ec2"><title>Amazon EC2</title>
<para>Performance questions are common on Amazon EC2 environments because it is a shared environment. You will
not see the same throughput as a dedicated server. In terms of running tests on EC2, run them several times for the same
@@ -527,4 +674,9 @@ htable.close();</programlisting></para>
because EC2 issues are practically a separate class of performance issues.
</para>
</section>
+
+ <section xml:id="perf.casestudy"><title>Case Studies</title>
+ <para>For Performance and Troubleshooting Case Studies, see <xref linkend="casestudies"/>.
+ </para>
+ </section>
</chapter>
Modified: hbase/branches/0.94/src/docbkx/preface.xml
URL: http://svn.apache.org/viewvc/hbase/branches/0.94/src/docbkx/preface.xml?rev=1455996&r1=1455995&r2=1455996&view=diff
==============================================================================
--- hbase/branches/0.94/src/docbkx/preface.xml (original)
+++ hbase/branches/0.94/src/docbkx/preface.xml Wed Mar 13 15:20:19 2013
@@ -33,7 +33,7 @@
Herein you will find either the definitive documentation on an HBase topic
as of its standing when the referenced HBase version shipped, or it
will point to the location in <link
- xlink:href="http://hbase.apache.org/docs/current/api/index.html">javadoc</link>,
+ xlink:href="http://hbase.apache.org/apidocs/index.html">javadoc</link>,
<link xlink:href="https://issues.apache.org/jira/browse/HBASE">JIRA</link>
or <link xlink:href="http://wiki.apache.org/hadoop/Hbase">wiki</link> where
the pertinent information can be found.</para>
Modified: hbase/branches/0.94/src/docbkx/shell.xml
URL: http://svn.apache.org/viewvc/hbase/branches/0.94/src/docbkx/shell.xml?rev=1455996&r1=1455995&r2=1455996&view=diff
==============================================================================
--- hbase/branches/0.94/src/docbkx/shell.xml (original)
+++ hbase/branches/0.94/src/docbkx/shell.xml Wed Mar 13 15:20:19 2013
@@ -26,13 +26,13 @@
* limitations under the License.
*/
-->
- <title>The HBase Shell</title>
+ <title>The Apache HBase Shell</title>
<para>
- The HBase Shell is <link xlink:href="http://jruby.org">(J)Ruby</link>'s
+ The Apache HBase (TM) Shell is <link xlink:href="http://jruby.org">(J)Ruby</link>'s
IRB with some HBase particular commands added. Anything you can do in
IRB, you should be able to do in the HBase Shell.</para>
- <para>To run the HBase shell,
+ <para>To run the HBase shell,
do as follows:
<programlisting>$ ./bin/hbase shell</programlisting>
</para>
@@ -47,7 +47,7 @@
for example basic shell operation.</para>
<section xml:id="scripting"><title>Scripting</title>
- <para>For examples scripting HBase, look in the
+ <para>For examples scripting Apache HBase, look in the
HBase <filename>bin</filename> directory. Look at the files
that end in <filename>*.rb</filename>. To run one of these
files, do as follows:
@@ -104,5 +104,16 @@
</para>
</section>
</section>
+ <section><title>Commands</title>
+ <section><title>count</title>
+ <para>Count command returns the number of rows in a table.
+ It's quite fast when configured with the right CACHE
+ <programlisting>hbase> count '<tablename>', CACHE => 1000</programlisting>
+ The above count fetches 1000 rows at a time. Set CACHE lower if your rows are big.
+ Default is to fetch one row at a time.
+ </para>
+ </section>
+ </section>
+
</section>
</chapter>