You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hbase.apache.org by st...@apache.org on 2014/11/12 00:07:06 UTC
[1/3] hbase git commit: HBASE-12455 Add 'description' to bean and
attribute output when you do /jmx?description=true
Repository: hbase
Updated Branches:
refs/heads/branch-1 8902fb0c1 -> bebf0181c
http://git-wip-us.apache.org/repos/asf/hbase/blob/d7f51e1c/src/main/docbkx/ops_mgt.xml
----------------------------------------------------------------------
diff --git a/src/main/docbkx/ops_mgt.xml b/src/main/docbkx/ops_mgt.xml
deleted file mode 100644
index 7341ead..0000000
--- a/src/main/docbkx/ops_mgt.xml
+++ /dev/null
@@ -1,1635 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<chapter
- version="5.0"
- xml:id="ops_mgt"
- xmlns="http://docbook.org/ns/docbook"
- xmlns:xlink="http://www.w3.org/1999/xlink"
- xmlns:xi="http://www.w3.org/2001/XInclude"
- xmlns:svg="http://www.w3.org/2000/svg"
- xmlns:m="http://www.w3.org/1998/Math/MathML"
- xmlns:html="http://www.w3.org/1999/xhtml"
- xmlns:db="http://docbook.org/ns/docbook">
- <!--
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work forf additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
--->
- <title>Apache HBase Operational Management</title>
- <para> This chapter will cover operational tools and practices required of a running Apache HBase
- cluster. The subject of operations is related to the topics of <xref
- linkend="trouble" />, <xref
- linkend="performance" />, and <xref
- linkend="configuration" /> but is a distinct topic in itself. </para>
-
- <section
- xml:id="tools">
- <title>HBase Tools and Utilities</title>
-
- <para>HBase provides several tools for administration, analysis, and debugging of your cluster.
- The entry-point to most of these tools is the <filename>bin/hbase</filename> command, though
- some tools are available in the <filename>dev-support/</filename> directory.</para>
- <para>To see usage instructions for <filename>bin/hbase</filename> command, run it with no
- arguments, or with the <option>-h</option> argument. These are the usage instructions for
- HBase 0.98.x. Some commands, such as <command>version</command>, <command>pe</command>,
- <command>ltt</command>, <command>clean</command>, are not available in previous
- versions.</para>
- <screen>
-$ <userinput>bin/hbase</userinput>
-<![CDATA[Usage: hbase [<options>] <command> [<args>]]]>
-Options:
- --config DIR Configuration direction to use. Default: ./conf
- --hosts HOSTS Override the list in 'regionservers' file
-
-Commands:
-Some commands take arguments. Pass no args or -h for usage.
- shell Run the HBase shell
- hbck Run the hbase 'fsck' tool
- hlog Write-ahead-log analyzer
- hfile Store file analyzer
- zkcli Run the ZooKeeper shell
- upgrade Upgrade hbase
- master Run an HBase HMaster node
- regionserver Run an HBase HRegionServer node
- zookeeper Run a Zookeeper server
- rest Run an HBase REST server
- thrift Run the HBase Thrift server
- thrift2 Run the HBase Thrift2 server
- clean Run the HBase clean up script
- classpath Dump hbase CLASSPATH
- mapredcp Dump CLASSPATH entries required by mapreduce
- pe Run PerformanceEvaluation
- ltt Run LoadTestTool
- version Print the version
- CLASSNAME Run the class named CLASSNAME
- </screen>
- <para>Some of the tools and utilities below are Java classes which are passed directly to the
- <filename>bin/hbase</filename> command, as referred to in the last line of the usage
- instructions. Others, such as <command>hbase shell</command> (<xref linkend="shell"/>),
- <command>hbase upgrade</command> (<xref linkend="upgrading"/>), and <command>hbase
- thrift</command> (<xref linkend="thrift"/>), are documented elsewhere in this guide.</para>
- <section
- xml:id="canary">
- <title>Canary</title>
- <para> There is a Canary class can help users to canary-test the HBase cluster status, with
- every column-family for every regions or regionservers granularity. To see the usage, use
- the <literal>--help</literal> parameter. </para>
- <screen>$ ${HBASE_HOME}/bin/hbase org.apache.hadoop.hbase.tool.Canary -help
-
-Usage: bin/hbase org.apache.hadoop.hbase.tool.Canary [opts] [table1 [table2]...] | [regionserver1 [regionserver2]..]
- where [opts] are:
- -help Show this help and exit.
- -regionserver replace the table argument to regionserver,
- which means to enable regionserver mode
- -daemon Continuous check at defined intervals.
- -interval <N> Interval between checks (sec)
- -e Use region/regionserver as regular expression
- which means the region/regionserver is regular expression pattern
- -f <B> stop whole program if first error occurs, default is true
- -t <N> timeout for a check, default is 600000 (milliseconds)</screen>
- <para> This tool will return non zero error codes to user for collaborating with other
- monitoring tools, such as Nagios. The error code definitions are: </para>
- <programlisting>private static final int USAGE_EXIT_CODE = 1;
-private static final int INIT_ERROR_EXIT_CODE = 2;
-private static final int TIMEOUT_ERROR_EXIT_CODE = 3;
-private static final int ERROR_EXIT_CODE = 4;</programlisting>
- <para> Here are some examples based on the following given case. There are two HTable called
- test-01 and test-02, they have two column family cf1 and cf2 respectively, and deployed on
- the 3 regionservers. see following table. </para>
-
- <informaltable>
- <tgroup
- cols="3"
- align="center"
- colsep="1"
- rowsep="1">
- <colspec
- colname="regionserver"
- align="center" />
- <colspec
- colname="test-01"
- align="center" />
- <colspec
- colname="test-02"
- align="center" />
- <thead>
- <row>
- <entry>RegionServer</entry>
- <entry>test-01</entry>
- <entry>test-02</entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry>rs1</entry>
- <entry>r1</entry>
- <entry>r2</entry>
- </row>
- <row>
- <entry>rs2</entry>
- <entry>r2</entry>
- <entry />
- </row>
- <row>
- <entry>rs3</entry>
- <entry>r2</entry>
- <entry>r1</entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <para> Following are some examples based on the previous given case. </para>
- <section>
- <title>Canary test for every column family (store) of every region of every table</title>
- <screen>$ ${HBASE_HOME}/bin/hbase org.apache.hadoop.hbase.tool.Canary
-
-3/12/09 03:26:32 INFO tool.Canary: read from region test-01,,1386230156732.0e3c7d77ffb6361ea1b996ac1042ca9a. column family cf1 in 2ms
-13/12/09 03:26:32 INFO tool.Canary: read from region test-01,,1386230156732.0e3c7d77ffb6361ea1b996ac1042ca9a. column family cf2 in 2ms
-13/12/09 03:26:32 INFO tool.Canary: read from region test-01,0004883,1386230156732.87b55e03dfeade00f441125159f8ca87. column family cf1 in 4ms
-13/12/09 03:26:32 INFO tool.Canary: read from region test-01,0004883,1386230156732.87b55e03dfeade00f441125159f8ca87. column family cf2 in 1ms
-...
-13/12/09 03:26:32 INFO tool.Canary: read from region test-02,,1386559511167.aa2951a86289281beee480f107bb36ee. column family cf1 in 5ms
-13/12/09 03:26:32 INFO tool.Canary: read from region test-02,,1386559511167.aa2951a86289281beee480f107bb36ee. column family cf2 in 3ms
-13/12/09 03:26:32 INFO tool.Canary: read from region test-02,0004883,1386559511167.cbda32d5e2e276520712d84eaaa29d84. column family cf1 in 31ms
-13/12/09 03:26:32 INFO tool.Canary: read from region test-02,0004883,1386559511167.cbda32d5e2e276520712d84eaaa29d84. column family cf2 in 8ms
-</screen>
- <para> So you can see, table test-01 has two regions and two column families, so the Canary
- tool will pick 4 small piece of data from 4 (2 region * 2 store) different stores. This is
- a default behavior of the this tool does. </para>
- </section>
-
- <section>
- <title>Canary test for every column family (store) of every region of specific
- table(s)</title>
- <para> You can also test one or more specific tables.</para>
- <screen>$ ${HBASE_HOME}/bin/hbase orghapache.hadoop.hbase.tool.Canary test-01 test-02</screen>
- </section>
-
- <section>
- <title>Canary test with regionserver granularity</title>
- <para> This will pick one small piece of data from each regionserver, and can also put your
- resionserver name as input options for canary-test specific regionservers.</para>
- <screen>$ ${HBASE_HOME}/bin/hbase org.apache.hadoop.hbase.tool.Canary -regionserver
-
-13/12/09 06:05:17 INFO tool.Canary: Read from table:test-01 on region server:rs2 in 72ms
-13/12/09 06:05:17 INFO tool.Canary: Read from table:test-02 on region server:rs3 in 34ms
-13/12/09 06:05:17 INFO tool.Canary: Read from table:test-01 on region server:rs1 in 56ms</screen>
- </section>
- <section>
- <title>Canary test with regular expression pattern</title>
- <para> This will test both table test-01 and test-02.</para>
- <screen>$ ${HBASE_HOME}/bin/hbase orghapache.hadoop.hbase.tool.Canary -e test-0[1-2]</screen>
- </section>
-
- <section>
- <title>Run canary test as daemon mode</title>
- <para> Run repeatedly with interval defined in option -interval whose default value is 6
- seconds. This daemon will stop itself and return non-zero error code if any error occurs,
- due to the default value of option -f is true.</para>
- <screen>$ ${HBASE_HOME}/bin/hbase orghapache.hadoop.hbase.tool.Canary -daemon</screen>
- <para>Run repeatedly with internal 5 seconds and will not stop itself even error occurs in
- the test.</para>
- <screen>$ ${HBASE_HOME}/bin/hbase orghapache.hadoop.hbase.tool.Canary -daemon -interval 50000 -f false</screen>
- </section>
-
- <section>
- <title>Force timeout if canary test stuck</title>
- <para>In some cases, we suffered the request stucked on the regionserver and not response
- back to the client. The regionserver in problem, would also not indicated to be dead by
- Master, which would bring the clients hung. So we provide the timeout option to kill the
- canary test forcefully and return non-zero error code as well. This run sets the timeout
- value to 60 seconds, the default value is 600 seconds.</para>
- <screen>$ ${HBASE_HOME}/bin/hbase orghapache.hadoop.hbase.tool.Canary -t 600000</screen>
- </section>
-
- </section>
-
- <section
- xml:id="health.check">
- <title>Health Checker</title>
- <para>You can configure HBase to run a script on a period and if it fails N times
- (configurable), have the server exit. See <link
- xlink:href="">HBASE-7351 Periodic health check script</link> for configurations and
- detail. </para>
- </section>
-
- <section
- xml:id="driver">
- <title>Driver</title>
- <para>Several frequently-accessed utilities are provided as <code>Driver</code> classes, and executed by
- the <filename>bin/hbase</filename> command. These utilities represent MapReduce jobs which
- run on your cluster. They are run in the following way, replacing
- <replaceable>UtilityName</replaceable> with the utility you want to run. This command
- assumes you have set the environment variable <literal>HBASE_HOME</literal> to the directory
- where HBase is unpacked on your server.</para>
- <screen>
-${HBASE_HOME}/bin/hbase org.apache.hadoop.hbase.mapreduce.<replaceable>UtilityName</replaceable>
- </screen>
- <para>The following utilities are available:</para>
- <variablelist>
- <varlistentry>
- <term><command>LoadIncrementalHFiles</command></term>
- <listitem><para>Complete a bulk data load.</para></listitem>
- </varlistentry>
- <varlistentry>
- <term><command>CopyTable</command></term>
- <listitem><para>Export a table from the local cluster to a peer cluster.</para></listitem>
- </varlistentry>
- <varlistentry>
- <term><command>Export</command></term>
- <listitem><para>Write table data to HDFS.</para></listitem>
- </varlistentry>
- <varlistentry>
- <term><command>Import</command></term>
- <listitem><para>Import data written by a previous <command>Export</command> operation.</para></listitem>
- </varlistentry>
- <varlistentry>
- <term><command>ImportTsv</command></term>
- <listitem><para>Import data in TSV format.</para></listitem>
- </varlistentry>
- <varlistentry>
- <term><command>RowCounter</command></term>
- <listitem><para>Count rows in an HBase table.</para></listitem>
- </varlistentry>
- <varlistentry>
- <term><command>replication.VerifyReplication</command></term>
- <listitem><para>Compare the data from tables in two different clusters. WARNING: It
- doesn't work for incrementColumnValues'd cells since the timestamp is changed. Note that
- this command is in a different package than the others.</para></listitem>
- </varlistentry>
- </variablelist>
- <para>Each command except <command>RowCounter</command> accepts a single
- <literal>--help</literal> argument to print usage instructions.</para>
- </section>
- <section
- xml:id="hbck">
- <title>HBase <application>hbck</application></title>
- <subtitle>An <command>fsck</command> for your HBase install</subtitle>
- <para>To run <application>hbck</application> against your HBase cluster run <command>$
- ./bin/hbase hbck</command> At the end of the command's output it prints
- <literal>OK</literal> or <literal>INCONSISTENCY</literal>. If your cluster reports
- inconsistencies, pass <command>-details</command> to see more detail emitted. If
- inconsistencies, run <command>hbck</command> a few times because the inconsistency may be
- transient (e.g. cluster is starting up or a region is splitting). Passing
- <command>-fix</command> may correct the inconsistency (This latter is an experimental
- feature). </para>
- <para>For more information, see <xref
- linkend="hbck.in.depth" />. </para>
- </section>
- <section
- xml:id="hfile_tool2">
- <title>HFile Tool</title>
- <para>See <xref
- linkend="hfile_tool" />.</para>
- </section>
- <section
- xml:id="wal_tools">
- <title>WAL Tools</title>
-
- <section
- xml:id="hlog_tool">
- <title><classname>FSHLog</classname> tool</title>
-
- <para>The main method on <classname>FSHLog</classname> offers manual split and dump
- facilities. Pass it WALs or the product of a split, the content of the
- <filename>recovered.edits</filename>. directory.</para>
-
- <para>You can get a textual dump of a WAL file content by doing the following:</para>
- <screen> $ ./bin/hbase org.apache.hadoop.hbase.regionserver.wal.FSHLog --dump hdfs://example.org:8020/hbase/.logs/example.org,60020,1283516293161/10.10.21.10%3A60020.1283973724012 </screen>
- <para>The return code will be non-zero if issues with the file so you can test wholesomeness
- of file by redirecting <varname>STDOUT</varname> to <code>/dev/null</code> and testing the
- program return.</para>
-
- <para>Similarly you can force a split of a log file directory by doing:</para>
- <screen> $ ./bin/hbase org.apache.hadoop.hbase.regionserver.wal.FSHLog --split hdfs://example.org:8020/hbase/.logs/example.org,60020,1283516293161/</screen>
-
- <section
- xml:id="hlog_tool.prettyprint">
- <title><classname>HLogPrettyPrinter</classname></title>
- <para><classname>HLogPrettyPrinter</classname> is a tool with configurable options to
- print the contents of an HLog. </para>
- </section>
-
- </section>
- </section>
- <section
- xml:id="compression.tool">
- <title>Compression Tool</title>
- <para>See <xref
- linkend="compression.test" />.</para>
- </section>
- <section
- xml:id="copytable">
- <title>CopyTable</title>
- <para> CopyTable is a utility that can copy part or of all of a table, either to the same
- cluster or another cluster. The target table must first exist. The usage is as
- follows:</para>
-
- <screen>
-$ <userinput>./bin/hbase org.apache.hadoop.hbase.mapreduce.CopyTable --help </userinput>
-/bin/hbase org.apache.hadoop.hbase.mapreduce.CopyTable --help
-Usage: CopyTable [general options] [--starttime=X] [--endtime=Y] [--new.name=NEW] [--peer.adr=ADR] <tablename>
-
-Options:
- rs.class hbase.regionserver.class of the peer cluster,
- specify if different from current cluster
- rs.impl hbase.regionserver.impl of the peer cluster,
- startrow the start row
- stoprow the stop row
- starttime beginning of the time range (unixtime in millis)
- without endtime means from starttime to forever
- endtime end of the time range. Ignored if no starttime specified.
- versions number of cell versions to copy
- new.name new table's name
- peer.adr Address of the peer cluster given in the format
- hbase.zookeeer.quorum:hbase.zookeeper.client.port:zookeeper.znode.parent
- families comma-separated list of families to copy
- To copy from cf1 to cf2, give sourceCfName:destCfName.
- To keep the same name, just give "cfName"
- all.cells also copy delete markers and deleted cells
-
-Args:
- tablename Name of the table to copy
-
-Examples:
- To copy 'TestTable' to a cluster that uses replication for a 1 hour window:
- $ bin/hbase org.apache.hadoop.hbase.mapreduce.CopyTable --starttime=1265875194289 --endtime=1265878794289 --peer.adr=server1,server2,server3:2181:/hbase --families=myOldCf:myNewCf,cf2,cf3 TestTable
-
-For performance consider the following general options:
- It is recommended that you set the following to >=100. A higher value uses more memory but
- decreases the round trip time to the server and may increase performance.
- -Dhbase.client.scanner.caching=100
- The following should always be set to false, to prevent writing data twice, which may produce
- inaccurate results.
- -Dmapred.map.tasks.speculative.execution=false
- </screen>
- <note>
- <title>Scanner Caching</title>
- <para>Caching for the input Scan is configured via <code>hbase.client.scanner.caching</code>
- in the job configuration. </para>
- </note>
- <note>
- <title>Versions</title>
- <para>By default, CopyTable utility only copies the latest version of row cells unless
- <code>--versions=n</code> is explicitly specified in the command. </para>
- </note>
- <para> See Jonathan Hsieh's <link
- xlink:href="http://www.cloudera.com/blog/2012/06/online-hbase-backups-with-copytable-2/">Online
- HBase Backups with CopyTable</link> blog post for more on <command>CopyTable</command>.
- </para>
- </section>
- <section
- xml:id="export">
- <title>Export</title>
- <para>Export is a utility that will dump the contents of table to HDFS in a sequence file.
- Invoke via:</para>
- <screen>$ bin/hbase org.apache.hadoop.hbase.mapreduce.Export <tablename> <outputdir> [<versions> [<starttime> [<endtime>]]]
-</screen>
-
- <para>Note: caching for the input Scan is configured via
- <code>hbase.client.scanner.caching</code> in the job configuration. </para>
- </section>
- <section
- xml:id="import">
- <title>Import</title>
- <para>Import is a utility that will load data that has been exported back into HBase. Invoke
- via:</para>
- <screen>$ bin/hbase org.apache.hadoop.hbase.mapreduce.Import <tablename> <inputdir>
-</screen>
- <para>To import 0.94 exported files in a 0.96 cluster or onwards, you need to set system
- property "hbase.import.version" when running the import command as below:</para>
- <screen>$ bin/hbase -Dhbase.import.version=0.94 org.apache.hadoop.hbase.mapreduce.Import <tablename> <inputdir>
-</screen>
- </section>
- <section
- xml:id="importtsv">
- <title>ImportTsv</title>
- <para>ImportTsv is a utility that will load data in TSV format into HBase. It has two distinct
- usages: loading data from TSV format in HDFS into HBase via Puts, and preparing StoreFiles
- to be loaded via the <code>completebulkload</code>. </para>
- <para>To load data via Puts (i.e., non-bulk loading):</para>
- <screen>$ bin/hbase org.apache.hadoop.hbase.mapreduce.ImportTsv -Dimporttsv.columns=a,b,c <tablename> <hdfs-inputdir>
-</screen>
-
- <para>To generate StoreFiles for bulk-loading:</para>
- <programlisting>$ bin/hbase org.apache.hadoop.hbase.mapreduce.ImportTsv -Dimporttsv.columns=a,b,c -Dimporttsv.bulk.output=hdfs://storefile-outputdir <tablename> <hdfs-data-inputdir>
-</programlisting>
- <para>These generated StoreFiles can be loaded into HBase via <xref
- linkend="completebulkload" />. </para>
- <section
- xml:id="importtsv.options">
- <title>ImportTsv Options</title>
- <para>Running <command>ImportTsv</command> with no arguments prints brief usage
- information:</para>
- <screen>
-Usage: importtsv -Dimporttsv.columns=a,b,c <tablename> <inputdir>
-
-Imports the given input directory of TSV data into the specified table.
-
-The column names of the TSV data must be specified using the -Dimporttsv.columns
-option. This option takes the form of comma-separated column names, where each
-column name is either a simple column family, or a columnfamily:qualifier. The special
-column name HBASE_ROW_KEY is used to designate that this column should be used
-as the row key for each imported record. You must specify exactly one column
-to be the row key, and you must specify a column name for every column that exists in the
-input data.
-
-By default importtsv will load data directly into HBase. To instead generate
-HFiles of data to prepare for a bulk data load, pass the option:
- -Dimporttsv.bulk.output=/path/for/output
- Note: the target table will be created with default column family descriptors if it does not already exist.
-
-Other options that may be specified with -D include:
- -Dimporttsv.skip.bad.lines=false - fail if encountering an invalid line
- '-Dimporttsv.separator=|' - eg separate on pipes instead of tabs
- -Dimporttsv.timestamp=currentTimeAsLong - use the specified timestamp for the import
- -Dimporttsv.mapper.class=my.Mapper - A user-defined Mapper to use instead of org.apache.hadoop.hbase.mapreduce.TsvImporterMapper
- </screen>
- </section>
- <section
- xml:id="importtsv.example">
- <title>ImportTsv Example</title>
- <para>For example, assume that we are loading data into a table called 'datatsv' with a
- ColumnFamily called 'd' with two columns "c1" and "c2". </para>
- <para>Assume that an input file exists as follows:
- <screen>
-row1 c1 c2
-row2 c1 c2
-row3 c1 c2
-row4 c1 c2
-row5 c1 c2
-row6 c1 c2
-row7 c1 c2
-row8 c1 c2
-row9 c1 c2
-row10 c1 c2
- </screen>
- </para>
- <para>For ImportTsv to use this imput file, the command line needs to look like this:</para>
- <screen>
- HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath` ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/hbase-VERSION.jar importtsv -Dimporttsv.columns=HBASE_ROW_KEY,d:c1,d:c2 -Dimporttsv.bulk.output=hdfs://storefileoutput datatsv hdfs://inputfile
- </screen>
- <para> ... and in this example the first column is the rowkey, which is why the
- HBASE_ROW_KEY is used. The second and third columns in the file will be imported as "d:c1"
- and "d:c2", respectively. </para>
- </section>
- <section
- xml:id="importtsv.warning">
- <title>ImportTsv Warning</title>
- <para>If you have preparing a lot of data for bulk loading, make sure the target HBase table
- is pre-split appropriately. </para>
- </section>
- <section
- xml:id="importtsv.also">
- <title>See Also</title>
- <para>For more information about bulk-loading HFiles into HBase, see <xref
- linkend="arch.bulk.load" /></para>
- </section>
- </section>
-
- <section
- xml:id="completebulkload">
- <title>CompleteBulkLoad</title>
- <para>The <code>completebulkload</code> utility will move generated StoreFiles into an HBase
- table. This utility is often used in conjunction with output from <xref
- linkend="importtsv" />. </para>
- <para>There are two ways to invoke this utility, with explicit classname and via the
- driver:</para>
- <screen>$ bin/hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles <hdfs://storefileoutput> <tablename>
-</screen>
- <para> .. and via the Driver..</para>
- <screen>HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath` ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/hbase-VERSION.jar completebulkload <hdfs://storefileoutput> <tablename>
-</screen>
- <section
- xml:id="completebulkload.warning">
- <title>CompleteBulkLoad Warning</title>
- <para>Data generated via MapReduce is often created with file permissions that are not
- compatible with the running HBase process. Assuming you're running HDFS with permissions
- enabled, those permissions will need to be updated before you run CompleteBulkLoad.</para>
- <para>For more information about bulk-loading HFiles into HBase, see <xref
- linkend="arch.bulk.load" />. </para>
- </section>
-
- </section>
- <section
- xml:id="walplayer">
- <title>WALPlayer</title>
- <para>WALPlayer is a utility to replay WAL files into HBase. </para>
- <para>The WAL can be replayed for a set of tables or all tables, and a timerange can be
- provided (in milliseconds). The WAL is filtered to this set of tables. The output can
- optionally be mapped to another set of tables. </para>
- <para>WALPlayer can also generate HFiles for later bulk importing, in that case only a single
- table and no mapping can be specified. </para>
- <para>Invoke via:</para>
- <screen>$ bin/hbase org.apache.hadoop.hbase.mapreduce.WALPlayer [options] <wal inputdir> <tables> [<tableMappings>]>
-</screen>
- <para>For example:</para>
- <screen>$ bin/hbase org.apache.hadoop.hbase.mapreduce.WALPlayer /backuplogdir oldTable1,oldTable2 newTable1,newTable2
-</screen>
- <para> WALPlayer, by default, runs as a mapreduce job. To NOT run WALPlayer as a mapreduce job
- on your cluster, force it to run all in the local process by adding the flags
- <code>-Dmapreduce.jobtracker.address=local</code> on the command line. </para>
- </section>
- <section
- xml:id="rowcounter">
- <title>RowCounter and CellCounter</title>
- <para><link
- xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/RowCounter.html">RowCounter</link>
- is a mapreduce job to count all the rows of a table. This is a good utility to use as a
- sanity check to ensure that HBase can read all the blocks of a table if there are any
- concerns of metadata inconsistency. It will run the mapreduce all in a single process but it
- will run faster if you have a MapReduce cluster in place for it to exploit.</para>
- <screen>$ bin/hbase org.apache.hadoop.hbase.mapreduce.RowCounter <tablename> [<column1> <column2>...]
-</screen>
- <para>Note: caching for the input Scan is configured via
- <code>hbase.client.scanner.caching</code> in the job configuration. </para>
- <para>HBase ships another diagnostic mapreduce job called <link
- xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/CellCounter.html">CellCounter</link>.
- Like RowCounter, it gathers more fine-grained statistics about your table. The statistics
- gathered by RowCounter are more fine-grained and include: </para>
- <itemizedlist>
- <listitem>
- <para>Total number of rows in the table.</para>
- </listitem>
- <listitem>
- <para>Total number of CFs across all rows.</para>
- </listitem>
- <listitem>
- <para>Total qualifiers across all rows.</para>
- </listitem>
- <listitem>
- <para>Total occurrence of each CF.</para>
- </listitem>
- <listitem>
- <para>Total occurrence of each qualifier.</para>
- </listitem>
- <listitem>
- <para>Total number of versions of each qualifier.</para>
- </listitem>
- </itemizedlist>
- <para>The program allows you to limit the scope of the run. Provide a row regex or prefix to
- limit the rows to analyze. Use <code>hbase.mapreduce.scan.column.family</code> to specify
- scanning a single column family.</para>
- <screen>$ bin/hbase org.apache.hadoop.hbase.mapreduce.CellCounter <tablename> <outputDir> [regex or prefix]</screen>
- <para>Note: just like RowCounter, caching for the input Scan is configured via
- <code>hbase.client.scanner.caching</code> in the job configuration. </para>
- </section>
- <section
- xml:id="mlockall">
- <title>mlockall</title>
- <para>It is possible to optionally pin your servers in physical memory making them less likely
- to be swapped out in oversubscribed environments by having the servers call <link
- xlink:href="http://linux.die.net/man/2/mlockall">mlockall</link> on startup. See <link
- xlink:href="https://issues.apache.org/jira/browse/HBASE-4391">HBASE-4391 Add ability to
- start RS as root and call mlockall</link> for how to build the optional library and have
- it run on startup. </para>
- </section>
- <section
- xml:id="compaction.tool">
- <title>Offline Compaction Tool</title>
- <para>See the usage for the <link
- xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/regionserver/CompactionTool.html">Compaction
- Tool</link>. Run it like this <command>./bin/hbase
- org.apache.hadoop.hbase.regionserver.CompactionTool</command>
- </para>
- </section>
-
- <section>
- <title><command>hbase clean</command></title>
- <para>The <command>hbase clean</command> command cleans HBase data from ZooKeeper, HDFS, or
- both. It is appropriate to use for testing. Run it with no options for usage instructions.
- The <command>hbase clean</command> command was introduced in HBase 0.98.</para>
- <screen>
-$ <userinput>bin/hbase clean</userinput>
-Usage: hbase clean (--cleanZk|--cleanHdfs|--cleanAll)
-Options:
- --cleanZk cleans hbase related data from zookeeper.
- --cleanHdfs cleans hbase related data from hdfs.
- --cleanAll cleans hbase related data from both zookeeper and hdfs.
- </screen>
- </section>
- <section>
- <title><command>hbase pe</command></title>
- <para>The <command>hbase pe</command> command is a shortcut provided to run the
- <code>org.apache.hadoop.hbase.PerformanceEvaluation</code> tool, which is used for
- testing. The <command>hbase pe</command> command was introduced in HBase 0.98.4.</para>
- <para>The PerformanceEvaluation tool accepts many different options and commands. For usage
- instructions, run the command with no options.</para>
- <para>To run PerformanceEvaluation prior to HBase 0.98.4, issue the command
- <command>hbase org.apache.hadoop.hbase.PerformanceEvaluation</command>.</para>
- <para>The PerformanceEvaluation tool has received many updates in recent HBase releases,
- including support for namespaces, support for tags, cell-level ACLs and visibility labels,
- multiget support for RPC calls, increased sampling sizes, an option to randomly sleep during
- testing, and ability to "warm up" the cluster before testing starts.</para>
- </section>
- <section>
- <title><command>hbase ltt</command></title>
- <para>The <command>hbase ltt</command> command is a shortcut provided to run the
- <code>org.apache.hadoop.hbase.util.LoadTestTool</code> utility, which is used for
- testing. The <command>hbase ltt</command> command was introduced in HBase 0.98.4.</para>
- <para>You must specify either <option>-write</option> or <option>-update-read</option> as the
- first option. For general usage instructions, pass the <option>-h</option> option.</para>
- <para>To run LoadTestTool prior to HBase 0.98.4, issue the command <command>hbase
- org.apache.hadoop.hbase.util.LoadTestTool</command>.</para>
- <para>The LoadTestTool has received many updates in recent HBase releases, including support
- for namespaces, support for tags, cell-level ACLS and visibility labels, testing
- security-related features, ability to specify the number of regions per server, tests for
- multi-get RPC calls, and tests relating to replication.</para>
- </section>
- </section>
- <!-- tools -->
-
- <section
- xml:id="ops.regionmgt">
- <title>Region Management</title>
- <section
- xml:id="ops.regionmgt.majorcompact">
- <title>Major Compaction</title>
- <para>Major compactions can be requested via the HBase shell or <link
- xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HBaseAdmin.html#majorCompact%28java.lang.String%29">HBaseAdmin.majorCompact</link>. </para>
- <para>Note: major compactions do NOT do region merges. See <xref
- linkend="compaction" /> for more information about compactions. </para>
- </section>
- <section
- xml:id="ops.regionmgt.merge">
- <title>Merge</title>
- <para>Merge is a utility that can merge adjoining regions in the same table (see
- org.apache.hadoop.hbase.util.Merge).</para>
- <programlisting>$ bin/hbase org.apache.hadoop.hbase.util.Merge <tablename> <region1> <region2>
-</programlisting>
- <para>If you feel you have too many regions and want to consolidate them, Merge is the utility
- you need. Merge must run be done when the cluster is down. See the <link
- xlink:href="http://ofps.oreilly.com/titles/9781449396107/performance.html">O'Reilly HBase
- Book</link> for an example of usage. </para>
- <para>You will need to pass 3 parameters to this application. The first one is the table name.
- The second one is the fully qualified name of the first region to merge, like
- "table_name,\x0A,1342956111995.7cef47f192318ba7ccc75b1bbf27a82b.". The third one is the
- fully qualified name for the second region to merge. </para>
- <para>Additionally, there is a Ruby script attached to <link
- xlink:href="https://issues.apache.org/jira/browse/HBASE-1621">HBASE-1621</link> for region
- merging. </para>
- </section>
- </section>
-
- <section
- xml:id="node.management">
- <title>Node Management</title>
- <section
- xml:id="decommission">
- <title>Node Decommission</title>
- <para>You can stop an individual RegionServer by running the following script in the HBase
- directory on the particular node:</para>
- <screen>$ ./bin/hbase-daemon.sh stop regionserver</screen>
- <para> The RegionServer will first close all regions and then shut itself down. On shutdown,
- the RegionServer's ephemeral node in ZooKeeper will expire. The master will notice the
- RegionServer gone and will treat it as a 'crashed' server; it will reassign the nodes the
- RegionServer was carrying. </para>
- <note>
- <title>Disable the Load Balancer before Decommissioning a node</title>
- <para>If the load balancer runs while a node is shutting down, then there could be
- contention between the Load Balancer and the Master's recovery of the just decommissioned
- RegionServer. Avoid any problems by disabling the balancer first. See <xref
- linkend="lb" /> below. </para>
- </note>
- <para> A downside to the above stop of a RegionServer is that regions could be offline for a
- good period of time. Regions are closed in order. If many regions on the server, the first
- region to close may not be back online until all regions close and after the master notices
- the RegionServer's znode gone. In Apache HBase 0.90.2, we added facility for having a node
- gradually shed its load and then shutdown itself down. Apache HBase 0.90.2 added the
- <filename>graceful_stop.sh</filename> script. Here is its usage:</para>
- <screen>$ ./bin/graceful_stop.sh
-Usage: graceful_stop.sh [--config &conf-dir>] [--restart] [--reload] [--thrift] [--rest] &hostname>
- thrift If we should stop/start thrift before/after the hbase stop/start
- rest If we should stop/start rest before/after the hbase stop/start
- restart If we should restart after graceful stop
- reload Move offloaded regions back on to the stopped server
- debug Move offloaded regions back on to the stopped server
- hostname Hostname of server we are to stop</screen>
- <para> To decommission a loaded RegionServer, run the following: <command>$
- ./bin/graceful_stop.sh HOSTNAME</command> where <varname>HOSTNAME</varname> is the host
- carrying the RegionServer you would decommission. </para>
- <note>
- <title>On <varname>HOSTNAME</varname></title>
- <para>The <varname>HOSTNAME</varname> passed to <filename>graceful_stop.sh</filename> must
- match the hostname that hbase is using to identify RegionServers. Check the list of
- RegionServers in the master UI for how HBase is referring to servers. Its usually hostname
- but can also be FQDN. Whatever HBase is using, this is what you should pass the
- <filename>graceful_stop.sh</filename> decommission script. If you pass IPs, the script
- is not yet smart enough to make a hostname (or FQDN) of it and so it will fail when it
- checks if server is currently running; the graceful unloading of regions will not run.
- </para>
- </note>
- <para> The <filename>graceful_stop.sh</filename> script will move the regions off the
- decommissioned RegionServer one at a time to minimize region churn. It will verify the
- region deployed in the new location before it will moves the next region and so on until the
- decommissioned server is carrying zero regions. At this point, the
- <filename>graceful_stop.sh</filename> tells the RegionServer <command>stop</command>. The
- master will at this point notice the RegionServer gone but all regions will have already
- been redeployed and because the RegionServer went down cleanly, there will be no WAL logs to
- split. </para>
- <note
- xml:id="lb">
- <title>Load Balancer</title>
- <para> It is assumed that the Region Load Balancer is disabled while the
- <command>graceful_stop</command> script runs (otherwise the balancer and the
- decommission script will end up fighting over region deployments). Use the shell to
- disable the balancer:</para>
- <programlisting>hbase(main):001:0> balance_switch false
-true
-0 row(s) in 0.3590 seconds</programlisting>
- <para> This turns the balancer OFF. To reenable, do:</para>
- <programlisting>hbase(main):001:0> balance_switch true
-false
-0 row(s) in 0.3590 seconds</programlisting>
- <para>The <command>graceful_stop</command> will check the balancer and if enabled, will turn
- it off before it goes to work. If it exits prematurely because of error, it will not have
- reset the balancer. Hence, it is better to manage the balancer apart from
- <command>graceful_stop</command> reenabling it after you are done w/ graceful_stop.
- </para>
- </note>
- <section
- xml:id="draining.servers">
- <title>Decommissioning several Regions Servers concurrently</title>
- <para>If you have a large cluster, you may want to decommission more than one machine at a
- time by gracefully stopping mutiple RegionServers concurrently. To gracefully drain
- multiple regionservers at the same time, RegionServers can be put into a "draining" state.
- This is done by marking a RegionServer as a draining node by creating an entry in
- ZooKeeper under the <filename>hbase_root/draining</filename> znode. This znode has format
- <code>name,port,startcode</code> just like the regionserver entries under
- <filename>hbase_root/rs</filename> znode. </para>
- <para>Without this facility, decommissioning mulitple nodes may be non-optimal because
- regions that are being drained from one region server may be moved to other regionservers
- that are also draining. Marking RegionServers to be in the draining state prevents this
- from happening. See this <link
- xlink:href="http://inchoate-clatter.blogspot.com/2012/03/hbase-ops-automation.html">blog
- post</link> for more details.</para>
- </section>
-
- <section
- xml:id="bad.disk">
- <title>Bad or Failing Disk</title>
- <para>It is good having <xref
- linkend="dfs.datanode.failed.volumes.tolerated" /> set if you have a decent number of
- disks per machine for the case where a disk plain dies. But usually disks do the "John
- Wayne" -- i.e. take a while to go down spewing errors in <filename>dmesg</filename> -- or
- for some reason, run much slower than their companions. In this case you want to
- decommission the disk. You have two options. You can <link
- xlink:href="http://wiki.apache.org/hadoop/FAQ#I_want_to_make_a_large_cluster_smaller_by_taking_out_a_bunch_of_nodes_simultaneously._How_can_this_be_done.3F">decommission
- the datanode</link> or, less disruptive in that only the bad disks data will be
- rereplicated, can stop the datanode, unmount the bad volume (You can't umount a volume
- while the datanode is using it), and then restart the datanode (presuming you have set
- dfs.datanode.failed.volumes.tolerated > 0). The regionserver will throw some errors in its
- logs as it recalibrates where to get its data from -- it will likely roll its WAL log too
- -- but in general but for some latency spikes, it should keep on chugging. </para>
- <note>
- <title>Short Circuit Reads</title>
- <para>If you are doing short-circuit reads, you will have to move the regions off the
- regionserver before you stop the datanode; when short-circuiting reading, though chmod'd
- so regionserver cannot have access, because it already has the files open, it will be
- able to keep reading the file blocks from the bad disk even though the datanode is down.
- Move the regions back after you restart the datanode.</para>
- </note>
- </section>
- </section>
- <section
- xml:id="rolling">
- <title>Rolling Restart</title>
-
- <para>Some cluster configuration changes require either the entire cluster, or the
- RegionServers, to be restarted in order to pick up the changes. In addition, rolling
- restarts are supported for upgrading to a minor or maintenance release, and to a major
- release if at all possible. See the release notes for release you want to upgrade to, to
- find out about limitations to the ability to perform a rolling upgrade.</para>
- <para>There are multiple ways to restart your cluster nodes, depending on your situation.
- These methods are detailed below.</para>
- <section>
- <title>Using the <command>rolling-restart.sh</command> Script</title>
-
- <para>HBase ships with a script, <filename>bin/rolling-restart.sh</filename>, that allows
- you to perform rolling restarts on the entire cluster, the master only, or the
- RegionServers only. The script is provided as a template for your own script, and is not
- explicitly tested. It requires password-less SSH login to be configured and assumes that
- you have deployed using a tarball. The script requires you to set some environment
- variables before running it. Examine the script and modify it to suit your needs.</para>
- <example>
- <title><filename>rolling-restart.sh</filename> General Usage</title>
- <screen language="bourne">
-$ <userinput>./bin/rolling-restart.sh --help</userinput><![CDATA[
-Usage: rolling-restart.sh [--config <hbase-confdir>] [--rs-only] [--master-only] [--graceful] [--maxthreads xx]
- ]]></screen>
- </example>
- <variablelist>
- <varlistentry>
- <term>Rolling Restart on RegionServers Only</term>
- <listitem>
- <para>To perform a rolling restart on the RegionServers only, use the
- <code>--rs-only</code> option. This might be necessary if you need to reboot the
- individual RegionServer or if you make a configuration change that only affects
- RegionServers and not the other HBase processes.</para>
- <para>If you need to restart only a single RegionServer, or if you need to do extra
- actions during the restart, use the <filename>bin/graceful_stop.sh</filename>
- command instead. See <xref linkend="rolling.restart.manual" />.</para>
- </listitem>
- </varlistentry>
- <varlistentry>
- <term>Rolling Restart on Masters Only</term>
- <listitem>
- <para>To perform a rolling restart on the active and backup Masters, use the
- <code>--master-only</code> option. You might use this if you know that your
- configuration change only affects the Master and not the RegionServers, or if you
- need to restart the server where the active Master is running.</para>
- <para>If you are not running backup Masters, the Master is simply restarted. If you
- are running backup Masters, they are all stopped before any are restarted, to avoid
- a race condition in ZooKeeper to determine which is the new Master. First the main
- Master is restarted, then the backup Masters are restarted. Directly after restart,
- it checks for and cleans out any regions in transition before taking on its normal
- workload.</para>
- </listitem>
- </varlistentry>
- <varlistentry>
- <term>Graceful Restart</term>
- <listitem>
- <para>If you specify the <code>--graceful</code> option, RegionServers are restarted
- using the <filename>bin/graceful_stop.sh</filename> script, which moves regions off
- a RegionServer before restarting it. This is safer, but can delay the
- restart.</para>
- </listitem>
- </varlistentry>
- <varlistentry>
- <term>Limiting the Number of Threads</term>
- <listitem>
- <para>To limit the rolling restart to using only a specific number of threads, use the
- <code>--maxthreads</code> option.</para>
- </listitem>
- </varlistentry>
- </variablelist>
- </section>
- <section xml:id="rolling.restart.manual">
- <title>Manual Rolling Restart</title>
- <para>To retain more control over the process, you may wish to manually do a rolling restart
- across your cluster. This uses the <command>graceful-stop.sh</command> command <xref
- linkend="decommission" />. In this method, you can restart each RegionServer
- individually and then move its old regions back into place, retaining locality. If you
- also need to restart the Master, you need to do it separately, and restart the Master
- before restarting the RegionServers using this method. The following is an example of such
- a command. You may need to tailor it to your environment. This script does a rolling
- restart of RegionServers only. It disables the load balancer before moving the
- regions.</para>
- <screen><![CDATA[
-$ for i in `cat conf/regionservers|sort`; do ./bin/graceful_stop.sh --restart --reload --debug $i; done &> /tmp/log.txt &;
- ]]></screen>
- <para>Monitor the output of the <filename>/tmp/log.txt</filename> file to follow the
- progress of the script. </para>
- </section>
-
- <section>
- <title>Logic for Crafting Your Own Rolling Restart Script</title>
- <para>Use the following guidelines if you want to create your own rolling restart script.</para>
- <orderedlist>
- <listitem>
- <para>Extract the new release, verify its configuration, and synchronize it to all nodes
- of your cluster using <command>rsync</command>, <command>scp</command>, or another
- secure synchronization mechanism.</para></listitem>
- <listitem><para>Use the hbck utility to ensure that the cluster is consistent.</para>
- <screen>
-$ ./bin/hbck
- </screen>
- <para>Perform repairs if required. See <xref linkend="hbck" /> for details.</para>
- </listitem>
- <listitem><para>Restart the master first. You may need to modify these commands if your
- new HBase directory is different from the old one, such as for an upgrade.</para>
- <screen>
-$ ./bin/hbase-daemon.sh stop master; ./bin/hbase-daemon.sh start master
- </screen>
- </listitem>
- <listitem><para>Gracefully restart each RegionServer, using a script such as the
- following, from the Master.</para>
- <screen><![CDATA[
-$ for i in `cat conf/regionservers|sort`; do ./bin/graceful_stop.sh --restart --reload --debug $i; done &> /tmp/log.txt &
- ]]></screen>
- <para>If you are running Thrift or REST servers, pass the --thrift or --rest options.
- For other available options, run the <command>bin/graceful-stop.sh --help</command>
- command.</para>
- <para>It is important to drain HBase regions slowly when restarting multiple
- RegionServers. Otherwise, multiple regions go offline simultaneously and must be
- reassigned to other nodes, which may also go offline soon. This can negatively affect
- performance. You can inject delays into the script above, for instance, by adding a
- Shell command such as <command>sleep</command>. To wait for 5 minutes between each
- RegionServer restart, modify the above script to the following:</para>
- <screen><![CDATA[
-$ for i in `cat conf/regionservers|sort`; do ./bin/graceful_stop.sh --restart --reload --debug $i & sleep 5m; done &> /tmp/log.txt &
- ]]></screen>
- </listitem>
- <listitem><para>Restart the Master again, to clear out the dead servers list and re-enable
- the load balancer.</para></listitem>
- <listitem><para>Run the <command>hbck</command> utility again, to be sure the cluster is
- consistent.</para></listitem>
- </orderedlist>
- </section>
- </section>
- <section
- xml:id="adding.new.node">
- <title>Adding a New Node</title>
- <para>Adding a new regionserver in HBase is essentially free, you simply start it like this:
- <command>$ ./bin/hbase-daemon.sh start regionserver</command> and it will register itself
- with the master. Ideally you also started a DataNode on the same machine so that the RS can
- eventually start to have local files. If you rely on ssh to start your daemons, don't forget
- to add the new hostname in <filename>conf/regionservers</filename> on the master. </para>
- <para>At this point the region server isn't serving data because no regions have moved to it
- yet. If the balancer is enabled, it will start moving regions to the new RS. On a
- small/medium cluster this can have a very adverse effect on latency as a lot of regions will
- be offline at the same time. It is thus recommended to disable the balancer the same way
- it's done when decommissioning a node and move the regions manually (or even better, using a
- script that moves them one by one). </para>
- <para>The moved regions will all have 0% locality and won't have any blocks in cache so the
- region server will have to use the network to serve requests. Apart from resulting in higher
- latency, it may also be able to use all of your network card's capacity. For practical
- purposes, consider that a standard 1GigE NIC won't be able to read much more than
- <emphasis>100MB/s</emphasis>. In this case, or if you are in a OLAP environment and
- require having locality, then it is recommended to major compact the moved regions. </para>
-
- </section>
- </section>
- <!-- node mgt -->
-
- <section
- xml:id="hbase_metrics">
- <title>HBase Metrics</title>
- <section
- xml:id="metric_setup">
- <title>Metric Setup</title>
- <para>See <link
- xlink:href="http://hbase.apache.org/metrics.html">Metrics</link> for an introduction and
- how to enable Metrics emission. Still valid for HBase 0.94.x. </para>
- <para>For HBase 0.95.x and up, see <link
- xlink:href="http://hadoop.apache.org/docs/current/api/org/apache/hadoop/metrics2/package-summary.html" />
- </para>
- </section>
- <section
- xml:id="rs_metrics_ganglia">
- <title>Warning To Ganglia Users</title>
- <para>Warning to Ganglia Users: by default, HBase will emit a LOT of metrics per RegionServer
- which may swamp your installation. Options include either increasing Ganglia server
- capacity, or configuring HBase to emit fewer metrics. </para>
- </section>
- <section>
- <title>Units of Measure for Metrics</title>
- <para>Different metrics are expressed in different units, as appropriate. Often, the unit of
- measure is in the name (as in the metric <code>shippedKBs</code>). Otherwise, use the
- following guidelines. When in doubt, you may need to examine the source for a given
- metric.</para>
- <itemizedlist>
- <listitem>
- <para>Metrics that refer to a point in time are usually expressed as a timestamp.</para>
- </listitem>
- <listitem>
- <para>Metrics that refer to an age (such as <code>ageOfLastShippedOp</code>) are usually
- expressed in milliseconds.</para>
- </listitem>
- <listitem>
- <para>Metrics that refer to memory sizes are in bytes.</para>
- </listitem>
- <listitem>
- <para>Sizes of queues (such as <code>sizeOfLogQueue</code>) are expressed as the number of
- items in the queue. Determine the size by multiplying by the block size (default is 64
- MB in HDFS).</para>
- </listitem>
- <listitem>
- <para>Metrics that refer to things like the number of a given type of operations (such as
- <code>logEditsRead</code>) are expressed as an integer.</para>
- </listitem>
- </itemizedlist>
- </section>
- <section xml:id="rs_metrics">
- <title>Most Important RegionServer Metrics</title>
- <para>Previously, this section contained a list of the most important RegionServer metrics.
- However, the list was extremely out of date. In some cases, the name of a given metric has
- changed. In other cases, the metric seems to no longer be exposed. An effort is underway to
- create automatic documentation for each metric based upon information pulled from its
- implementation.</para>
- </section>
- </section>
-
- <section
- xml:id="ops.monitoring">
- <title>HBase Monitoring</title>
- <section
- xml:id="ops.monitoring.overview">
- <title>Overview</title>
- <para>The following metrics are arguably the most important to monitor for each RegionServer
- for "macro monitoring", preferably with a system like <link
- xlink:href="http://opentsdb.net/">OpenTSDB</link>. If your cluster is having performance
- issues it's likely that you'll see something unusual with this group. </para>
- <itemizedlist>
- <title>HBase:</title>
- <listitem>
- <para>See <xref
- linkend="rs_metrics" /></para>
- </listitem>
- </itemizedlist>
-
- <itemizedlist>
- <title>OS:</title>
- <listitem>
- <para>IO Wait</para>
- </listitem>
- <listitem>
- <para>User CPU</para>
- </listitem>
- </itemizedlist>
- <itemizedlist>
- <title>Java:</title>
- <listitem>
- <para>GC</para>
- </listitem>
- </itemizedlist>
- <para> For more information on HBase metrics, see <xref
- linkend="hbase_metrics" />. </para>
- </section>
-
- <section
- xml:id="ops.slow.query">
- <title>Slow Query Log</title>
- <para>The HBase slow query log consists of parseable JSON structures describing the properties
- of those client operations (Gets, Puts, Deletes, etc.) that either took too long to run, or
- produced too much output. The thresholds for "too long to run" and "too much output" are
- configurable, as described below. The output is produced inline in the main region server
- logs so that it is easy to discover further details from context with other logged events.
- It is also prepended with identifying tags <constant>(responseTooSlow)</constant>,
- <constant>(responseTooLarge)</constant>, <constant>(operationTooSlow)</constant>, and
- <constant>(operationTooLarge)</constant> in order to enable easy filtering with grep, in
- case the user desires to see only slow queries. </para>
-
- <section>
- <title>Configuration</title>
- <para>There are two configuration knobs that can be used to adjust the thresholds for when
- queries are logged. </para>
-
- <itemizedlist>
- <listitem>
- <para><varname>hbase.ipc.warn.response.time</varname> Maximum number of milliseconds
- that a query can be run without being logged. Defaults to 10000, or 10 seconds. Can be
- set to -1 to disable logging by time. </para>
- </listitem>
- <listitem>
- <para><varname>hbase.ipc.warn.response.size</varname> Maximum byte size of response that
- a query can return without being logged. Defaults to 100 megabytes. Can be set to -1
- to disable logging by size. </para>
- </listitem>
- </itemizedlist>
- </section>
-
- <section>
- <title>Metrics</title>
- <para>The slow query log exposes to metrics to JMX.</para>
- <itemizedlist>
- <listitem>
- <para><varname>hadoop.regionserver_rpc_slowResponse</varname> a global metric reflecting
- the durations of all responses that triggered logging.</para>
- </listitem>
- <listitem>
- <para><varname>hadoop.regionserver_rpc_methodName.aboveOneSec</varname> A metric
- reflecting the durations of all responses that lasted for more than one second.</para>
- </listitem>
- </itemizedlist>
-
- </section>
-
- <section>
- <title>Output</title>
- <para>The output is tagged with operation e.g. <constant>(operationTooSlow)</constant> if
- the call was a client operation, such as a Put, Get, or Delete, which we expose detailed
- fingerprint information for. If not, it is tagged <constant>(responseTooSlow)</constant>
- and still produces parseable JSON output, but with less verbose information solely
- regarding its duration and size in the RPC itself. <constant>TooLarge</constant> is
- substituted for <constant>TooSlow</constant> if the response size triggered the logging,
- with <constant>TooLarge</constant> appearing even in the case that both size and duration
- triggered logging. </para>
- </section>
- <section>
- <title>Example</title>
- <para>
- <programlisting>2011-09-08 10:01:25,824 WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow): {"tables":{"riley2":{"puts":[{"totalColumns":11,"families":{"actions":[{"timestamp":1315501284459,"qualifier":"0","vlen":9667580},{"timestamp":1315501284459,"qualifier":"1","vlen":10122412},{"timestamp":1315501284459,"qualifier":"2","vlen":11104617},{"timestamp":1315501284459,"qualifier":"3","vlen":13430635}]},"row":"cfcd208495d565ef66e7dff9f98764da:0"}],"families":["actions"]}},"processingtimems":956,"client":"10.47.34.63:33623","starttimems":1315501284456,"queuetimems":0,"totalPuts":1,"class":"HRegionServer","responsesize":0,"method":"multiPut"}</programlisting>
- </para>
-
- <para>Note that everything inside the "tables" structure is output produced by MultiPut's
- fingerprint, while the rest of the information is RPC-specific, such as processing time
- and client IP/port. Other client operations follow the same pattern and the same general
- structure, with necessary differences due to the nature of the individual operations. In
- the case that the call is not a client operation, that detailed fingerprint information
- will be completely absent. </para>
-
- <para>This particular example, for example, would indicate that the likely cause of slowness
- is simply a very large (on the order of 100MB) multiput, as we can tell by the "vlen," or
- value length, fields of each put in the multiPut. </para>
- </section>
- </section>
-
-
-
- </section>
-
- <section
- xml:id="cluster_replication">
- <title>Cluster Replication</title>
- <para>See <link
- xlink:href="http://hbase.apache.org/replication.html">Cluster Replication</link>. </para>
- </section>
- <section
- xml:id="ops.backup">
- <title>HBase Backup</title>
- <para>There are two broad strategies for performing HBase backups: backing up with a full
- cluster shutdown, and backing up on a live cluster. Each approach has pros and cons. </para>
- <para>For additional information, see <link
- xlink:href="http://blog.sematext.com/2011/03/11/hbase-backup-options/">HBase Backup
- Options</link> over on the Sematext Blog. </para>
- <section
- xml:id="ops.backup.fullshutdown">
- <title>Full Shutdown Backup</title>
- <para>Some environments can tolerate a periodic full shutdown of their HBase cluster, for
- example if it is being used a back-end analytic capacity and not serving front-end
- web-pages. The benefits are that the NameNode/Master are RegionServers are down, so there is
- no chance of missing any in-flight changes to either StoreFiles or metadata. The obvious con
- is that the cluster is down. The steps include: </para>
- <section
- xml:id="ops.backup.fullshutdown.stop">
- <title>Stop HBase</title>
- <para> </para>
- </section>
- <section
- xml:id="ops.backup.fullshutdown.distcp">
- <title>Distcp</title>
- <para>Distcp could be used to either copy the contents of the HBase directory in HDFS to
- either the same cluster in another directory, or to a different cluster. </para>
- <para>Note: Distcp works in this situation because the cluster is down and there are no
- in-flight edits to files. Distcp-ing of files in the HBase directory is not generally
- recommended on a live cluster. </para>
- </section>
- <section
- xml:id="ops.backup.fullshutdown.restore">
- <title>Restore (if needed)</title>
- <para>The backup of the hbase directory from HDFS is copied onto the 'real' hbase directory
- via distcp. The act of copying these files creates new HDFS metadata, which is why a
- restore of the NameNode edits from the time of the HBase backup isn't required for this
- kind of restore, because it's a restore (via distcp) of a specific HDFS directory (i.e.,
- the HBase part) not the entire HDFS file-system. </para>
- </section>
- </section>
- <section
- xml:id="ops.backup.live.replication">
- <title>Live Cluster Backup - Replication</title>
- <para>This approach assumes that there is a second cluster. See the HBase page on <link
- xlink:href="http://hbase.apache.org/replication.html">replication</link> for more
- information. </para>
- </section>
- <section
- xml:id="ops.backup.live.copytable">
- <title>Live Cluster Backup - CopyTable</title>
- <para>The <xref
- linkend="copytable" /> utility could either be used to copy data from one table to another
- on the same cluster, or to copy data to another table on another cluster. </para>
- <para>Since the cluster is up, there is a risk that edits could be missed in the copy process.
- </para>
- </section>
- <section
- xml:id="ops.backup.live.export">
- <title>Live Cluster Backup - Export</title>
- <para>The <xref
- linkend="export" /> approach dumps the content of a table to HDFS on the same cluster. To
- restore the data, the <xref
- linkend="import" /> utility would be used. </para>
- <para>Since the cluster is up, there is a risk that edits could be missed in the export
- process. </para>
- </section>
- </section>
- <!-- backup -->
-
- <section
- xml:id="ops.snapshots">
- <title>HBase Snapshots</title>
- <para>HBase Snapshots allow you to take a snapshot of a table without too much impact on Region
- Servers. Snapshot, Clone and restore operations don't involve data copying. Also, Exporting
- the snapshot to another cluster doesn't have impact on the Region Servers. </para>
- <para>Prior to version 0.94.6, the only way to backup or to clone a table is to use
- CopyTable/ExportTable, or to copy all the hfiles in HDFS after disabling the table. The
- disadvantages of these methods are that you can degrade region server performance (Copy/Export
- Table) or you need to disable the table, that means no reads or writes; and this is usually
- unacceptable. </para>
- <section
- xml:id="ops.snapshots.configuration">
- <title>Configuration</title>
- <para>To turn on the snapshot support just set the <varname>hbase.snapshot.enabled</varname>
- property to true. (Snapshots are enabled by default in 0.95+ and off by default in
- 0.94.6+)</para>
- <programlisting>
- <property>
- <name>hbase.snapshot.enabled</name>
- <value>true</value>
- </property>
- </programlisting>
- </section>
- <section
- xml:id="ops.snapshots.takeasnapshot">
- <title>Take a Snapshot</title>
- <para>You can take a snapshot of a table regardless of whether it is enabled or disabled. The
- snapshot operation doesn't involve any data copying.</para>
- <screen>
-$ ./bin/hbase shell
-hbase> snapshot 'myTable', 'myTableSnapshot-122112'
- </screen>
- <formalpara>
- <title>Take a Snapshot Without Flushing</title>
- <para>The default behavior is to perform a flush of data in memory before the snapshot is
- taken. This means that data in memory is included in the snapshot. In most cases, this is
- the desired behavior. However, if your set-up can tolerate data in memory being excluded
- from the snapshot, you can use the <option>SKIP_FLUSH</option> option of the
- <command>snapshot</command> command to disable and flushing while taking the
- snapshot.</para>
- </formalpara>
- <screen>hbase> snapshot 'mytable', 'snapshot123', {SKIP_FLUSH => true}</screen>
- <warning>
- <para>There is no way to determine or predict whether a very concurrent insert or update
- will be included in a given snapshot, whether flushing is enabled or disabled. A snapshot
- is only a representation of a table during a window of time. The amount of time the
- snapshot operation will take to reach each Region Server may vary from a few seconds to a
- minute, depending on the resource load and speed of the hardware or network, among other
- factors. There is also no way to know whether a given insert or update is in memory or has
- been flushed.</para>
- </warning>
- </section>
- <section
- xml:id="ops.snapshots.list">
- <title>Listing Snapshots</title>
- <para>List all snapshots taken (by printing the names and relative information).</para>
- <screen>
-$ ./bin/hbase shell
-hbase> list_snapshots
- </screen>
- </section>
- <section
- xml:id="ops.snapshots.delete">
- <title>Deleting Snapshots</title>
- <para>You can remove a snapshot, and the files retained for that snapshot will be removed if
- no longer needed.</para>
- <screen>
-$ ./bin/hbase shell
-hbase> delete_snapshot 'myTableSnapshot-122112'
- </screen>
- </section>
- <section
- xml:id="ops.snapshots.clone">
- <title>Clone a table from snapshot</title>
- <para>From a snapshot you can create a new table (clone operation) with the same data that you
- had when the snapshot was taken. The clone operation, doesn't involve data copies, and a
- change to the cloned table doesn't impact the snapshot or the original table.</para>
- <screen>
-$ ./bin/hbase shell
-hbase> clone_snapshot 'myTableSnapshot-122112', 'myNewTestTable'
- </screen>
- </section>
- <section
- xml:id="ops.snapshots.restore">
- <title>Restore a snapshot</title>
- <para>The restore operation requires the table to be disabled, and the table will be restored
- to the state at the time when the snapshot was taken, changing both data and schema if
- required.</para>
- <screen>
-$ ./bin/hbase shell
-hbase> disable 'myTable'
-hbase> restore_snapshot 'myTableSnapshot-122112'
- </screen>
- <note>
- <para>Since Replication works at log level and snapshots at file-system level, after a
- restore, the replicas will be in a different state from the master. If you want to use
- restore, you need to stop replication and redo the bootstrap. </para>
- </note>
- <para>In case of partial data-loss due to misbehaving client, instead of a full restore that
- requires the table to be disabled, you can clone the table from the snapshot and use a
- Map-Reduce job to copy the data that you need, from the clone to the main one. </para>
- </section>
- <section
- xml:id="ops.snapshots.acls">
- <title>Snapshots operations and ACLs</title>
- <para>If you are using security with the AccessController Coprocessor (See <xref
- linkend="hbase.accesscontrol.configuration" />), only a global administrator can take,
- clone, or restore a snapshot, and these actions do not capture the ACL rights. This means
- that restoring a table preserves the ACL rights of the existing table, while cloning a table
- creates a new table that has no ACL rights until the administrator adds them.</para>
- </section>
- <section
- xml:id="ops.snapshots.export">
- <title>Export to another cluster</title>
- <para>The ExportSnapshot tool copies all the data related to a snapshot (hfiles, logs,
- snapshot metadata) to another cluster. The tool executes a Map-Reduce job, similar to
- distcp, to copy files between the two clusters, and since it works at file-system level the
- hbase cluster does not have to be online.</para>
- <para>To copy a snapshot called MySnapshot to an HBase cluster srv2 (hdfs:///srv2:8082/hbase)
- using 16 mappers:</para>
- <programlisting>$ bin/hbase class org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot MySnapshot -copy-to hdfs://srv2:8082/hbase -mappers 16</programlisting>
- <formalpara>
- <title>Limiting Bandwidth Consumption</title>
- <para>You can limit the bandwidth consumption when exporting a snapshot, by specifying the
- <code>-bandwidth</code> parameter, which expects an integer representing megabytes per
- second. The following example limits the above example to 200 MB/sec.</para>
- </formalpara>
- <programlisting>$ bin/hbase class org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot MySnapshot -copy-to hdfs://srv2:8082/hbase -mappers 16 -bandwidth 200</programlisting>
- </section>
- </section>
- <!-- snapshots -->
-
- <section
- xml:id="ops.capacity">
- <title>Capacity Planning and Region Sizing</title>
- <para>There are several considerations when planning the capacity for an HBase cluster and
- performing the initial configuration. Start with a solid understanding of how HBase handles
- data internally.</para>
- <section
- xml:id="ops.capacity.nodes">
- <title>Node count and hardware/VM configuration</title>
- <section
- xml:id="ops.capacity.nodes.datasize">
- <title>Physical data size</title>
- <para>Physical data size on disk is distinct from logical size of your data and is affected
- by the following: </para>
- <itemizedlist>
- <listitem>
- <para>Increased by HBase overhead</para>
- <itemizedlist>
- <listitem>
- <para>See <xref
- linkend="keyvalue" /> and <xref
- linkend="keysize" />. At least 24 bytes per key-value (cell), can be more. Small
- keys/values means more relative overhead.</para>
- </listitem>
- <listitem>
- <para>KeyValue instances are aggregated into blocks, which are indexed. Indexes also
- have to be stored. Blocksize is configurable on a per-ColumnFamily basis. See <xref
- linkend="regions.arch" />.</para>
- </listitem>
- </itemizedlist>
- </listitem>
- <listitem>
- <para>Decreased by <xref
- linkend="compression"
- xrefstyle="template:compression" /> and data block encoding, depending on data. See
- also <link
- xlink:href="http://search-hadoop.com/m/lL12B1PFVhp1">this thread</link>. You might
- want to test what compression and encoding (if any) make sense for your data.</para>
- </listitem>
- <listitem>
- <para>Increased by size of region server <xref
- linkend="wal"
- xrefstyle="template:WAL" /> (usually fixed and negligible - less than half of RS
- memory size, per RS).</para>
- </listitem>
- <listitem>
- <para>Increased by HDFS replication - usually x3.</para>
- </listitem>
- </itemizedlist>
- <para>Aside from the disk space necessary to store the data, one RS may not be able to serve
- arbitrarily large amounts of data due to some practical limits on region count and size
- (see <xref
- linkend="ops.capacity.regions"
- xrefstyle="template:below" />).</para>
- </section>
- <!-- ops.capacity.nodes.datasize -->
- <section
- xml:id="ops.capacity.nodes.throughput">
- <title>Read/Write throughput</title>
- <para>Number of nodes can also be driven by required thoughput for reads and/or writes. The
- throughput one can get per node depends a lot on data (esp. key/value sizes) and request
- patterns, as well as node and system configuration. Planning should be done for peak load
- if it is likely that the load would be the main driver of the increase of the node count.
- PerformanceEvaluation and <xref
- linkend="ycsb"
- xrefstyle="template:YCSB" /> tools can be used to test single node or a test
- cluster.</para>
- <para>For write, usually 5-15Mb/s per RS can be expected, since every region server has only
- one active WAL. There's no good estimate for reads, as it depends vastly on data,
- requests, and cache hit rate. <xref
- linkend="perf.casestudy" /> might be helpful.</para>
- </section>
- <!-- ops.capacity.nodes.throughput -->
- <section
- xml:id="ops.capacity.nodes.gc">
- <title>JVM GC limitations</title>
- <para>RS cannot currently utilize very large heap due to cost of GC. There's also no good
- way of running multiple RS-es per server (other than running several VMs per machine).
- Thus, ~20-24Gb or less memory dedicated to one RS is recommended. GC tuning is required
- for large heap sizes. See <xref
- linkend="gcpause" />, <xref
- linkend="trouble.log.gc" /> and elsewhere (TODO: where?)</para>
- </section>
- <!-- ops.capacity.nodes.gc -->
- </section>
- <!-- ops.capacity.nodes -->
- <section
- xml:id="ops.capacity.regions">
- <title>Determining region count and size</title>
- <para>Generally less regions makes for a smoother running cluster (you can always manually
- split the big regions later (if necessary) to spread the data, or request load, over the
- cluster); 20-200 regions per RS is a reasonable range. The number of regions cannot be
- configured directly (unless you go for fully <xref
- linkend="disable.splitting"
- xrefstyle="template:manual splitting" />); adjust the region size to achieve the target
- region size given table size.</para>
- <para>When configuring regions for multiple tables, note that most region settings can be set
- on a per-table basis via <link
- xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/HTableDescriptor.html">HTableDescriptor</link>,
- as well as shell commands. These settings will override the ones in
- <varname>hbase-site.xml</varname>. That is useful if your tables have different
- workloads/use cases.</para>
- <para>Also note that in the discussion of region sizes here, <emphasis
- role="bold">HDFS replication factor is not (and should not be) taken into account, whereas
- other factors <xref
- linkend="ops.capacity.nodes.datasize"
- xrefstyle="template:above" /> should be.</emphasis> So, if your data is compressed and
- replicated 3 ways by HDFS, "9 Gb region" means 9 Gb of compressed data. HDFS replication
- factor only affects your disk usage and is invisible to most HBase code.</para>
- <section>
- <title>Viewing the Current Number of Regions</title>
- <para>You can view the current number of regions for a given table using the HMaster UI. In
- the <guilabel>Tables</guilabel> section, the number of online regions for each table is
- listed in the <guilabel>Online Regions</guilabel> column. This total only includes the
- in-memory state and does not include disabled or offline regions. If you do not want to
- use the HMaster UI, you can determine the number of regions by counting the number of
- subdirectories of the /hbase/<table>/ subdirectories in HDFS, or by running the
- <command>bin/hbase hbck</command> command. Each of these methods may return a slightly
- different number, depending on the status of each region.</para>
- </section>
- <section
- xml:id="ops.capacity.regions.count">
- <title>Number of regions per RS - upper bound</title>
- <para>In production scenarios, where you have a lot of data, you are normally concerned with
- the maximum number of regions you can have per server. <xref
- linkend="too_many_regions" /> has technical discussion on the subject; in short, maximum
- number of regions is mostly determined by memstore memory usage. Each region has its own
- memstores; these grow up to a configurable size; usually in 128-256Mb range, see <xref
- linkend="hbase.hregion.memstore.flush.size" />. There's one memstore per column family
- (so there's only one per region if there's one CF in the table). RS dedicates some
- fraction of total memory (see <xref
- linkend="hbase.regionserver.global.memstore.size" />) to region memstores. If this
- memory is exceeded (too much memstore usage), undesirable consequences such as
- unresponsive server, or later compaction storms, can result. Thus, a good starting point
- for the number of regions per RS (assuming one table) is:</para>
-
- <programlisting>(RS memory)*(total memstore fraction)/((memstore size)*(# column families))</programlisting>
- <para> E.g. if RS has 16Gb RAM, with default settings, it is 16384*0.4/128 ~ 51 regions per
- RS is a starting point. The formula can be extended to multiple tables; if they all have
- the same configuration, just use total number of families.</para>
- <para>This number can be adjusted; the formula above assumes all your regions are filled at
- approximately the same rate. If only a fraction of your regions are going to be actively
- written to, you can divide the result by that fraction to get a larger region count. Then,
- even if all regions are written to, all region memstores are not filled evenly, and
- eventually jitter appears even if they are (due to limited number of concurrent flushes).
- Thus, one can have as many as 2-3 times more regions than the starting point; however,
- increased numbers carry increased risk.</para>
- <para>For write-heavy workload, memstore fraction can be increased in configuration at the
- expense of block cache; this will also allow one to have more regions.</para>
- </section>
- <!-- ops.capacity.regions.count -->
- <section
- xml:id="ops.capacity.regions.mincount">
- <title>Number of regions per RS - lower bound</title>
- <para>HBase scales by having regions across many servers. Thus if you have 2 regions for
- 16GB data, on a 20 node machine your data will be concentrated on just a few machines -
- nearly the entire cluster will be idle. This really can't be stressed enough, since a
- common problem is loading 200MB data into HBase and then wondering why your awesome 10
- node cluster isn't doing anything.</para>
- <para>On the other hand, if you have a very large amount of data, you may also want to go
- for a larger number of regions to avoid having regions that are too large.</para>
- </section>
- <!-- ops.capacity.regions.mincount -->
- <section
- xml:id="ops.capacity.regions.size">
- <title>Maximum region size</title>
- <para>For large tables in production scenarios, maximum region size is mostly limited by
- compactions - very large compactions, esp. major, can degrade cluster performance.
- Currently, the recommended maximum region size is 10-20Gb, and 5-10Gb is optimal. For
- older 0.90.x codebase, the upper-bound of regionsize is about 4Gb, with a default of
- 256Mb.</para>
- <para>The size at which the region is split into two is generally configured via <xref
- linkend="hbase.hregion.max.filesize" />; for details, see <xref
- linkend="arch.region.splits" />.</para>
- <para>If you cannot estimate the size of your tables well, when starting off, it's probably
- best to stick to the default region size, perhaps going smaller for hot tables (or
- manually split hot regions to spread the load over the cluster), or go with larger region
- sizes if your cell sizes tend to be largish (100k and up).</para>
- <para>In HBase 0.98, experimental stripe compactions feature was added that would allow for
- larger regions, especially for log data. See <xref
- linkend="ops.stripe" />.</para>
- </section>
- <!-- ops.capacity.regions.size -->
- <section
- xml:id="ops.capacity.regions.total">
- <title>Total data size per region server</title>
- <para>According to above numbers for region size and number of regions per region server, in
- an optimistic estimate 10 GB x 100 regions per RS will give up to 1TB served per region
- server, which is in line with some of the reported multi-PB use cases. However, it is
- important to think about the data vs cache size ratio at the RS level. With 1TB of data
- per server and 10 GB block cache, only 1% of the data will be cached, which may barely
- cover all block indices.</para>
- </section>
- <!-- ops.capacity.regions.total -->
- </section>
- <!-- ops.capacity.regions -->
- <section
- xml:id="ops.capacity.config">
- <title>Initial configuration and tuning</title>
- <para>First, see <xref
- linkend="important_configurations" />. Note that some configurations, more than others,
- depend on specific scenarios. Pay special attention to:</para>
- <itemizedlist>
- <listitem>
- <para><xref
- linkend="hbase.regionserver.handler.count" /> - request handler thread count, vital
- for high-throughput workloads.</para>
- </listitem>
- <listitem>
- <para><xref
- linkend="config.wals" /> - the blocking number of WAL files depends on your memstore
- configuration and should be set accordingly to prevent potential blocking when doing
- high volume of writes.</para>
- </listitem>
- </itemizedlist>
- <para>Then, there are some considerations when setting up your cluster and tables.</para>
- <section
- xml:id="ops.capacity.config.compactions">
- <title>Compactions</title>
- <para>Depending on read/write volume and latency requirements, optimal compaction settings
- may be different. See <xref
- linkend="compaction" /> for some details.</para>
- <para>When provisioning for large data sizes, however, it's good to keep in mind that
- compactions can affect write throughput. Thus, for write-intensive workloads, you may opt
-
<TRUNCATED>
[2/3] hbase git commit: HBASE-12455 Add 'description' to bean and
attribute output when you do /jmx?description=true
Posted by st...@apache.org.
HBASE-12455 Add 'description' to bean and attribute output when you do /jmx?description=true
Conflicts:
hbase-server/src/main/java/org/apache/hadoop/hbase/http/jmx/JMXJsonServlet.java
src/main/docbkx/ops_mgt.xml
Project: http://git-wip-us.apache.org/repos/asf/hbase/repo
Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/d7f51e1c
Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/d7f51e1c
Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/d7f51e1c
Branch: refs/heads/branch-1
Commit: d7f51e1cfa51d4af245be7b66f9a5c2d53d64a90
Parents: 8902fb0
Author: stack <st...@apache.org>
Authored: Tue Nov 11 12:59:30 2014 -0800
Committer: stack <st...@apache.org>
Committed: Tue Nov 11 15:06:25 2014 -0800
----------------------------------------------------------------------
src/main/docbkx/ops_mgt.xml | 1635 --------------------------------------
1 file changed, 1635 deletions(-)
----------------------------------------------------------------------
[3/3] hbase git commit: HBASE-12447 Add support for setTimeRange for
RowCounter and CellCounter
Posted by st...@apache.org.
HBASE-12447 Add support for setTimeRange for RowCounter and CellCounter
Signed-off-by: stack <st...@apache.org>
Project: http://git-wip-us.apache.org/repos/asf/hbase/repo
Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/bebf0181
Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/bebf0181
Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/bebf0181
Branch: refs/heads/branch-1
Commit: bebf0181c8609870034dbcbe21f1bd45b3db3ea7
Parents: d7f51e1
Author: Esteban Gutierrez <es...@cloudera.com>
Authored: Tue Nov 11 13:38:59 2014 -0800
Committer: stack <st...@apache.org>
Committed: Tue Nov 11 15:06:56 2014 -0800
----------------------------------------------------------------------
.../hadoop/hbase/mapreduce/CellCounter.java | 35 ++++-
.../hadoop/hbase/mapreduce/RowCounter.java | 20 +++
.../hadoop/hbase/mapreduce/TestCellCounter.java | 121 +++++++++++++++-
.../hadoop/hbase/mapreduce/TestRowCounter.java | 144 +++++++++++++------
4 files changed, 275 insertions(+), 45 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hbase/blob/bebf0181/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/CellCounter.java
----------------------------------------------------------------------
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/CellCounter.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/CellCounter.java
index df97f63..b36f3d2 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/CellCounter.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/CellCounter.java
@@ -22,6 +22,7 @@ import java.io.IOException;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.classification.InterfaceAudience;
import org.apache.hadoop.hbase.classification.InterfaceStability;
import org.apache.hadoop.conf.Configuration;
@@ -37,6 +38,7 @@ import org.apache.hadoop.hbase.filter.PrefixFilter;
import org.apache.hadoop.hbase.filter.RegexStringComparator;
import org.apache.hadoop.hbase.filter.RowFilter;
import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.io.TimeRange;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
@@ -217,6 +219,12 @@ public class CellCounter {
LOG.info("Setting Row Filter for counter.");
s.setFilter(rowFilter);
}
+ // Set TimeRange if defined
+ long timeRange[] = getTimeRange(args);
+ if (timeRange != null) {
+ LOG.info("Setting TimeRange for counter.");
+ s.setTimeRange(timeRange[0], timeRange[1]);
+ }
return s;
}
@@ -234,6 +242,28 @@ public class CellCounter {
return rowFilter;
}
+ private static long[] getTimeRange(String[] args) throws IOException {
+ final String startTimeArgKey = "--starttime=";
+ final String endTimeArgKey = "--endtime=";
+ long startTime = 0L;
+ long endTime = 0L;
+
+ for (int i = 1; i < args.length; i++) {
+ System.out.println("i:" + i + "arg[i]" + args[i]);
+ if (args[i].startsWith(startTimeArgKey)) {
+ startTime = Long.parseLong(args[i].substring(startTimeArgKey.length()));
+ }
+ if (args[i].startsWith(endTimeArgKey)) {
+ endTime = Long.parseLong(args[i].substring(endTimeArgKey.length()));
+ }
+ }
+
+ if (startTime == 0 && endTime == 0)
+ return null;
+
+ endTime = endTime == 0 ? HConstants.LATEST_TIMESTAMP : endTime;
+ return new long [] {startTime, endTime};
+ }
/**
* Main entry point.
*
@@ -245,8 +275,9 @@ public class CellCounter {
String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
if (otherArgs.length < 2) {
System.err.println("ERROR: Wrong number of parameters: " + args.length);
- System.err.println("Usage: CellCounter <tablename> <outputDir> <reportSeparator> " +
- "[^[regex pattern] or [Prefix] for row filter]] ");
+ System.err.println("Usage: CellCounter ");
+ System.err.println(" <outputDir> <reportSeparator> [^[regex pattern] or " +
+ "[Prefix] for row filter]] <tablename> --starttime=[starttime] --endtime=[endtime]");
System.err.println(" Note: -D properties will be applied to the conf used. ");
System.err.println(" Additionally, the following SCAN properties can be specified");
System.err.println(" to get fine grained control on what is counted..");
http://git-wip-us.apache.org/repos/asf/hbase/blob/bebf0181/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/RowCounter.java
----------------------------------------------------------------------
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/RowCounter.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/RowCounter.java
index c2b627c..04450f8 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/RowCounter.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/RowCounter.java
@@ -23,6 +23,7 @@ import java.util.Set;
import java.util.TreeSet;
import org.apache.commons.lang.StringUtils;
+import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.classification.InterfaceAudience;
import org.apache.hadoop.hbase.classification.InterfaceStability;
import org.apache.hadoop.conf.Configuration;
@@ -89,9 +90,14 @@ public class RowCounter {
String tableName = args[0];
String startKey = null;
String endKey = null;
+ long startTime = 0;
+ long endTime = 0;
+
StringBuilder sb = new StringBuilder();
final String rangeSwitch = "--range=";
+ final String startTimeArgKey = "--starttime=";
+ final String endTimeArgKey = "--endtime=";
// First argument is table name, starting from second
for (int i = 1; i < args.length; i++) {
@@ -105,6 +111,18 @@ public class RowCounter {
startKey = startEnd[0];
endKey = startEnd[1];
}
+ if (startTime < endTime) {
+ printUsage("--endtime=" + endTime + " needs to be greater than --starttime=" + startTime);
+ return null;
+ }
+ if (args[i].startsWith(startTimeArgKey)) {
+ startTime = Long.parseLong(args[i].substring(startTimeArgKey.length()));
+ continue;
+ }
+ if (args[i].startsWith(endTimeArgKey)) {
+ endTime = Long.parseLong(args[i].substring(endTimeArgKey.length()));
+ continue;
+ }
else {
// if no switch, assume column names
sb.append(args[i]);
@@ -144,6 +162,7 @@ public class RowCounter {
} else {
scan.setFilter(new FirstKeyValueMatchingQualifiersFilter(qualifiers));
}
+ scan.setTimeRange(startTime, endTime == 0 ? HConstants.LATEST_TIMESTAMP : endTime);
job.setOutputFormatClass(NullOutputFormat.class);
TableMapReduceUtil.initTableMapperJob(tableName, scan,
RowCounterMapper.class, ImmutableBytesWritable.class, Result.class, job);
@@ -164,6 +183,7 @@ public class RowCounter {
*/
private static void printUsage() {
System.err.println("Usage: RowCounter [options] <tablename> " +
+ "[--starttime=[start] --endtime=[end] " +
"[--range=[startKey],[endKey]] [<column1> <column2>...]");
System.err.println("For performance consider the following options:\n"
+ "-Dhbase.client.scanner.caching=100\n"
http://git-wip-us.apache.org/repos/asf/hbase/blob/bebf0181/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestCellCounter.java
----------------------------------------------------------------------
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestCellCounter.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestCellCounter.java
index 64c9d98..16fd013 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestCellCounter.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestCellCounter.java
@@ -74,7 +74,7 @@ public class TestCellCounter {
/**
* Test CellCounter all data should print to output
- *
+ *
*/
@Test (timeout=300000)
public void testCellCounter() throws Exception {
@@ -94,7 +94,7 @@ public class TestCellCounter {
t.put(p);
String[] args = { sourceTable, FQ_OUTPUT_DIR.toString(), ";", "^row1" };
runCount(args);
- FileInputStream inputStream = new FileInputStream(OUTPUT_DIR + File.separator +
+ FileInputStream inputStream = new FileInputStream(OUTPUT_DIR + File.separator +
"part-r-00000");
String data = IOUtils.toString(inputStream);
inputStream.close();
@@ -107,10 +107,127 @@ public class TestCellCounter {
assertTrue(data.contains("row1;b;q_Versions" + "\t" + "1"));
}finally{
t.close();
+ FileUtil.fullyDelete(new File(OUTPUT_DIR));
}
}
+ /**
+ * Test CellCounter with time range all data should print to output
+ */
+ @Test (timeout=300000)
+ public void testCellCounterStartTimeRange() throws Exception {
+ String sourceTable = "testCellCounterStartTimeRange";
+ byte[][] families = { FAMILY_A, FAMILY_B };
+ Table t = UTIL.createTable(Bytes.toBytes(sourceTable), families);
+ try{
+ Put p = new Put(ROW1);
+ p.add(FAMILY_A, QUALIFIER, now, Bytes.toBytes("Data11"));
+ p.add(FAMILY_B, QUALIFIER, now + 1, Bytes.toBytes("Data12"));
+ p.add(FAMILY_A, QUALIFIER, now + 2, Bytes.toBytes("Data13"));
+ t.put(p);
+ p = new Put(ROW2);
+ p.add(FAMILY_B, QUALIFIER, now, Bytes.toBytes("Dat21"));
+ p.add(FAMILY_A, QUALIFIER, now + 1, Bytes.toBytes("Data22"));
+ p.add(FAMILY_B, QUALIFIER, now + 2, Bytes.toBytes("Data23"));
+ t.put(p);
+ String[] args = {
+ sourceTable, FQ_OUTPUT_DIR.toString(), ";", "^row1", "--starttime=" + now,
+ "--endtime=" + now + 2 };
+ runCount(args);
+ FileInputStream inputStream = new FileInputStream(OUTPUT_DIR + File.separator +
+ "part-r-00000");
+ String data = IOUtils.toString(inputStream);
+ inputStream.close();
+ assertTrue(data.contains("Total Families Across all Rows" + "\t" + "2"));
+ assertTrue(data.contains("Total Qualifiers across all Rows" + "\t" + "2"));
+ assertTrue(data.contains("Total ROWS" + "\t" + "1"));
+ assertTrue(data.contains("b;q" + "\t" + "1"));
+ assertTrue(data.contains("a;q" + "\t" + "1"));
+ assertTrue(data.contains("row1;a;q_Versions" + "\t" + "1"));
+ assertTrue(data.contains("row1;b;q_Versions" + "\t" + "1"));
+ }finally{
+ t.close();
+ FileUtil.fullyDelete(new File(OUTPUT_DIR));
+ }
+ }
+
+ /**
+ * Test CellCounter with time range all data should print to output
+ */
+ @Test (timeout=300000)
+ public void testCellCounteEndTimeRange() throws Exception {
+ String sourceTable = "testCellCounterEndTimeRange";
+ byte[][] families = { FAMILY_A, FAMILY_B };
+ Table t = UTIL.createTable(Bytes.toBytes(sourceTable), families);
+ try{
+ Put p = new Put(ROW1);
+ p.add(FAMILY_A, QUALIFIER, now, Bytes.toBytes("Data11"));
+ p.add(FAMILY_B, QUALIFIER, now + 1, Bytes.toBytes("Data12"));
+ p.add(FAMILY_A, QUALIFIER, now + 2, Bytes.toBytes("Data13"));
+ t.put(p);
+ p = new Put(ROW2);
+ p.add(FAMILY_B, QUALIFIER, now, Bytes.toBytes("Dat21"));
+ p.add(FAMILY_A, QUALIFIER, now + 1, Bytes.toBytes("Data22"));
+ p.add(FAMILY_B, QUALIFIER, now + 2, Bytes.toBytes("Data23"));
+ t.put(p);
+ String[] args = {
+ sourceTable, FQ_OUTPUT_DIR.toString(), ";", "^row1", "--endtime=" + now + 1 };
+ runCount(args);
+ FileInputStream inputStream = new FileInputStream(OUTPUT_DIR + File.separator +
+ "part-r-00000");
+ String data = IOUtils.toString(inputStream);
+ inputStream.close();
+ assertTrue(data.contains("Total Families Across all Rows" + "\t" + "2"));
+ assertTrue(data.contains("Total Qualifiers across all Rows" + "\t" + "2"));
+ assertTrue(data.contains("Total ROWS" + "\t" + "1"));
+ assertTrue(data.contains("b;q" + "\t" + "1"));
+ assertTrue(data.contains("a;q" + "\t" + "1"));
+ assertTrue(data.contains("row1;a;q_Versions" + "\t" + "1"));
+ assertTrue(data.contains("row1;b;q_Versions" + "\t" + "1"));
+ }finally{
+ t.close();
+ FileUtil.fullyDelete(new File(OUTPUT_DIR));
+ }
+ }
+
+ /**
+ * Test CellCounter with time range all data should print to output
+ */
+ @Test (timeout=300000)
+ public void testCellCounteOutOfTimeRange() throws Exception {
+ String sourceTable = "testCellCounterOutTimeRange";
+ byte[][] families = { FAMILY_A, FAMILY_B };
+ Table t = UTIL.createTable(Bytes.toBytes(sourceTable), families);
+ try{
+ Put p = new Put(ROW1);
+ p.add(FAMILY_A, QUALIFIER, now, Bytes.toBytes("Data11"));
+ p.add(FAMILY_B, QUALIFIER, now + 1, Bytes.toBytes("Data12"));
+ p.add(FAMILY_A, QUALIFIER, now + 2, Bytes.toBytes("Data13"));
+ t.put(p);
+ p = new Put(ROW2);
+ p.add(FAMILY_B, QUALIFIER, now, Bytes.toBytes("Dat21"));
+ p.add(FAMILY_A, QUALIFIER, now + 1, Bytes.toBytes("Data22"));
+ p.add(FAMILY_B, QUALIFIER, now + 2, Bytes.toBytes("Data23"));
+ t.put(p);
+ String[] args = {
+ sourceTable, FQ_OUTPUT_DIR.toString(), ";", "--starttime=" + now + 1,
+ "--endtime=" + now + 2 };
+
+ runCount(args);
+ FileInputStream inputStream = new FileInputStream(OUTPUT_DIR + File.separator +
+ "part-r-00000");
+ String data = IOUtils.toString(inputStream);
+ inputStream.close();
+ // nothing should hace been emitted to the reducer
+ assertTrue(data.isEmpty());
+ }finally{
+ t.close();
+ FileUtil.fullyDelete(new File(OUTPUT_DIR));
+ }
+ }
+
+
private boolean runCount(String[] args) throws IOException, InterruptedException,
ClassNotFoundException {
// need to make a copy of the configuration because to make sure
http://git-wip-us.apache.org/repos/asf/hbase/blob/bebf0181/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestRowCounter.java
----------------------------------------------------------------------
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestRowCounter.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestRowCounter.java
index 4027d6d..5e8c216 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestRowCounter.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestRowCounter.java
@@ -25,7 +25,9 @@ import static org.junit.Assert.fail;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.PrintStream;
+import java.sql.Time;
import java.util.ArrayList;
+import java.util.Arrays;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
@@ -33,7 +35,12 @@ import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseTestingUtility;
import org.apache.hadoop.hbase.MediumTests;
import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Delete;
+import org.apache.hadoop.hbase.client.HTable;
import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.ResultScanner;
+import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.client.Table;
import org.apache.hadoop.hbase.mapreduce.RowCounter.RowCounterMapper;
import org.apache.hadoop.hbase.util.Bytes;
@@ -52,20 +59,29 @@ import org.junit.experimental.categories.Category;
@Category(MediumTests.class)
public class TestRowCounter {
final Log LOG = LogFactory.getLog(getClass());
+
private final static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility();
+
private final static String TABLE_NAME = "testRowCounter";
+
private final static String COL_FAM = "col_fam";
+
private final static String COL1 = "c1";
+
private final static String COL2 = "c2";
+
private final static String COMPOSITE_COLUMN = "C:A:A";
+
private final static int TOTAL_ROWS = 10;
+
private final static int ROWS_WITH_ONE_COL = 2;
/**
* @throws java.lang.Exception
*/
@BeforeClass
- public static void setUpBeforeClass() throws Exception {
+ public static void setUpBeforeClass()
+ throws Exception {
TEST_UTIL.startMiniCluster();
TEST_UTIL.startMiniMapReduceCluster();
Table table = TEST_UTIL.createTable(TableName.valueOf(TABLE_NAME), Bytes.toBytes(COL_FAM));
@@ -77,35 +93,34 @@ public class TestRowCounter {
* @throws java.lang.Exception
*/
@AfterClass
- public static void tearDownAfterClass() throws Exception {
+ public static void tearDownAfterClass()
+ throws Exception {
TEST_UTIL.shutdownMiniCluster();
TEST_UTIL.shutdownMiniMapReduceCluster();
}
/**
* Test a case when no column was specified in command line arguments.
- *
+ *
* @throws Exception
*/
@Test
- public void testRowCounterNoColumn() throws Exception {
- String[] args = new String[] {
- TABLE_NAME
- };
+ public void testRowCounterNoColumn()
+ throws Exception {
+ String[] args = new String[] {TABLE_NAME};
runRowCount(args, 10);
}
/**
* Test a case when the column specified in command line arguments is
* exclusive for few rows.
- *
+ *
* @throws Exception
*/
@Test
- public void testRowCounterExclusiveColumn() throws Exception {
- String[] args = new String[] {
- TABLE_NAME, COL_FAM + ":" + COL1
- };
+ public void testRowCounterExclusiveColumn()
+ throws Exception {
+ String[] args = new String[] {TABLE_NAME, COL_FAM + ":" + COL1};
runRowCount(args, 8);
}
@@ -116,55 +131,100 @@ public class TestRowCounter {
* @throws Exception
*/
@Test
- public void testRowCounterColumnWithColonInQualifier() throws Exception {
- String[] args = new String[] {
- TABLE_NAME, COL_FAM + ":" + COMPOSITE_COLUMN
- };
+ public void testRowCounterColumnWithColonInQualifier()
+ throws Exception {
+ String[] args = new String[] {TABLE_NAME, COL_FAM + ":" + COMPOSITE_COLUMN};
runRowCount(args, 8);
}
/**
* Test a case when the column specified in command line arguments is not part
* of first KV for a row.
- *
+ *
* @throws Exception
*/
@Test
- public void testRowCounterHiddenColumn() throws Exception {
- String[] args = new String[] {
- TABLE_NAME, COL_FAM + ":" + COL2
- };
+ public void testRowCounterHiddenColumn()
+ throws Exception {
+ String[] args = new String[] {TABLE_NAME, COL_FAM + ":" + COL2};
runRowCount(args, 10);
}
/**
+ * Test a case when the timerange is specified with --starttime and --endtime options
+ *
+ * @throws Exception
+ */
+ @Test
+ public void testRowCounterTimeRange()
+ throws Exception {
+ final byte[] family = Bytes.toBytes(COL_FAM);
+ final byte[] col1 = Bytes.toBytes(COL1);
+ Put put1 = new Put(Bytes.toBytes("row_timerange_" + 1));
+ Put put2 = new Put(Bytes.toBytes("row_timerange_" + 2));
+ Put put3 = new Put(Bytes.toBytes("row_timerange_" + 3));
+
+ long ts;
+
+ // clean up content of TABLE_NAME
+ HTable table = TEST_UTIL.truncateTable(TableName.valueOf(TABLE_NAME));
+ ts = System.currentTimeMillis();
+ put1.add(family, col1, ts, Bytes.toBytes("val1"));
+ table.put(put1);
+ Thread.sleep(100);
+
+ ts = System.currentTimeMillis();
+ put2.add(family, col1, ts, Bytes.toBytes("val2"));
+ put3.add(family, col1, ts, Bytes.toBytes("val3"));
+ table.put(put2);
+ table.put(put3);
+ table.close();
+
+ String[] args = new String[] {TABLE_NAME, COL_FAM + ":" + COL1, "--starttime=" + 0,
+ "--endtime=" + ts};
+ runRowCount(args, 1);
+
+ args = new String[] {TABLE_NAME, COL_FAM + ":" + COL1, "--starttime=" + 0,
+ "--endtime=" + (ts - 10)};
+ runRowCount(args, 1);
+
+ args = new String[] {TABLE_NAME, COL_FAM + ":" + COL1, "--starttime=" + ts,
+ "--endtime=" + (ts + 1000)};
+ runRowCount(args, 2);
+
+ args = new String[] {TABLE_NAME, COL_FAM + ":" + COL1, "--starttime=" + (ts - 30 * 1000),
+ "--endtime=" + (ts + 30 * 1000),};
+ runRowCount(args, 3);
+ }
+
+ /**
* Run the RowCounter map reduce job and verify the row count.
- *
+ *
* @param args the command line arguments to be used for rowcounter job.
* @param expectedCount the expected row count (result of map reduce job).
* @throws Exception
*/
- private void runRowCount(String[] args, int expectedCount) throws Exception {
- GenericOptionsParser opts = new GenericOptionsParser(
- TEST_UTIL.getConfiguration(), args);
+ private void runRowCount(String[] args, int expectedCount)
+ throws Exception {
+ GenericOptionsParser opts = new GenericOptionsParser(TEST_UTIL.getConfiguration(), args);
Configuration conf = opts.getConfiguration();
args = opts.getRemainingArgs();
Job job = RowCounter.createSubmittableJob(conf, args);
job.waitForCompletion(true);
assertTrue(job.isSuccessful());
- Counter counter = job.getCounters().findCounter(
- RowCounterMapper.Counters.ROWS);
+ Counter counter = job.getCounters().findCounter(RowCounterMapper.Counters.ROWS);
assertEquals(expectedCount, counter.getValue());
}
/**
* Writes TOTAL_ROWS number of distinct rows in to the table. Few rows have
* two columns, Few have one.
- *
+ *
* @param table
* @throws IOException
*/
- private static void writeRows(Table table) throws IOException {
+ private static void writeRows(Table table)
+ throws IOException {
final byte[] family = Bytes.toBytes(COL_FAM);
final byte[] value = Bytes.toBytes("abcd");
final byte[] col1 = Bytes.toBytes(COL1);
@@ -196,10 +256,11 @@ public class TestRowCounter {
* test main method. Import should print help and call System.exit
*/
@Test
- public void testImportMain() throws Exception {
+ public void testImportMain()
+ throws Exception {
PrintStream oldPrintStream = System.err;
SecurityManager SECURITY_MANAGER = System.getSecurityManager();
- LauncherSecurityManager newSecurityManager= new LauncherSecurityManager();
+ LauncherSecurityManager newSecurityManager = new LauncherSecurityManager();
System.setSecurityManager(newSecurityManager);
ByteArrayOutputStream data = new ByteArrayOutputStream();
String[] args = {};
@@ -213,9 +274,10 @@ public class TestRowCounter {
} catch (SecurityException e) {
assertEquals(-1, newSecurityManager.getExitCode());
assertTrue(data.toString().contains("Wrong number of parameters:"));
- assertTrue(data.toString().contains(
- "Usage: RowCounter [options] <tablename> [--range=[startKey],[endKey]] " +
- "[<column1> <column2>...]"));
+ assertTrue(data.toString().contains("Usage: RowCounter [options] <tablename> " +
+ "[--starttime=[start] --endtime=[end] " +
+ "[--range=[startKey],[endKey]] " +
+ "[<column1> <column2>...]"));
assertTrue(data.toString().contains("-Dhbase.client.scanner.caching=100"));
assertTrue(data.toString().contains("-Dmapreduce.map.speculative=false"));
}
@@ -228,12 +290,13 @@ public class TestRowCounter {
fail("should be SecurityException");
} catch (SecurityException e) {
assertEquals(-1, newSecurityManager.getExitCode());
- assertTrue(data.toString().contains(
- "Please specify range in such format as \"--range=a,b\" or, with only one boundary," +
- " \"--range=,b\" or \"--range=a,\""));
- assertTrue(data.toString().contains(
- "Usage: RowCounter [options] <tablename> [--range=[startKey],[endKey]] " +
- "[<column1> <column2>...]"));
+ assertTrue(data.toString().contains("Please specify range in such format as \"--range=a,b\" or, with only one boundary," +
+
+ " \"--range=,b\" or \"--range=a,\""));
+ assertTrue(data.toString().contains("Usage: RowCounter [options] <tablename> " +
+ "[--starttime=[start] --endtime=[end] " +
+ "[--range=[startKey],[endKey]] " +
+ "[<column1> <column2>...]"));
}
} finally {
@@ -242,5 +305,4 @@ public class TestRowCounter {
}
}
-
}