You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hbase.apache.org by mi...@apache.org on 2014/12/22 06:45:42 UTC
[4/8] hbase git commit: HBASE-12738 Chunk Ref Guide into
file-per-chapter
http://git-wip-us.apache.org/repos/asf/hbase/blob/a1fe1e09/src/main/docbkx/compression.xml
----------------------------------------------------------------------
diff --git a/src/main/docbkx/compression.xml b/src/main/docbkx/compression.xml
new file mode 100644
index 0000000..d1971b1
--- /dev/null
+++ b/src/main/docbkx/compression.xml
@@ -0,0 +1,535 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<appendix
+ xml:id="compression"
+ version="5.0"
+ xmlns="http://docbook.org/ns/docbook"
+ xmlns:xlink="http://www.w3.org/1999/xlink"
+ xmlns:xi="http://www.w3.org/2001/XInclude"
+ xmlns:svg="http://www.w3.org/2000/svg"
+ xmlns:m="http://www.w3.org/1998/Math/MathML"
+ xmlns:html="http://www.w3.org/1999/xhtml"
+ xmlns:db="http://docbook.org/ns/docbook">
+ <!--/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+-->
+
+ <title>Compression and Data Block Encoding In
+ HBase<indexterm><primary>Compression</primary><secondary>Data Block
+ Encoding</secondary><seealso>codecs</seealso></indexterm></title>
+ <note>
+ <para>Codecs mentioned in this section are for encoding and decoding data blocks or row keys.
+ For information about replication codecs, see <xref
+ linkend="cluster.replication.preserving.tags" />.</para>
+ </note>
+ <para>Some of the information in this section is pulled from a <link
+ xlink:href="http://search-hadoop.com/m/lL12B1PFVhp1/v=threaded">discussion</link> on the
+ HBase Development mailing list.</para>
+ <para>HBase supports several different compression algorithms which can be enabled on a
+ ColumnFamily. Data block encoding attempts to limit duplication of information in keys, taking
+ advantage of some of the fundamental designs and patterns of HBase, such as sorted row keys
+ and the schema of a given table. Compressors reduce the size of large, opaque byte arrays in
+ cells, and can significantly reduce the storage space needed to store uncompressed
+ data.</para>
+ <para>Compressors and data block encoding can be used together on the same ColumnFamily.</para>
+
+ <formalpara>
+ <title>Changes Take Effect Upon Compaction</title>
+ <para>If you change compression or encoding for a ColumnFamily, the changes take effect during
+ compaction.</para>
+ </formalpara>
+
+ <para>Some codecs take advantage of capabilities built into Java, such as GZip compression.
+ Others rely on native libraries. Native libraries may be available as part of Hadoop, such as
+ LZ4. In this case, HBase only needs access to the appropriate shared library. Other codecs,
+ such as Google Snappy, need to be installed first. Some codecs are licensed in ways that
+ conflict with HBase's license and cannot be shipped as part of HBase.</para>
+
+ <para>This section discusses common codecs that are used and tested with HBase. No matter what
+ codec you use, be sure to test that it is installed correctly and is available on all nodes in
+ your cluster. Extra operational steps may be necessary to be sure that codecs are available on
+ newly-deployed nodes. You can use the <xref
+ linkend="compression.test" /> utility to check that a given codec is correctly
+ installed.</para>
+
+ <para>To configure HBase to use a compressor, see <xref
+ linkend="compressor.install" />. To enable a compressor for a ColumnFamily, see <xref
+ linkend="changing.compression" />. To enable data block encoding for a ColumnFamily, see
+ <xref linkend="data.block.encoding.enable" />.</para>
+ <itemizedlist>
+ <title>Block Compressors</title>
+ <listitem>
+ <para>none</para>
+ </listitem>
+ <listitem>
+ <para>Snappy</para>
+ </listitem>
+ <listitem>
+ <para>LZO</para>
+ </listitem>
+ <listitem>
+ <para>LZ4</para>
+ </listitem>
+ <listitem>
+ <para>GZ</para>
+ </listitem>
+ </itemizedlist>
+
+
+ <itemizedlist xml:id="data.block.encoding.types">
+ <title>Data Block Encoding Types</title>
+ <listitem>
+ <para>Prefix - Often, keys are very similar. Specifically, keys often share a common prefix
+ and only differ near the end. For instance, one key might be
+ <literal>RowKey:Family:Qualifier0</literal> and the next key might be
+ <literal>RowKey:Family:Qualifier1</literal>. In Prefix encoding, an extra column is
+ added which holds the length of the prefix shared between the current key and the previous
+ key. Assuming the first key here is totally different from the key before, its prefix
+ length is 0. The second key's prefix length is <literal>23</literal>, since they have the
+ first 23 characters in common.</para>
+ <para>Obviously if the keys tend to have nothing in common, Prefix will not provide much
+ benefit.</para>
+ <para>The following image shows a hypothetical ColumnFamily with no data block encoding.</para>
+ <figure>
+ <title>ColumnFamily with No Encoding</title>
+ <mediaobject>
+ <imageobject>
+ <imagedata fileref="data_block_no_encoding.png" width="800"/>
+ </imageobject>
+ <caption><para>A ColumnFamily with no encoding></para></caption>
+ </mediaobject>
+ </figure>
+ <para>Here is the same data with prefix data encoding.</para>
+ <figure>
+ <title>ColumnFamily with Prefix Encoding</title>
+ <mediaobject>
+ <imageobject>
+ <imagedata fileref="data_block_prefix_encoding.png" width="800"/>
+ </imageobject>
+ <caption><para>A ColumnFamily with prefix encoding</para></caption>
+ </mediaobject>
+ </figure>
+ </listitem>
+ <listitem>
+ <para>Diff - Diff encoding expands upon Prefix encoding. Instead of considering the key
+ sequentially as a monolithic series of bytes, each key field is split so that each part of
+ the key can be compressed more efficiently. Two new fields are added: timestamp and type.
+ If the ColumnFamily is the same as the previous row, it is omitted from the current row.
+ If the key length, value length or type are the same as the previous row, the field is
+ omitted. In addition, for increased compression, the timestamp is stored as a Diff from
+ the previous row's timestamp, rather than being stored in full. Given the two row keys in
+ the Prefix example, and given an exact match on timestamp and the same type, neither the
+ value length, or type needs to be stored for the second row, and the timestamp value for
+ the second row is just 0, rather than a full timestamp.</para>
+ <para>Diff encoding is disabled by default because writing and scanning are slower but more
+ data is cached.</para>
+ <para>This image shows the same ColumnFamily from the previous images, with Diff encoding.</para>
+ <figure>
+ <title>ColumnFamily with Diff Encoding</title>
+ <mediaobject>
+ <imageobject>
+ <imagedata fileref="data_block_diff_encoding.png" width="800"/>
+ </imageobject>
+ <caption><para>A ColumnFamily with diff encoding</para></caption>
+ </mediaobject>
+ </figure>
+ </listitem>
+ <listitem>
+ <para>Fast Diff - Fast Diff works similar to Diff, but uses a faster implementation. It also
+ adds another field which stores a single bit to track whether the data itself is the same
+ as the previous row. If it is, the data is not stored again. Fast Diff is the recommended
+ codec to use if you have long keys or many columns. The data format is nearly identical to
+ Diff encoding, so there is not an image to illustrate it.</para>
+ </listitem>
+ <listitem>
+ <para>Prefix Tree encoding was introduced as an experimental feature in HBase 0.96. It
+ provides similar memory savings to the Prefix, Diff, and Fast Diff encoder, but provides
+ faster random access at a cost of slower encoding speed. Prefix Tree may be appropriate
+ for applications that have high block cache hit ratios. It introduces new 'tree' fields
+ for the row and column. The row tree field contains a list of offsets/references
+ corresponding to the cells in that row. This allows for a good deal of compression. For
+ more details about Prefix Tree encoding, see <link
+ xlink:href="https://issues.apache.org/jira/browse/HBASE-4676">HBASE-4676</link>. It is
+ difficult to graphically illustrate a prefix tree, so no image is included. See the
+ Wikipedia article for <link
+ xlink:href="http://en.wikipedia.org/wiki/Trie">Trie</link> for more general information
+ about this data structure.</para>
+ </listitem>
+ </itemizedlist>
+
+ <section>
+ <title>Which Compressor or Data Block Encoder To Use</title>
+ <para>The compression or codec type to use depends on the characteristics of your data.
+ Choosing the wrong type could cause your data to take more space rather than less, and can
+ have performance implications. In general, you need to weigh your options between smaller
+ size and faster compression/decompression. Following are some general guidelines, expanded from a discussion at <link xlink:href="http://search-hadoop.com/m/lL12B1PFVhp1">Documenting Guidance on compression and codecs</link>. </para>
+ <itemizedlist>
+ <listitem>
+ <para>If you have long keys (compared to the values) or many columns, use a prefix
+ encoder. FAST_DIFF is recommended, as more testing is needed for Prefix Tree
+ encoding.</para>
+ </listitem>
+ <listitem>
+ <para>If the values are large (and not precompressed, such as images), use a data block
+ compressor.</para>
+ </listitem>
+ <listitem>
+ <para>Use GZIP for <firstterm>cold data</firstterm>, which is accessed infrequently. GZIP
+ compression uses more CPU resources than Snappy or LZO, but provides a higher
+ compression ratio.</para>
+ </listitem>
+ <listitem>
+ <para>Use Snappy or LZO for <firstterm>hot data</firstterm>, which is accessed
+ frequently. Snappy and LZO use fewer CPU resources than GZIP, but do not provide as high
+ of a compression ratio.</para>
+ </listitem>
+ <listitem>
+ <para>In most cases, enabling Snappy or LZO by default is a good choice, because they have
+ a low performance overhead and provide space savings.</para>
+ </listitem>
+ <listitem>
+ <para>Before Snappy became available by Google in 2011, LZO was the default. Snappy has
+ similar qualities as LZO but has been shown to perform better.</para>
+ </listitem>
+ </itemizedlist>
+ </section>
+ <section xml:id="hadoop.native.lib">
+ <title>Making use of Hadoop Native Libraries in HBase</title>
+ <para>The Hadoop shared library has a bunch of facility including
+ compression libraries and fast crc'ing. To make this facility available
+ to HBase, do the following. HBase/Hadoop will fall back to use
+ alternatives if it cannot find the native library versions -- or
+ fail outright if you asking for an explicit compressor and there is
+ no alternative available.</para>
+ <para>If you see the following in your HBase logs, you know that HBase was unable
+ to locate the Hadoop native libraries:
+ <programlisting>2014-08-07 09:26:20,139 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable</programlisting>
+ If the libraries loaded successfully, the WARN message does not show.
+ </para>
+ <para>Lets presume your Hadoop shipped with a native library that
+ suits the platform you are running HBase on. To check if the Hadoop
+ native library is available to HBase, run the following tool (available in
+ Hadoop 2.1 and greater):
+ <programlisting>$ ./bin/hbase --config ~/conf_hbase org.apache.hadoop.util.NativeLibraryChecker
+2014-08-26 13:15:38,717 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
+Native library checking:
+hadoop: false
+zlib: false
+snappy: false
+lz4: false
+bzip2: false
+2014-08-26 13:15:38,863 INFO [main] util.ExitUtil: Exiting with status 1</programlisting>
+Above shows that the native hadoop library is not available in HBase context.
+ </para>
+ <para>To fix the above, either copy the Hadoop native libraries local or symlink to
+ them if the Hadoop and HBase stalls are adjacent in the filesystem.
+ You could also point at their location by setting the <varname>LD_LIBRARY_PATH</varname> environment
+ variable.</para>
+ <para>Where the JVM looks to find native librarys is "system dependent"
+ (See <classname>java.lang.System#loadLibrary(name)</classname>). On linux, by default,
+ is going to look in <filename>lib/native/PLATFORM</filename> where <varname>PLATFORM</varname>
+ is the label for the platform your HBase is installed on.
+ On a local linux machine, it seems to be the concatenation of the java properties
+ <varname>os.name</varname> and <varname>os.arch</varname> followed by whether 32 or 64 bit.
+ HBase on startup prints out all of the java system properties so find the os.name and os.arch
+ in the log. For example:
+ <programlisting>....
+ 2014-08-06 15:27:22,853 INFO [main] zookeeper.ZooKeeper: Client environment:os.name=Linux
+ 2014-08-06 15:27:22,853 INFO [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64
+ ...
+ </programlisting>
+ So in this case, the PLATFORM string is <varname>Linux-amd64-64</varname>.
+ Copying the Hadoop native libraries or symlinking at <filename>lib/native/Linux-amd64-64</filename>
+ will ensure they are found. Check with the Hadoop <filename>NativeLibraryChecker</filename>.
+ </para>
+
+ <para>Here is example of how to point at the Hadoop libs with <varname>LD_LIBRARY_PATH</varname>
+ environment variable:
+ <programlisting>$ LD_LIBRARY_PATH=~/hadoop-2.5.0-SNAPSHOT/lib/native ./bin/hbase --config ~/conf_hbase org.apache.hadoop.util.NativeLibraryChecker
+2014-08-26 13:42:49,332 INFO [main] bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
+2014-08-26 13:42:49,337 INFO [main] zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
+Native library checking:
+hadoop: true /home/stack/hadoop-2.5.0-SNAPSHOT/lib/native/libhadoop.so.1.0.0
+zlib: true /lib64/libz.so.1
+snappy: true /usr/lib64/libsnappy.so.1
+lz4: true revision:99
+bzip2: true /lib64/libbz2.so.1</programlisting>
+Set in <filename>hbase-env.sh</filename> the LD_LIBRARY_PATH environment variable when starting your HBase.
+ </para>
+ </section>
+
+ <section>
+ <title>Compressor Configuration, Installation, and Use</title>
+ <section
+ xml:id="compressor.install">
+ <title>Configure HBase For Compressors</title>
+ <para>Before HBase can use a given compressor, its libraries need to be available. Due to
+ licensing issues, only GZ compression is available to HBase (via native Java libraries) in
+ a default installation. Other compression libraries are available via the shared library
+ bundled with your hadoop. The hadoop native library needs to be findable when HBase
+ starts. See </para>
+ <section>
+ <title>Compressor Support On the Master</title>
+ <para>A new configuration setting was introduced in HBase 0.95, to check the Master to
+ determine which data block encoders are installed and configured on it, and assume that
+ the entire cluster is configured the same. This option,
+ <code>hbase.master.check.compression</code>, defaults to <literal>true</literal>. This
+ prevents the situation described in <link
+ xlink:href="https://issues.apache.org/jira/browse/HBASE-6370">HBASE-6370</link>, where
+ a table is created or modified to support a codec that a region server does not support,
+ leading to failures that take a long time to occur and are difficult to debug. </para>
+ <para>If <code>hbase.master.check.compression</code> is enabled, libraries for all desired
+ compressors need to be installed and configured on the Master, even if the Master does
+ not run a region server.</para>
+ </section>
+ <section>
+ <title>Install GZ Support Via Native Libraries</title>
+ <para>HBase uses Java's built-in GZip support unless the native Hadoop libraries are
+ available on the CLASSPATH. The recommended way to add libraries to the CLASSPATH is to
+ set the environment variable <envar>HBASE_LIBRARY_PATH</envar> for the user running
+ HBase. If native libraries are not available and Java's GZIP is used, <literal>Got
+ brand-new compressor</literal> reports will be present in the logs. See <xref
+ linkend="brand.new.compressor" />).</para>
+ </section>
+ <section
+ xml:id="lzo.compression">
+ <title>Install LZO Support</title>
+ <para>HBase cannot ship with LZO because of incompatibility between HBase, which uses an
+ Apache Software License (ASL) and LZO, which uses a GPL license. See the <link
+ xlink:href="http://wiki.apache.org/hadoop/UsingLzoCompression">Using LZO
+ Compression</link> wiki page for information on configuring LZO support for HBase. </para>
+ <para>If you depend upon LZO compression, consider configuring your RegionServers to fail
+ to start if LZO is not available. See <xref
+ linkend="hbase.regionserver.codecs" />.</para>
+ </section>
+ <section
+ xml:id="lz4.compression">
+ <title>Configure LZ4 Support</title>
+ <para>LZ4 support is bundled with Hadoop. Make sure the hadoop shared library
+ (libhadoop.so) is accessible when you start
+ HBase. After configuring your platform (see <xref
+ linkend="hbase.native.platform" />), you can make a symbolic link from HBase to the native Hadoop
+ libraries. This assumes the two software installs are colocated. For example, if my
+ 'platform' is Linux-amd64-64:
+ <programlisting language="bourne">$ cd $HBASE_HOME
+$ mkdir lib/native
+$ ln -s $HADOOP_HOME/lib/native lib/native/Linux-amd64-64</programlisting>
+ Use the compression tool to check that LZ4 is installed on all nodes. Start up (or restart)
+ HBase. Afterward, you can create and alter tables to enable LZ4 as a
+ compression codec.:
+ <screen>
+hbase(main):003:0> <userinput>alter 'TestTable', {NAME => 'info', COMPRESSION => 'LZ4'}</userinput>
+ </screen>
+ </para>
+ </section>
+ <section
+ xml:id="snappy.compression.installation">
+ <title>Install Snappy Support</title>
+ <para>HBase does not ship with Snappy support because of licensing issues. You can install
+ Snappy binaries (for instance, by using <command>yum install snappy</command> on CentOS)
+ or build Snappy from source. After installing Snappy, search for the shared library,
+ which will be called <filename>libsnappy.so.X</filename> where X is a number. If you
+ built from source, copy the shared library to a known location on your system, such as
+ <filename>/opt/snappy/lib/</filename>.</para>
+ <para>In addition to the Snappy library, HBase also needs access to the Hadoop shared
+ library, which will be called something like <filename>libhadoop.so.X.Y</filename>,
+ where X and Y are both numbers. Make note of the location of the Hadoop library, or copy
+ it to the same location as the Snappy library.</para>
+ <note>
+ <para>The Snappy and Hadoop libraries need to be available on each node of your cluster.
+ See <xref
+ linkend="compression.test" /> to find out how to test that this is the case.</para>
+ <para>See <xref
+ linkend="hbase.regionserver.codecs" /> to configure your RegionServers to fail to
+ start if a given compressor is not available.</para>
+ </note>
+ <para>Each of these library locations need to be added to the environment variable
+ <envar>HBASE_LIBRARY_PATH</envar> for the operating system user that runs HBase. You
+ need to restart the RegionServer for the changes to take effect.</para>
+ </section>
+
+
+ <section
+ xml:id="compression.test">
+ <title>CompressionTest</title>
+ <para>You can use the CompressionTest tool to verify that your compressor is available to
+ HBase:</para>
+ <screen language="bourne">
+ $ hbase org.apache.hadoop.hbase.util.CompressionTest hdfs://<replaceable>host/path/to/hbase</replaceable> snappy
+ </screen>
+ </section>
+
+
+ <section
+ xml:id="hbase.regionserver.codecs">
+ <title>Enforce Compression Settings On a RegionServer</title>
+ <para>You can configure a RegionServer so that it will fail to restart if compression is
+ configured incorrectly, by adding the option hbase.regionserver.codecs to the
+ <filename>hbase-site.xml</filename>, and setting its value to a comma-separated list
+ of codecs that need to be available. For example, if you set this property to
+ <literal>lzo,gz</literal>, the RegionServer would fail to start if both compressors
+ were not available. This would prevent a new server from being added to the cluster
+ without having codecs configured properly.</para>
+ </section>
+ </section>
+
+ <section
+ xml:id="changing.compression">
+ <title>Enable Compression On a ColumnFamily</title>
+ <para>To enable compression for a ColumnFamily, use an <code>alter</code> command. You do
+ not need to re-create the table or copy data. If you are changing codecs, be sure the old
+ codec is still available until all the old StoreFiles have been compacted.</para>
+ <example>
+ <title>Enabling Compression on a ColumnFamily of an Existing Table using HBase
+ Shell</title>
+ <screen><![CDATA[
+hbase> disable 'test'
+hbase> alter 'test', {NAME => 'cf', COMPRESSION => 'GZ'}
+hbase> enable 'test']]>
+ </screen>
+ </example>
+ <example>
+ <title>Creating a New Table with Compression On a ColumnFamily</title>
+ <screen><![CDATA[
+hbase> create 'test2', { NAME => 'cf2', COMPRESSION => 'SNAPPY' }
+ ]]></screen>
+ </example>
+ <example>
+ <title>Verifying a ColumnFamily's Compression Settings</title>
+ <screen><![CDATA[
+hbase> describe 'test'
+DESCRIPTION ENABLED
+ 'test', {NAME => 'cf', DATA_BLOCK_ENCODING => 'NONE false
+ ', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0',
+ VERSIONS => '1', COMPRESSION => 'GZ', MIN_VERSIONS
+ => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'fa
+ lse', BLOCKSIZE => '65536', IN_MEMORY => 'false', B
+ LOCKCACHE => 'true'}
+1 row(s) in 0.1070 seconds
+ ]]></screen>
+ </example>
+ </section>
+
+ <section>
+ <title>Testing Compression Performance</title>
+ <para>HBase includes a tool called LoadTestTool which provides mechanisms to test your
+ compression performance. You must specify either <literal>-write</literal> or
+ <literal>-update-read</literal> as your first parameter, and if you do not specify another
+ parameter, usage advice is printed for each option.</para>
+ <example>
+ <title><command>LoadTestTool</command> Usage</title>
+ <screen language="bourne"><![CDATA[
+$ bin/hbase org.apache.hadoop.hbase.util.LoadTestTool -h
+usage: bin/hbase org.apache.hadoop.hbase.util.LoadTestTool <options>
+Options:
+ -batchupdate Whether to use batch as opposed to separate
+ updates for every column in a row
+ -bloom <arg> Bloom filter type, one of [NONE, ROW, ROWCOL]
+ -compression <arg> Compression type, one of [LZO, GZ, NONE, SNAPPY,
+ LZ4]
+ -data_block_encoding <arg> Encoding algorithm (e.g. prefix compression) to
+ use for data blocks in the test column family, one
+ of [NONE, PREFIX, DIFF, FAST_DIFF, PREFIX_TREE].
+ -encryption <arg> Enables transparent encryption on the test table,
+ one of [AES]
+ -generator <arg> The class which generates load for the tool. Any
+ args for this class can be passed as colon
+ separated after class name
+ -h,--help Show usage
+ -in_memory Tries to keep the HFiles of the CF inmemory as far
+ as possible. Not guaranteed that reads are always
+ served from inmemory
+ -init_only Initialize the test table only, don't do any
+ loading
+ -key_window <arg> The 'key window' to maintain between reads and
+ writes for concurrent write/read workload. The
+ default is 0.
+ -max_read_errors <arg> The maximum number of read errors to tolerate
+ before terminating all reader threads. The default
+ is 10.
+ -multiput Whether to use multi-puts as opposed to separate
+ puts for every column in a row
+ -num_keys <arg> The number of keys to read/write
+ -num_tables <arg> A positive integer number. When a number n is
+ speicfied, load test tool will load n table
+ parallely. -tn parameter value becomes table name
+ prefix. Each table name is in format
+ <tn>_1...<tn>_n
+ -read <arg> <verify_percent>[:<#threads=20>]
+ -regions_per_server <arg> A positive integer number. When a number n is
+ specified, load test tool will create the test
+ table with n regions per server
+ -skip_init Skip the initialization; assume test table already
+ exists
+ -start_key <arg> The first key to read/write (a 0-based index). The
+ default value is 0.
+ -tn <arg> The name of the table to read or write
+ -update <arg> <update_percent>[:<#threads=20>][:<#whether to
+ ignore nonce collisions=0>]
+ -write <arg> <avg_cols_per_key>:<avg_data_size>[:<#threads=20>]
+ -zk <arg> ZK quorum as comma-separated host names without
+ port numbers
+ -zk_root <arg> name of parent znode in zookeeper
+ ]]></screen>
+ </example>
+ <example>
+ <title>Example Usage of LoadTestTool</title>
+ <screen language="bourne">
+$ hbase org.apache.hadoop.hbase.util.LoadTestTool -write 1:10:100 -num_keys 1000000
+ -read 100:30 -num_tables 1 -data_block_encoding NONE -tn load_test_tool_NONE
+ </screen>
+ </example>
+ </section>
+ </section>
+
+ <section xml:id="data.block.encoding.enable">
+ <title>Enable Data Block Encoding</title>
+ <para>Codecs are built into HBase so no extra configuration is needed. Codecs are enabled on a
+ table by setting the <code>DATA_BLOCK_ENCODING</code> property. Disable the table before
+ altering its DATA_BLOCK_ENCODING setting. Following is an example using HBase Shell:</para>
+ <example>
+ <title>Enable Data Block Encoding On a Table</title>
+ <screen><![CDATA[
+hbase> disable 'test'
+hbase> alter 'test', { NAME => 'cf', DATA_BLOCK_ENCODING => 'FAST_DIFF' }
+Updating all regions with the new schema...
+0/1 regions updated.
+1/1 regions updated.
+Done.
+0 row(s) in 2.2820 seconds
+hbase> enable 'test'
+0 row(s) in 0.1580 seconds
+ ]]></screen>
+ </example>
+ <example>
+ <title>Verifying a ColumnFamily's Data Block Encoding</title>
+ <screen><![CDATA[
+hbase> describe 'test'
+DESCRIPTION ENABLED
+ 'test', {NAME => 'cf', DATA_BLOCK_ENCODING => 'FAST true
+ _DIFF', BLOOMFILTER => 'ROW', REPLICATION_SCOPE =>
+ '0', VERSIONS => '1', COMPRESSION => 'GZ', MIN_VERS
+ IONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS =
+ > 'false', BLOCKSIZE => '65536', IN_MEMORY => 'fals
+ e', BLOCKCACHE => 'true'}
+1 row(s) in 0.0650 seconds
+ ]]></screen>
+ </example>
+ </section>
+
+
+</appendix>
http://git-wip-us.apache.org/repos/asf/hbase/blob/a1fe1e09/src/main/docbkx/configuration.xml
----------------------------------------------------------------------
diff --git a/src/main/docbkx/configuration.xml b/src/main/docbkx/configuration.xml
index 74b8e52..a0b7d11 100644
--- a/src/main/docbkx/configuration.xml
+++ b/src/main/docbkx/configuration.xml
@@ -925,8 +925,8 @@ stopping hbase...............</screen>
<!--presumes the pre-site target has put the hbase-default.xml at this location-->
<xi:include
xmlns:xi="http://www.w3.org/2001/XInclude"
- href="../../../target/docbkx/hbase-default.xml">
- <xi:fallback>
+ href="hbase-default.xml">
+ <!--<xi:fallback>
<section
xml:id="hbase_default_configurations">
<title />
@@ -1007,7 +1007,7 @@ stopping hbase...............</screen>
</section>
</section>
</section>
- </xi:fallback>
+ </xi:fallback>-->
</xi:include>
</section>
http://git-wip-us.apache.org/repos/asf/hbase/blob/a1fe1e09/src/main/docbkx/customization-pdf.xsl
----------------------------------------------------------------------
diff --git a/src/main/docbkx/customization-pdf.xsl b/src/main/docbkx/customization-pdf.xsl
new file mode 100644
index 0000000..b21236f
--- /dev/null
+++ b/src/main/docbkx/customization-pdf.xsl
@@ -0,0 +1,129 @@
+<?xml version="1.0"?>
+<xsl:stylesheet
+ xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
+ version="1.0">
+<!--
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+-->
+ <xsl:import href="urn:docbkx:stylesheet/docbook.xsl"/>
+ <xsl:import href="urn:docbkx:stylesheet/highlight.xsl"/>
+
+
+ <!--###################################################
+ Paper & Page Size
+ ################################################### -->
+
+ <!-- Paper type, no headers on blank pages, no double sided printing -->
+ <xsl:param name="paper.type" select="'USletter'"/>
+ <xsl:param name="double.sided">0</xsl:param>
+ <xsl:param name="headers.on.blank.pages">0</xsl:param>
+ <xsl:param name="footers.on.blank.pages">0</xsl:param>
+
+ <!-- Space between paper border and content (chaotic stuff, don't touch) -->
+ <xsl:param name="page.margin.top">5mm</xsl:param>
+ <xsl:param name="region.before.extent">10mm</xsl:param>
+ <xsl:param name="body.margin.top">10mm</xsl:param>
+
+ <xsl:param name="body.margin.bottom">15mm</xsl:param>
+ <xsl:param name="region.after.extent">10mm</xsl:param>
+ <xsl:param name="page.margin.bottom">0mm</xsl:param>
+
+ <xsl:param name="page.margin.outer">18mm</xsl:param>
+ <xsl:param name="page.margin.inner">18mm</xsl:param>
+
+ <!-- No intendation of Titles -->
+ <xsl:param name="title.margin.left">0pc</xsl:param>
+
+ <!--###################################################
+ Fonts & Styles
+ ################################################### -->
+
+ <!-- Left aligned text and no hyphenation -->
+ <xsl:param name="alignment">justify</xsl:param>
+ <xsl:param name="hyphenate">true</xsl:param>
+
+ <!-- Default Font size -->
+ <xsl:param name="body.font.master">11</xsl:param>
+ <xsl:param name="body.font.small">8</xsl:param>
+
+ <!-- Line height in body text -->
+ <xsl:param name="line-height">1.4</xsl:param>
+
+ <!-- Force line break in long URLs -->
+ <xsl:param name="ulink.hyphenate.chars">/&?</xsl:param>
+ <xsl:param name="ulink.hyphenate">​</xsl:param>
+
+ <!-- Monospaced fonts are smaller than regular text -->
+ <xsl:attribute-set name="monospace.properties">
+ <xsl:attribute name="font-family">
+ <xsl:value-of select="$monospace.font.family"/>
+ </xsl:attribute>
+ <xsl:attribute name="font-size">0.8em</xsl:attribute>
+ <xsl:attribute name="wrap-option">wrap</xsl:attribute>
+ <xsl:attribute name="hyphenate">true</xsl:attribute>
+ </xsl:attribute-set>
+
+
+ <!-- add page break after abstract block -->
+ <xsl:attribute-set name="abstract.properties">
+ <xsl:attribute name="break-after">page</xsl:attribute>
+ </xsl:attribute-set>
+
+ <!-- add page break after toc -->
+ <xsl:attribute-set name="toc.margin.properties">
+ <xsl:attribute name="break-after">page</xsl:attribute>
+ </xsl:attribute-set>
+
+ <!-- add page break after first level sections -->
+ <xsl:attribute-set name="section.level1.properties">
+ <xsl:attribute name="break-after">page</xsl:attribute>
+ </xsl:attribute-set>
+
+ <!-- Show only Sections up to level 3 in the TOCs -->
+ <xsl:param name="toc.section.depth">2</xsl:param>
+
+ <!-- Dot and Whitespace as separator in TOC between Label and Title-->
+ <xsl:param name="autotoc.label.separator" select="'. '"/>
+
+ <!-- program listings / examples formatting -->
+ <xsl:attribute-set name="monospace.verbatim.properties">
+ <xsl:attribute name="font-family">Courier</xsl:attribute>
+ <xsl:attribute name="font-size">8pt</xsl:attribute>
+ <xsl:attribute name="keep-together.within-column">always</xsl:attribute>
+ </xsl:attribute-set>
+
+ <xsl:param name="shade.verbatim" select="1" />
+
+ <xsl:attribute-set name="shade.verbatim.style">
+ <xsl:attribute name="background-color">#E8E8E8</xsl:attribute>
+ <xsl:attribute name="border-width">0.5pt</xsl:attribute>
+ <xsl:attribute name="border-style">solid</xsl:attribute>
+ <xsl:attribute name="border-color">#575757</xsl:attribute>
+ <xsl:attribute name="padding">3pt</xsl:attribute>
+ </xsl:attribute-set>
+
+ <!-- callouts customization -->
+ <xsl:param name="callout.unicode" select="1" />
+ <xsl:param name="callout.graphics" select="0" />
+ <xsl:param name="callout.defaultcolumn">90</xsl:param>
+
+ <!-- Syntax Highlighting -->
+
+
+</xsl:stylesheet>
http://git-wip-us.apache.org/repos/asf/hbase/blob/a1fe1e09/src/main/docbkx/datamodel.xml
----------------------------------------------------------------------
diff --git a/src/main/docbkx/datamodel.xml b/src/main/docbkx/datamodel.xml
new file mode 100644
index 0000000..bdf697d
--- /dev/null
+++ b/src/main/docbkx/datamodel.xml
@@ -0,0 +1,865 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<chapter
+ xml:id="datamodel"
+ version="5.0"
+ xmlns="http://docbook.org/ns/docbook"
+ xmlns:xlink="http://www.w3.org/1999/xlink"
+ xmlns:xi="http://www.w3.org/2001/XInclude"
+ xmlns:svg="http://www.w3.org/2000/svg"
+ xmlns:m="http://www.w3.org/1998/Math/MathML"
+ xmlns:html="http://www.w3.org/1999/xhtml"
+ xmlns:db="http://docbook.org/ns/docbook">
+ <!--/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+-->
+
+ <title>Data Model</title>
+ <para>In HBase, data is stored in tables, which have rows and columns. This is a terminology
+ overlap with relational databases (RDBMSs), but this is not a helpful analogy. Instead, it can
+ be helpful to think of an HBase table as a multi-dimensional map.</para>
+ <variablelist>
+ <title>HBase Data Model Terminology</title>
+ <varlistentry>
+ <term>Table</term>
+ <listitem>
+ <para>An HBase table consists of multiple rows.</para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>Row</term>
+ <listitem>
+ <para>A row in HBase consists of a row key and one or more columns with values associated
+ with them. Rows are sorted alphabetically by the row key as they are stored. For this
+ reason, the design of the row key is very important. The goal is to store data in such a
+ way that related rows are near each other. A common row key pattern is a website domain.
+ If your row keys are domains, you should probably store them in reverse (org.apache.www,
+ org.apache.mail, org.apache.jira). This way, all of the Apache domains are near each
+ other in the table, rather than being spread out based on the first letter of the
+ subdomain.</para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>Column</term>
+ <listitem>
+ <para>A column in HBase consists of a column family and a column qualifier, which are
+ delimited by a <literal>:</literal> (colon) character.</para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>Column Family</term>
+ <listitem>
+ <para>Column families physically colocate a set of columns and their values, often for
+ performance reasons. Each column family has a set of storage properties, such as whether
+ its values should be cached in memory, how its data is compressed or its row keys are
+ encoded, and others. Each row in a table has the same column
+ families, though a given row might not store anything in a given column family.</para>
+ <para>Column families are specified when you create your table, and influence the way your
+ data is stored in the underlying filesystem. Therefore, the column families should be
+ considered carefully during schema design.</para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>Column Qualifier</term>
+ <listitem>
+ <para>A column qualifier is added to a column family to provide the index for a given
+ piece of data. Given a column family <literal>content</literal>, a column qualifier
+ might be <literal>content:html</literal>, and another might be
+ <literal>content:pdf</literal>. Though column families are fixed at table creation,
+ column qualifiers are mutable and may differ greatly between rows.</para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>Cell</term>
+ <listitem>
+ <para>A cell is a combination of row, column family, and column qualifier, and contains a
+ value and a timestamp, which represents the value's version.</para>
+ <para>A cell's value is an uninterpreted array of bytes.</para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>Timestamp</term>
+ <listitem>
+ <para>A timestamp is written alongside each value, and is the identifier for a given
+ version of a value. By default, the timestamp represents the time on the RegionServer
+ when the data was written, but you can specify a different timestamp value when you put
+ data into the cell.</para>
+ <caution>
+ <para>Direct manipulation of timestamps is an advanced feature which is only exposed for
+ special cases that are deeply integrated with HBase, and is discouraged in general.
+ Encoding a timestamp at the application level is the preferred pattern.</para>
+ </caution>
+ <para>You can specify the maximum number of versions of a value that HBase retains, per column
+ family. When the maximum number of versions is reached, the oldest versions are
+ eventually deleted. By default, only the newest version is kept.</para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+
+ <section
+ xml:id="conceptual.view">
+ <title>Conceptual View</title>
+ <para>You can read a very understandable explanation of the HBase data model in the blog post <link
+ xlink:href="http://jimbojw.com/wiki/index.php?title=Understanding_Hbase_and_BigTable">Understanding
+ HBase and BigTable</link> by Jim R. Wilson. Another good explanation is available in the
+ PDF <link
+ xlink:href="http://0b4af6cdc2f0c5998459-c0245c5c937c5dedcca3f1764ecc9b2f.r43.cf2.rackcdn.com/9353-login1210_khurana.pdf">Introduction
+ to Basic Schema Design</link> by Amandeep Khurana. It may help to read different
+ perspectives to get a solid understanding of HBase schema design. The linked articles cover
+ the same ground as the information in this section.</para>
+ <para> The following example is a slightly modified form of the one on page 2 of the <link
+ xlink:href="http://research.google.com/archive/bigtable.html">BigTable</link> paper. There
+ is a table called <varname>webtable</varname> that contains two rows
+ (<literal>com.cnn.www</literal>
+ and <literal>com.example.www</literal>), three column families named
+ <varname>contents</varname>, <varname>anchor</varname>, and <varname>people</varname>. In
+ this example, for the first row (<literal>com.cnn.www</literal>),
+ <varname>anchor</varname> contains two columns (<varname>anchor:cssnsi.com</varname>,
+ <varname>anchor:my.look.ca</varname>) and <varname>contents</varname> contains one column
+ (<varname>contents:html</varname>). This example contains 5 versions of the row with the
+ row key <literal>com.cnn.www</literal>, and one version of the row with the row key
+ <literal>com.example.www</literal>. The <varname>contents:html</varname> column qualifier contains the entire
+ HTML of a given website. Qualifiers of the <varname>anchor</varname> column family each
+ contain the external site which links to the site represented by the row, along with the
+ text it used in the anchor of its link. The <varname>people</varname> column family represents
+ people associated with the site.
+ </para>
+ <note>
+ <title>Column Names</title>
+ <para> By convention, a column name is made of its column family prefix and a
+ <emphasis>qualifier</emphasis>. For example, the column
+ <emphasis>contents:html</emphasis> is made up of the column family
+ <varname>contents</varname> and the <varname>html</varname> qualifier. The colon
+ character (<literal>:</literal>) delimits the column family from the column family
+ <emphasis>qualifier</emphasis>. </para>
+ </note>
+ <table
+ frame="all">
+ <title>Table <varname>webtable</varname></title>
+ <tgroup
+ cols="5"
+ align="left"
+ colsep="1"
+ rowsep="1">
+ <colspec
+ colname="c1" />
+ <colspec
+ colname="c2" />
+ <colspec
+ colname="c3" />
+ <colspec
+ colname="c4" />
+ <colspec
+ colname="c5" />
+ <thead>
+ <row>
+ <entry>Row Key</entry>
+ <entry>Time Stamp</entry>
+ <entry>ColumnFamily <varname>contents</varname></entry>
+ <entry>ColumnFamily <varname>anchor</varname></entry>
+ <entry>ColumnFamily <varname>people</varname></entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>"com.cnn.www"</entry>
+ <entry>t9</entry>
+ <entry />
+ <entry><varname>anchor:cnnsi.com</varname> = "CNN"</entry>
+ <entry />
+ </row>
+ <row>
+ <entry>"com.cnn.www"</entry>
+ <entry>t8</entry>
+ <entry />
+ <entry><varname>anchor:my.look.ca</varname> = "CNN.com"</entry>
+ <entry />
+ </row>
+ <row>
+ <entry>"com.cnn.www"</entry>
+ <entry>t6</entry>
+ <entry><varname>contents:html</varname> = "<html>..."</entry>
+ <entry />
+ <entry />
+ </row>
+ <row>
+ <entry>"com.cnn.www"</entry>
+ <entry>t5</entry>
+ <entry><varname>contents:html</varname> = "<html>..."</entry>
+ <entry />
+ <entry />
+ </row>
+ <row>
+ <entry>"com.cnn.www"</entry>
+ <entry>t3</entry>
+ <entry><varname>contents:html</varname> = "<html>..."</entry>
+ <entry />
+ <entry />
+ </row>
+ <row>
+ <entry>"com.example.www"</entry>
+ <entry>t5</entry>
+ <entry><varname>contents:html</varname> = "<html>..."</entry>
+ <entry></entry>
+ <entry>people:author = "John Doe"</entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+ <para>Cells in this table that appear to be empty do not take space, or in fact exist, in
+ HBase. This is what makes HBase "sparse." A tabular view is not the only possible way to
+ look at data in HBase, or even the most accurate. The following represents the same
+ information as a multi-dimensional map. This is only a mock-up for illustrative
+ purposes and may not be strictly accurate.</para>
+ <programlisting><![CDATA[
+{
+ "com.cnn.www": {
+ contents: {
+ t6: contents:html: "<html>..."
+ t5: contents:html: "<html>..."
+ t3: contents:html: "<html>..."
+ }
+ anchor: {
+ t9: anchor:cnnsi.com = "CNN"
+ t8: anchor:my.look.ca = "CNN.com"
+ }
+ people: {}
+ }
+ "com.example.www": {
+ contents: {
+ t5: contents:html: "<html>..."
+ }
+ anchor: {}
+ people: {
+ t5: people:author: "John Doe"
+ }
+ }
+}
+ ]]></programlisting>
+
+ </section>
+ <section
+ xml:id="physical.view">
+ <title>Physical View</title>
+ <para> Although at a conceptual level tables may be viewed as a sparse set of rows, they are
+ physically stored by column family. A new column qualifier (column_family:column_qualifier)
+ can be added to an existing column family at any time.</para>
+ <table
+ frame="all">
+ <title>ColumnFamily <varname>anchor</varname></title>
+ <tgroup
+ cols="3"
+ align="left"
+ colsep="1"
+ rowsep="1">
+ <colspec
+ colname="c1" />
+ <colspec
+ colname="c2" />
+ <colspec
+ colname="c3" />
+ <thead>
+ <row>
+ <entry>Row Key</entry>
+ <entry>Time Stamp</entry>
+ <entry>Column Family <varname>anchor</varname></entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>"com.cnn.www"</entry>
+ <entry>t9</entry>
+ <entry><varname>anchor:cnnsi.com</varname> = "CNN"</entry>
+ </row>
+ <row>
+ <entry>"com.cnn.www"</entry>
+ <entry>t8</entry>
+ <entry><varname>anchor:my.look.ca</varname> = "CNN.com"</entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+ <table
+ frame="all">
+ <title>ColumnFamily <varname>contents</varname></title>
+ <tgroup
+ cols="3"
+ align="left"
+ colsep="1"
+ rowsep="1">
+ <colspec
+ colname="c1" />
+ <colspec
+ colname="c2" />
+ <colspec
+ colname="c3" />
+ <thead>
+ <row>
+ <entry>Row Key</entry>
+ <entry>Time Stamp</entry>
+ <entry>ColumnFamily "contents:"</entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>"com.cnn.www"</entry>
+ <entry>t6</entry>
+ <entry><varname>contents:html</varname> = "<html>..."</entry>
+ </row>
+ <row>
+ <entry>"com.cnn.www"</entry>
+ <entry>t5</entry>
+ <entry><varname>contents:html</varname> = "<html>..."</entry>
+ </row>
+ <row>
+ <entry>"com.cnn.www"</entry>
+ <entry>t3</entry>
+ <entry><varname>contents:html</varname> = "<html>..."</entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+ <para>The empty cells shown in the
+ conceptual view are not stored at all.
+ Thus a request for the value of the <varname>contents:html</varname> column at time stamp
+ <literal>t8</literal> would return no value. Similarly, a request for an
+ <varname>anchor:my.look.ca</varname> value at time stamp <literal>t9</literal> would
+ return no value. However, if no timestamp is supplied, the most recent value for a
+ particular column would be returned. Given multiple versions, the most recent is also the
+ first one found, since timestamps
+ are stored in descending order. Thus a request for the values of all columns in the row
+ <varname>com.cnn.www</varname> if no timestamp is specified would be: the value of
+ <varname>contents:html</varname> from timestamp <literal>t6</literal>, the value of
+ <varname>anchor:cnnsi.com</varname> from timestamp <literal>t9</literal>, the value of
+ <varname>anchor:my.look.ca</varname> from timestamp <literal>t8</literal>. </para>
+ <para>For more information about the internals of how Apache HBase stores data, see <xref
+ linkend="regions.arch" />. </para>
+ </section>
+
+ <section
+ xml:id="namespace">
+ <title>Namespace</title>
+ <para> A namespace is a logical grouping of tables analogous to a database in relation
+ database systems. This abstraction lays the groundwork for upcoming multi-tenancy related
+ features: <itemizedlist>
+ <listitem>
+ <para>Quota Management (HBASE-8410) - Restrict the amount of resources (ie regions,
+ tables) a namespace can consume.</para>
+ </listitem>
+ <listitem>
+ <para>Namespace Security Administration (HBASE-9206) - provide another level of security
+ administration for tenants.</para>
+ </listitem>
+ <listitem>
+ <para>Region server groups (HBASE-6721) - A namespace/table can be pinned onto a subset
+ of regionservers thus guaranteeing a course level of isolation.</para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ <section
+ xml:id="namespace_creation">
+ <title>Namespace management</title>
+ <para> A namespace can be created, removed or altered. Namespace membership is determined
+ during table creation by specifying a fully-qualified table name of the form:</para>
+
+ <programlisting language="xml"><![CDATA[<table namespace>:<table qualifier>]]></programlisting>
+
+
+ <example>
+ <title>Examples</title>
+
+ <programlisting language="bourne">
+#Create a namespace
+create_namespace 'my_ns'
+ </programlisting>
+ <programlisting language="bourne">
+#create my_table in my_ns namespace
+create 'my_ns:my_table', 'fam'
+ </programlisting>
+ <programlisting language="bourne">
+#drop namespace
+drop_namespace 'my_ns'
+ </programlisting>
+ <programlisting language="bourne">
+#alter namespace
+alter_namespace 'my_ns', {METHOD => 'set', 'PROPERTY_NAME' => 'PROPERTY_VALUE'}
+ </programlisting>
+ </example>
+ </section>
+ <section
+ xml:id="namespace_special">
+ <title>Predefined namespaces</title>
+ <para> There are two predefined special namespaces: </para>
+ <itemizedlist>
+ <listitem>
+ <para>hbase - system namespace, used to contain hbase internal tables</para>
+ </listitem>
+ <listitem>
+ <para>default - tables with no explicit specified namespace will automatically fall into
+ this namespace.</para>
+ </listitem>
+ </itemizedlist>
+ <example>
+ <title>Examples</title>
+
+ <programlisting language="bourne">
+#namespace=foo and table qualifier=bar
+create 'foo:bar', 'fam'
+
+#namespace=default and table qualifier=bar
+create 'bar', 'fam'
+</programlisting>
+ </example>
+ </section>
+ </section>
+
+ <section
+ xml:id="table">
+ <title>Table</title>
+ <para> Tables are declared up front at schema definition time. </para>
+ </section>
+
+ <section
+ xml:id="row">
+ <title>Row</title>
+ <para>Row keys are uninterrpreted bytes. Rows are lexicographically sorted with the lowest
+ order appearing first in a table. The empty byte array is used to denote both the start and
+ end of a tables' namespace.</para>
+ </section>
+
+ <section
+ xml:id="columnfamily">
+ <title>Column Family<indexterm><primary>Column Family</primary></indexterm></title>
+ <para> Columns in Apache HBase are grouped into <emphasis>column families</emphasis>. All
+ column members of a column family have the same prefix. For example, the columns
+ <emphasis>courses:history</emphasis> and <emphasis>courses:math</emphasis> are both
+ members of the <emphasis>courses</emphasis> column family. The colon character
+ (<literal>:</literal>) delimits the column family from the <indexterm><primary>column
+ family qualifier</primary><secondary>Column Family Qualifier</secondary></indexterm>.
+ The column family prefix must be composed of <emphasis>printable</emphasis> characters. The
+ qualifying tail, the column family <emphasis>qualifier</emphasis>, can be made of any
+ arbitrary bytes. Column families must be declared up front at schema definition time whereas
+ columns do not need to be defined at schema time but can be conjured on the fly while the
+ table is up an running.</para>
+ <para>Physically, all column family members are stored together on the filesystem. Because
+ tunings and storage specifications are done at the column family level, it is advised that
+ all column family members have the same general access pattern and size
+ characteristics.</para>
+
+ </section>
+ <section
+ xml:id="cells">
+ <title>Cells<indexterm><primary>Cells</primary></indexterm></title>
+ <para>A <emphasis>{row, column, version} </emphasis>tuple exactly specifies a
+ <literal>cell</literal> in HBase. Cell content is uninterrpreted bytes</para>
+ </section>
+ <section
+ xml:id="data_model_operations">
+ <title>Data Model Operations</title>
+ <para>The four primary data model operations are Get, Put, Scan, and Delete. Operations are
+ applied via <link
+ xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Table.html">Table</link>
+ instances.
+ </para>
+ <section
+ xml:id="get">
+ <title>Get</title>
+ <para><link
+ xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Get.html">Get</link>
+ returns attributes for a specified row. Gets are executed via <link
+ xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Table.html#get(org.apache.hadoop.hbase.client.Get)">
+ Table.get</link>. </para>
+ </section>
+ <section
+ xml:id="put">
+ <title>Put</title>
+ <para><link
+ xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Put.html">Put</link>
+ either adds new rows to a table (if the key is new) or can update existing rows (if the
+ key already exists). Puts are executed via <link
+ xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Table.html#put(org.apache.hadoop.hbase.client.Put)">
+ Table.put</link> (writeBuffer) or <link
+ xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Table.html#batch(java.util.List, java.lang.Object[])">
+ Table.batch</link> (non-writeBuffer). </para>
+ </section>
+ <section
+ xml:id="scan">
+ <title>Scans</title>
+ <para><link
+ xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Scan.html">Scan</link>
+ allow iteration over multiple rows for specified attributes. </para>
+ <para>The following is an example of a Scan on a Table instance. Assume that a table is
+ populated with rows with keys "row1", "row2", "row3", and then another set of rows with
+ the keys "abc1", "abc2", and "abc3". The following example shows how to set a Scan
+ instance to return the rows beginning with "row".</para>
+<programlisting language="java">
+public static final byte[] CF = "cf".getBytes();
+public static final byte[] ATTR = "attr".getBytes();
+...
+
+Table table = ... // instantiate a Table instance
+
+Scan scan = new Scan();
+scan.addColumn(CF, ATTR);
+scan.setRowPrefixFilter(Bytes.toBytes("row"));
+ResultScanner rs = table.getScanner(scan);
+try {
+ for (Result r = rs.next(); r != null; r = rs.next()) {
+ // process result...
+} finally {
+ rs.close(); // always close the ResultScanner!
+</programlisting>
+ <para>Note that generally the easiest way to specify a specific stop point for a scan is by
+ using the <link
+ xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/filter/InclusiveStopFilter.html">InclusiveStopFilter</link>
+ class. </para>
+ </section>
+ <section
+ xml:id="delete">
+ <title>Delete</title>
+ <para><link
+ xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Delete.html">Delete</link>
+ removes a row from a table. Deletes are executed via <link
+ xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Table.html#delete(org.apache.hadoop.hbase.client.Delete)">
+ HTable.delete</link>. </para>
+ <para>HBase does not modify data in place, and so deletes are handled by creating new
+ markers called <emphasis>tombstones</emphasis>. These tombstones, along with the dead
+ values, are cleaned up on major compactions. </para>
+ <para>See <xref
+ linkend="version.delete" /> for more information on deleting versions of columns, and
+ see <xref
+ linkend="compaction" /> for more information on compactions. </para>
+
+ </section>
+
+ </section>
+
+
+ <section
+ xml:id="versions">
+ <title>Versions<indexterm><primary>Versions</primary></indexterm></title>
+
+ <para>A <emphasis>{row, column, version} </emphasis>tuple exactly specifies a
+ <literal>cell</literal> in HBase. It's possible to have an unbounded number of cells where
+ the row and column are the same but the cell address differs only in its version
+ dimension.</para>
+
+ <para>While rows and column keys are expressed as bytes, the version is specified using a long
+ integer. Typically this long contains time instances such as those returned by
+ <code>java.util.Date.getTime()</code> or <code>System.currentTimeMillis()</code>, that is:
+ <quote>the difference, measured in milliseconds, between the current time and midnight,
+ January 1, 1970 UTC</quote>.</para>
+
+ <para>The HBase version dimension is stored in decreasing order, so that when reading from a
+ store file, the most recent values are found first.</para>
+
+ <para>There is a lot of confusion over the semantics of <literal>cell</literal> versions, in
+ HBase. In particular:</para>
+ <itemizedlist>
+ <listitem>
+ <para>If multiple writes to a cell have the same version, only the last written is
+ fetchable.</para>
+ </listitem>
+
+ <listitem>
+ <para>It is OK to write cells in a non-increasing version order.</para>
+ </listitem>
+ </itemizedlist>
+
+ <para>Below we describe how the version dimension in HBase currently works. See <link
+ xlink:href="https://issues.apache.org/jira/browse/HBASE-2406">HBASE-2406</link> for
+ discussion of HBase versions. <link
+ xlink:href="http://outerthought.org/blog/417-ot.html">Bending time in HBase</link>
+ makes for a good read on the version, or time, dimension in HBase. It has more detail on
+ versioning than is provided here. As of this writing, the limiitation
+ <emphasis>Overwriting values at existing timestamps</emphasis> mentioned in the
+ article no longer holds in HBase. This section is basically a synopsis of this article
+ by Bruno Dumon.</para>
+
+ <section xml:id="specify.number.of.versions">
+ <title>Specifying the Number of Versions to Store</title>
+ <para>The maximum number of versions to store for a given column is part of the column
+ schema and is specified at table creation, or via an <command>alter</command> command, via
+ <code>HColumnDescriptor.DEFAULT_VERSIONS</code>. Prior to HBase 0.96, the default number
+ of versions kept was <literal>3</literal>, but in 0.96 and newer has been changed to
+ <literal>1</literal>.</para>
+ <example>
+ <title>Modify the Maximum Number of Versions for a Column</title>
+ <para>This example uses HBase Shell to keep a maximum of 5 versions of column
+ <code>f1</code>. You could also use <link
+ xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/HColumnDescriptor.html"
+ >HColumnDescriptor</link>.</para>
+ <screen><![CDATA[hbase> alter ‘t1′, NAME => ‘f1′, VERSIONS => 5]]></screen>
+ </example>
+ <example>
+ <title>Modify the Minimum Number of Versions for a Column</title>
+ <para>You can also specify the minimum number of versions to store. By default, this is
+ set to 0, which means the feature is disabled. The following example sets the minimum
+ number of versions on field <code>f1</code> to <literal>2</literal>, via HBase Shell.
+ You could also use <link
+ xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/HColumnDescriptor.html"
+ >HColumnDescriptor</link>.</para>
+ <screen><![CDATA[hbase> alter ‘t1′, NAME => ‘f1′, MIN_VERSIONS => 2]]></screen>
+ </example>
+ <para>Starting with HBase 0.98.2, you can specify a global default for the maximum number of
+ versions kept for all newly-created columns, by setting
+ <option>hbase.column.max.version</option> in <filename>hbase-site.xml</filename>. See
+ <xref linkend="hbase.column.max.version"/>.</para>
+ </section>
+
+ <section
+ xml:id="versions.ops">
+ <title>Versions and HBase Operations</title>
+
+ <para>In this section we look at the behavior of the version dimension for each of the core
+ HBase operations.</para>
+
+ <section>
+ <title>Get/Scan</title>
+
+ <para>Gets are implemented on top of Scans. The below discussion of <link
+ xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Get.html">Get</link>
+ applies equally to <link
+ xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Scan.html">Scans</link>.</para>
+
+ <para>By default, i.e. if you specify no explicit version, when doing a
+ <literal>get</literal>, the cell whose version has the largest value is returned
+ (which may or may not be the latest one written, see later). The default behavior can be
+ modified in the following ways:</para>
+
+ <itemizedlist>
+ <listitem>
+ <para>to return more than one version, see <link
+ xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Get.html#setMaxVersions()">Get.setMaxVersions()</link></para>
+ </listitem>
+
+ <listitem>
+ <para>to return versions other than the latest, see <link
+ xlink:href="???">Get.setTimeRange()</link></para>
+
+ <para>To retrieve the latest version that is less than or equal to a given value, thus
+ giving the 'latest' state of the record at a certain point in time, just use a range
+ from 0 to the desired version and set the max versions to 1.</para>
+ </listitem>
+ </itemizedlist>
+
+ </section>
+ <section
+ xml:id="default_get_example">
+ <title>Default Get Example</title>
+ <para>The following Get will only retrieve the current version of the row</para>
+ <programlisting language="java">
+public static final byte[] CF = "cf".getBytes();
+public static final byte[] ATTR = "attr".getBytes();
+...
+Get get = new Get(Bytes.toBytes("row1"));
+Result r = table.get(get);
+byte[] b = r.getValue(CF, ATTR); // returns current version of value
+</programlisting>
+ </section>
+ <section
+ xml:id="versioned_get_example">
+ <title>Versioned Get Example</title>
+ <para>The following Get will return the last 3 versions of the row.</para>
+ <programlisting language="java">
+public static final byte[] CF = "cf".getBytes();
+public static final byte[] ATTR = "attr".getBytes();
+...
+Get get = new Get(Bytes.toBytes("row1"));
+get.setMaxVersions(3); // will return last 3 versions of row
+Result r = table.get(get);
+byte[] b = r.getValue(CF, ATTR); // returns current version of value
+List<KeyValue> kv = r.getColumn(CF, ATTR); // returns all versions of this column
+</programlisting>
+ </section>
+
+ <section>
+ <title>Put</title>
+
+ <para>Doing a put always creates a new version of a <literal>cell</literal>, at a certain
+ timestamp. By default the system uses the server's <literal>currentTimeMillis</literal>,
+ but you can specify the version (= the long integer) yourself, on a per-column level.
+ This means you could assign a time in the past or the future, or use the long value for
+ non-time purposes.</para>
+
+ <para>To overwrite an existing value, do a put at exactly the same row, column, and
+ version as that of the cell you would overshadow.</para>
+ <section
+ xml:id="implicit_version_example">
+ <title>Implicit Version Example</title>
+ <para>The following Put will be implicitly versioned by HBase with the current
+ time.</para>
+ <programlisting language="java">
+public static final byte[] CF = "cf".getBytes();
+public static final byte[] ATTR = "attr".getBytes();
+...
+Put put = new Put(Bytes.toBytes(row));
+put.add(CF, ATTR, Bytes.toBytes( data));
+table.put(put);
+</programlisting>
+ </section>
+ <section
+ xml:id="explicit_version_example">
+ <title>Explicit Version Example</title>
+ <para>The following Put has the version timestamp explicitly set.</para>
+ <programlisting language="java">
+public static final byte[] CF = "cf".getBytes();
+public static final byte[] ATTR = "attr".getBytes();
+...
+Put put = new Put( Bytes.toBytes(row));
+long explicitTimeInMs = 555; // just an example
+put.add(CF, ATTR, explicitTimeInMs, Bytes.toBytes(data));
+table.put(put);
+</programlisting>
+ <para>Caution: the version timestamp is internally by HBase for things like time-to-live
+ calculations. It's usually best to avoid setting this timestamp yourself. Prefer using
+ a separate timestamp attribute of the row, or have the timestamp a part of the rowkey,
+ or both. </para>
+ </section>
+
+ </section>
+
+ <section
+ xml:id="version.delete">
+ <title>Delete</title>
+
+ <para>There are three different types of internal delete markers. See Lars Hofhansl's blog
+ for discussion of his attempt adding another, <link
+ xlink:href="http://hadoop-hbase.blogspot.com/2012/01/scanning-in-hbase.html">Scanning
+ in HBase: Prefix Delete Marker</link>. </para>
+ <itemizedlist>
+ <listitem>
+ <para>Delete: for a specific version of a column.</para>
+ </listitem>
+ <listitem>
+ <para>Delete column: for all versions of a column.</para>
+ </listitem>
+ <listitem>
+ <para>Delete family: for all columns of a particular ColumnFamily</para>
+ </listitem>
+ </itemizedlist>
+ <para>When deleting an entire row, HBase will internally create a tombstone for each
+ ColumnFamily (i.e., not each individual column). </para>
+ <para>Deletes work by creating <emphasis>tombstone</emphasis> markers. For example, let's
+ suppose we want to delete a row. For this you can specify a version, or else by default
+ the <literal>currentTimeMillis</literal> is used. What this means is <quote>delete all
+ cells where the version is less than or equal to this version</quote>. HBase never
+ modifies data in place, so for example a delete will not immediately delete (or mark as
+ deleted) the entries in the storage file that correspond to the delete condition.
+ Rather, a so-called <emphasis>tombstone</emphasis> is written, which will mask the
+ deleted values. When HBase does a major compaction, the tombstones are processed to
+ actually remove the dead values, together with the tombstones themselves. If the version
+ you specified when deleting a row is larger than the version of any value in the row,
+ then you can consider the complete row to be deleted.</para>
+ <para>For an informative discussion on how deletes and versioning interact, see the thread <link
+ xlink:href="http://comments.gmane.org/gmane.comp.java.hadoop.hbase.user/28421">Put w/
+ timestamp -> Deleteall -> Put w/ timestamp fails</link> up on the user mailing
+ list.</para>
+ <para>Also see <xref
+ linkend="keyvalue" /> for more information on the internal KeyValue format. </para>
+ <para>Delete markers are purged during the next major compaction of the store, unless the
+ <option>KEEP_DELETED_CELLS</option> option is set in the column family. To keep the
+ deletes for a configurable amount of time, you can set the delete TTL via the
+ <option>hbase.hstore.time.to.purge.deletes</option> property in
+ <filename>hbase-site.xml</filename>. If
+ <option>hbase.hstore.time.to.purge.deletes</option> is not set, or set to 0, all
+ delete markers, including those with timestamps in the future, are purged during the
+ next major compaction. Otherwise, a delete marker with a timestamp in the future is kept
+ until the major compaction which occurs after the time represented by the marker's
+ timestamp plus the value of <option>hbase.hstore.time.to.purge.deletes</option>, in
+ milliseconds. </para>
+ <note>
+ <para>This behavior represents a fix for an unexpected change that was introduced in
+ HBase 0.94, and was fixed in <link
+ xlink:href="https://issues.apache.org/jira/browse/HBASE-10118">HBASE-10118</link>.
+ The change has been backported to HBase 0.94 and newer branches.</para>
+ </note>
+ </section>
+ </section>
+
+ <section>
+ <title>Current Limitations</title>
+
+ <section>
+ <title>Deletes mask Puts</title>
+
+ <para>Deletes mask puts, even puts that happened after the delete
+ was entered. See <link xlink:href="https://issues.apache.org/jira/browse/HBASE-2256"
+ >HBASE-2256</link>. Remember that a delete writes a tombstone, which only
+ disappears after then next major compaction has run. Suppose you do
+ a delete of everything <= T. After this you do a new put with a
+ timestamp <= T. This put, even if it happened after the delete,
+ will be masked by the delete tombstone. Performing the put will not
+ fail, but when you do a get you will notice the put did have no
+ effect. It will start working again after the major compaction has
+ run. These issues should not be a problem if you use
+ always-increasing versions for new puts to a row. But they can occur
+ even if you do not care about time: just do delete and put
+ immediately after each other, and there is some chance they happen
+ within the same millisecond.</para>
+ </section>
+
+ <section
+ xml:id="major.compactions.change.query.results">
+ <title>Major compactions change query results</title>
+
+ <para><quote>...create three cell versions at t1, t2 and t3, with a maximum-versions
+ setting of 2. So when getting all versions, only the values at t2 and t3 will be
+ returned. But if you delete the version at t2 or t3, the one at t1 will appear again.
+ Obviously, once a major compaction has run, such behavior will not be the case
+ anymore...</quote> (See <emphasis>Garbage Collection</emphasis> in <link
+ xlink:href="http://outerthought.org/blog/417-ot.html">Bending time in
+ HBase</link>.)</para>
+ </section>
+ </section>
+ </section>
+ <section xml:id="dm.sort">
+ <title>Sort Order</title>
+ <para>All data model operations HBase return data in sorted order. First by row,
+ then by ColumnFamily, followed by column qualifier, and finally timestamp (sorted
+ in reverse, so newest records are returned first).
+ </para>
+ </section>
+ <section xml:id="dm.column.metadata">
+ <title>Column Metadata</title>
+ <para>There is no store of column metadata outside of the internal KeyValue instances for a ColumnFamily.
+ Thus, while HBase can support not only a wide number of columns per row, but a heterogenous set of columns
+ between rows as well, it is your responsibility to keep track of the column names.
+ </para>
+ <para>The only way to get a complete set of columns that exist for a ColumnFamily is to process all the rows.
+ For more information about how HBase stores data internally, see <xref linkend="keyvalue" />.
+ </para>
+ </section>
+ <section xml:id="joins"><title>Joins</title>
+ <para>Whether HBase supports joins is a common question on the dist-list, and there is a simple answer: it doesn't,
+ at not least in the way that RDBMS' support them (e.g., with equi-joins or outer-joins in SQL). As has been illustrated
+ in this chapter, the read data model operations in HBase are Get and Scan.
+ </para>
+ <para>However, that doesn't mean that equivalent join functionality can't be supported in your application, but
+ you have to do it yourself. The two primary strategies are either denormalizing the data upon writing to HBase,
+ or to have lookup tables and do the join between HBase tables in your application or MapReduce code (and as RDBMS'
+ demonstrate, there are several strategies for this depending on the size of the tables, e.g., nested loops vs.
+ hash-joins). So which is the best approach? It depends on what you are trying to do, and as such there isn't a single
+ answer that works for every use case.
+ </para>
+ </section>
+ <section xml:id="acid"><title>ACID</title>
+ <para>See <link xlink:href="http://hbase.apache.org/acid-semantics.html">ACID Semantics</link>.
+ Lars Hofhansl has also written a note on
+ <link xlink:href="http://hadoop-hbase.blogspot.com/2012/03/acid-in-hbase.html">ACID in HBase</link>.</para>
+ </section>
+ </chapter>