You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hbase.apache.org by st...@apache.org on 2013/04/06 08:08:58 UTC
svn commit: r1465200 [3/21] - in /hbase/hbase.apache.org/trunk: ./ book/
case_studies/ community/ configuration/ css/ developer/ getting_started/
images/ ops_mgt/ performance/ rpc/ schema_design/ security/ shell/
troubleshooting/ upgrading/
Modified: hbase/hbase.apache.org/trunk/book/apbs02.html
URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/apbs02.html?rev=1465200&r1=1465199&r2=1465200&view=diff
==============================================================================
--- hbase/hbase.apache.org/trunk/book/apbs02.html (original)
+++ hbase/hbase.apache.org/trunk/book/apbs02.html Sat Apr 6 06:08:56 2013
@@ -1,6 +1,6 @@
<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
- <title>B.2. Inconsistencies</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="hbck.in.depth.html" title="Appendix B. hbck In Depth"><link rel="prev" href="hbck.in.depth.html" title="Appendix B. hbck In Depth"><link rel="next" href="apbs03.html" title="B.3. Localized repairs"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">B.2. Inconsistencies</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="hbck.in.depth.html">Prev</a> </td><th width="60%" align="center">Appendix B. hbck In Depth</th><td width="20%" align="right"> <a accesskey="n" href="apbs03.html">Next</a></td
></tr></table><hr></div><div class="section" title="B.2. Inconsistencies"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="d2475e11980"></a>B.2. Inconsistencies</h2></div></div></div><p>
+ <title>B.2. Inconsistencies</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="hbck.in.depth.html" title="Appendix B. hbck In Depth"><link rel="prev" href="hbck.in.depth.html" title="Appendix B. hbck In Depth"><link rel="next" href="apbs03.html" title="B.3. Localized repairs"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">B.2. Inconsistencies</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="hbck.in.depth.html">Prev</a> </td><th width="60%" align="center">Appendix B. hbck In Depth</th><td width="20%" align="right"> <a accesskey="n" href="apbs03.html">Next</a></td
></tr></table><hr></div><div class="section" title="B.2. Inconsistencies"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="d2519e12186"></a>B.2. Inconsistencies</h2></div></div></div><p>
If after several runs, inconsistencies continue to be reported, you may have encountered a
corruption. These should be rare, but in the event they occur newer versions of HBase include
the hbck tool enabled with automatic repair options.
Modified: hbase/hbase.apache.org/trunk/book/apbs03.html
URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/apbs03.html?rev=1465200&r1=1465199&r2=1465200&view=diff
==============================================================================
--- hbase/hbase.apache.org/trunk/book/apbs03.html (original)
+++ hbase/hbase.apache.org/trunk/book/apbs03.html Sat Apr 6 06:08:56 2013
@@ -1,6 +1,6 @@
<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
- <title>B.3. Localized repairs</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="hbck.in.depth.html" title="Appendix B. hbck In Depth"><link rel="prev" href="apbs02.html" title="B.2. Inconsistencies"><link rel="next" href="apbs04.html" title="B.4. Region Overlap Repairs"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">B.3. Localized repairs</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="apbs02.html">Prev</a> </td><th width="60%" align="center">Appendix B. hbck In Depth</th><td width="20%" align="right"> <a accesskey="n" href="apbs04.html">Next</a></td></tr></table><
hr></div><div class="section" title="B.3. Localized repairs"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="d2475e11999"></a>B.3. Localized repairs</h2></div></div></div><p>
+ <title>B.3. Localized repairs</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="hbck.in.depth.html" title="Appendix B. hbck In Depth"><link rel="prev" href="apbs02.html" title="B.2. Inconsistencies"><link rel="next" href="apbs04.html" title="B.4. Region Overlap Repairs"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">B.3. Localized repairs</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="apbs02.html">Prev</a> </td><th width="60%" align="center">Appendix B. hbck In Depth</th><td width="20%" align="right"> <a accesskey="n" href="apbs04.html">Next</a></td></tr></table><
hr></div><div class="section" title="B.3. Localized repairs"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="d2519e12205"></a>B.3. Localized repairs</h2></div></div></div><p>
When repairing a corrupted HBase, it is best to repair the lowest risk inconsistencies first.
These are generally region consistency repairs -- localized single region repairs, that only modify
in-memory data, ephemeral zookeeper data, or patch holes in the META table.
Modified: hbase/hbase.apache.org/trunk/book/apbs04.html
URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/apbs04.html?rev=1465200&r1=1465199&r2=1465200&view=diff
==============================================================================
--- hbase/hbase.apache.org/trunk/book/apbs04.html (original)
+++ hbase/hbase.apache.org/trunk/book/apbs04.html Sat Apr 6 06:08:56 2013
@@ -1,6 +1,6 @@
<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
- <title>B.4. Region Overlap Repairs</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="hbck.in.depth.html" title="Appendix B. hbck In Depth"><link rel="prev" href="apbs03.html" title="B.3. Localized repairs"><link rel="next" href="compression.html" title="Appendix C. Compression In HBase"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">B.4. Region Overlap Repairs</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="apbs03.html">Prev</a> </td><th width="60%" align="center">Appendix B. hbck In Depth</th><td width="20%" align="right"> <a accesskey="n" href="compression.h
tml">Next</a></td></tr></table><hr></div><div class="section" title="B.4. Region Overlap Repairs"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="d2475e12037"></a>B.4. Region Overlap Repairs</h2></div></div></div>
+ <title>B.4. Region Overlap Repairs</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="hbck.in.depth.html" title="Appendix B. hbck In Depth"><link rel="prev" href="apbs03.html" title="B.3. Localized repairs"><link rel="next" href="compression.html" title="Appendix C. Compression In HBase"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">B.4. Region Overlap Repairs</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="apbs03.html">Prev</a> </td><th width="60%" align="center">Appendix B. hbck In Depth</th><td width="20%" align="right"> <a accesskey="n" href="compression.h
tml">Next</a></td></tr></table><hr></div><div class="section" title="B.4. Region Overlap Repairs"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="d2519e12243"></a>B.4. Region Overlap Repairs</h2></div></div></div>
Table integrity problems can require repairs that deal with overlaps. This is a riskier operation
because it requires modifications to the file system, requires some decision making, and may
require some manual steps. For these repairs it is best to analyze the output of a <code class="code">hbck -details</code>
@@ -39,17 +39,17 @@ Finally, there are safeguards to limit r
command would only attempt to check and repair table TableFoo and TableBar.
<pre class="programlisting">
$ ./bin/hbase hbck -repair TableFoo TableBar
-</pre><div class="section" title="B.4.1. Special cases: Meta is not properly assigned"><div class="titlepage"><div><div><h3 class="title"><a name="d2475e12076"></a>B.4.1. Special cases: Meta is not properly assigned</h3></div></div></div>
+</pre><div class="section" title="B.4.1. Special cases: Meta is not properly assigned"><div class="titlepage"><div><div><h3 class="title"><a name="d2519e12282"></a>B.4.1. Special cases: Meta is not properly assigned</h3></div></div></div>
There are a few special cases that hbck can handle as well.
Sometimes the meta table’s only region is inconsistently assigned or deployed. In this case
there is a special <code class="code">-fixMetaOnly</code> option that can try to fix meta assignments.
<pre class="programlisting">
$ ./bin/hbase hbck -fixMetaOnly -fixAssignments
-</pre></div><div class="section" title="B.4.2. Special cases: HBase version file is missing"><div class="titlepage"><div><div><h3 class="title"><a name="d2475e12085"></a>B.4.2. Special cases: HBase version file is missing</h3></div></div></div>
+</pre></div><div class="section" title="B.4.2. Special cases: HBase version file is missing"><div class="titlepage"><div><div><h3 class="title"><a name="d2519e12291"></a>B.4.2. Special cases: HBase version file is missing</h3></div></div></div>
HBase’s data on the file system requires a version file in order to start. If this flie is missing, you
can use the <code class="code">-fixVersionFile</code> option to fabricating a new HBase version file. This assumes that
the version of hbck you are running is the appropriate version for the HBase cluster.
- </div><div class="section" title="B.4.3. Special case: Root and META are corrupt."><div class="titlepage"><div><div><h3 class="title"><a name="d2475e12092"></a>B.4.3. Special case: Root and META are corrupt.</h3></div></div></div>
+ </div><div class="section" title="B.4.3. Special case: Root and META are corrupt."><div class="titlepage"><div><div><h3 class="title"><a name="d2519e12298"></a>B.4.3. Special case: Root and META are corrupt.</h3></div></div></div>
The most drastic corruption scenario is the case where the ROOT or META is corrupted and
HBase will not start. In this case you can use the OfflineMetaRepair tool create new ROOT
and META regions and tables.
@@ -64,7 +64,7 @@ $ ./bin/hbase org.apache.hadoop.hbase.ut
NOTE: This tool is not as clever as uberhbck but can be used to bootstrap repairs that uberhbck
can complete.
If the tool succeeds you should be able to start hbase and run online repairs if necessary.
- </div><div class="section" title="B.4.4. Special cases: Offline split parent"><div class="titlepage"><div><div><h3 class="title"><a name="d2475e12099"></a>B.4.4. Special cases: Offline split parent</h3></div></div></div><p>
+ </div><div class="section" title="B.4.4. Special cases: Offline split parent"><div class="titlepage"><div><div><h3 class="title"><a name="d2519e12305"></a>B.4.4. Special cases: Offline split parent</h3></div></div></div><p>
Once a region is split, the offline parent will be cleaned up automatically. Sometimes, daughter regions
are split again before their parents are cleaned up. HBase can clean up parents in the right order. However,
there could be some lingering offline split parents sometimes. They are in META, in HDFS, and not deployed.
Modified: hbase/hbase.apache.org/trunk/book/apd.html
URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/apd.html?rev=1465200&r1=1465199&r2=1465200&view=diff
==============================================================================
--- hbase/hbase.apache.org/trunk/book/apd.html (original)
+++ hbase/hbase.apache.org/trunk/book/apd.html Sat Apr 6 06:08:56 2013
@@ -1,6 +1,6 @@
<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
- <title>Appendix D. YCSB: The Yahoo! Cloud Serving Benchmark and HBase</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="prev" href="changing.compression.html" title="C.6. Changing Compression Schemes"><link rel="next" href="hfilev2.html" title="Appendix E. HFile format version 2"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">Appendix D. YCSB: The Yahoo! Cloud Serving Benchmark and HBase</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="changing.compression.html">Prev</a> </td><th width="60%" align="center">&
nbsp;</th><td width="20%" align="right"> <a accesskey="n" href="hfilev2.html">Next</a></td></tr></table><hr></div><div class="appendix" title="Appendix D. YCSB: The Yahoo! Cloud Serving Benchmark and HBase"><div class="titlepage"><div><div><h2 class="title"><a name="d2475e12243"></a>Appendix D. <a class="link" href="https://github.com/brianfrankcooper/YCSB/" target="_top">YCSB: The Yahoo! Cloud Serving Benchmark</a> and HBase</h2></div></div></div><p>TODO: Describe how YCSB is poor for putting up a decent cluster load.</p><p>TODO: Describe setup of YCSB for HBase</p><p>Ted Dunning redid YCSB so it's mavenized and added facility for verifying workloads. See <a class="link" href="https://github.com/tdunning/YCSB" target="_top">Ted Dunning's YCSB</a>.</p></div><div id="disqus_thread"></div><script type="text/javascript">
+ <title>Appendix D. YCSB: The Yahoo! Cloud Serving Benchmark and HBase</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="prev" href="changing.compression.html" title="C.6. Changing Compression Schemes"><link rel="next" href="hfilev2.html" title="Appendix E. HFile format version 2"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">Appendix D. YCSB: The Yahoo! Cloud Serving Benchmark and HBase</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="changing.compression.html">Prev</a> </td><th width="60%" align="center">&
nbsp;</th><td width="20%" align="right"> <a accesskey="n" href="hfilev2.html">Next</a></td></tr></table><hr></div><div class="appendix" title="Appendix D. YCSB: The Yahoo! Cloud Serving Benchmark and HBase"><div class="titlepage"><div><div><h2 class="title"><a name="d2519e12449"></a>Appendix D. <a class="link" href="https://github.com/brianfrankcooper/YCSB/" target="_top">YCSB: The Yahoo! Cloud Serving Benchmark</a> and HBase</h2></div></div></div><p>TODO: Describe how YCSB is poor for putting up a decent cluster load.</p><p>TODO: Describe setup of YCSB for HBase</p><p>Ted Dunning redid YCSB so it's mavenized and added facility for verifying workloads. See <a class="link" href="https://github.com/tdunning/YCSB" target="_top">Ted Dunning's YCSB</a>.</p></div><div id="disqus_thread"></div><script type="text/javascript">
var disqus_shortname = 'hbase'; // required: replace example with your forum shortname
var disqus_url = 'http://hbase.apache.org/book';
var disqus_identifier = '';
Modified: hbase/hbase.apache.org/trunk/book/apes02.html
URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/apes02.html?rev=1465200&r1=1465199&r2=1465200&view=diff
==============================================================================
--- hbase/hbase.apache.org/trunk/book/apes02.html (original)
+++ hbase/hbase.apache.org/trunk/book/apes02.html Sat Apr 6 06:08:56 2013
@@ -1,12 +1,12 @@
<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
- <title>E.2. HFile format version 1 overview</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="hfilev2.html" title="Appendix E. HFile format version 2"><link rel="prev" href="hfilev2.html" title="Appendix E. HFile format version 2"><link rel="next" href="apes03.html" title="E.3. HBase file format with inline blocks (version 2)"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">E.2. HFile format version 1 overview </th></tr><tr><td width="20%" align="left"><a accesskey="p" href="hfilev2.html">Prev</a> </td><th width="60%" align="center">Appendix E. HFile format version 2</th><td width=
"20%" align="right"> <a accesskey="n" href="apes03.html">Next</a></td></tr></table><hr></div><div class="section" title="E.2. HFile format version 1 overview"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="d2475e12273"></a>E.2. HFile format version 1 overview </h2></div></div></div><p>As we will be discussing the changes we are making to the HFile format, it is useful to give a short overview of the previous (HFile version 1) format. An HFile in the existing format is structured as follows:
+ <title>E.2. HFile format version 1 overview</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="hfilev2.html" title="Appendix E. HFile format version 2"><link rel="prev" href="hfilev2.html" title="Appendix E. HFile format version 2"><link rel="next" href="apes03.html" title="E.3. HBase file format with inline blocks (version 2)"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">E.2. HFile format version 1 overview </th></tr><tr><td width="20%" align="left"><a accesskey="p" href="hfilev2.html">Prev</a> </td><th width="60%" align="center">Appendix E. HFile format version 2</th><td width=
"20%" align="right"> <a accesskey="n" href="apes03.html">Next</a></td></tr></table><hr></div><div class="section" title="E.2. HFile format version 1 overview"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="d2519e12479"></a>E.2. HFile format version 1 overview </h2></div></div></div><p>As we will be discussing the changes we are making to the HFile format, it is useful to give a short overview of the previous (HFile version 1) format. An HFile in the existing format is structured as follows:
<span class="inlinemediaobject"><img src="../images/hfile.png" align="middle" alt="HFile Version 1"></span>
- <sup>[<a name="d2475e12288" href="#ftn.d2475e12288" class="footnote">33</a>]</sup>
- </p><div class="section" title="E.2.1. Block index format in version 1"><div class="titlepage"><div><div><h3 class="title"><a name="d2475e12295"></a>E.2.1. Block index format in version 1 </h3></div></div></div><p>The block index in version 1 is very straightforward. For each entry, it contains: </p><div class="orderedlist"><ol class="orderedlist" type="1"><li class="listitem"><p>Offset (long)</p></li><li class="listitem"><p>Uncompressed size (int)</p></li><li class="listitem"><p>Key (a serialized byte array written using Bytes.writeByteArray) </p><div class="orderedlist"><ol class="orderedlist" type="a"><li class="listitem"><p>Key length as a variable-length integer (VInt)
+ <sup>[<a name="d2519e12494" href="#ftn.d2519e12494" class="footnote">34</a>]</sup>
+ </p><div class="section" title="E.2.1. Block index format in version 1"><div class="titlepage"><div><div><h3 class="title"><a name="d2519e12501"></a>E.2.1. Block index format in version 1 </h3></div></div></div><p>The block index in version 1 is very straightforward. For each entry, it contains: </p><div class="orderedlist"><ol class="orderedlist" type="1"><li class="listitem"><p>Offset (long)</p></li><li class="listitem"><p>Uncompressed size (int)</p></li><li class="listitem"><p>Key (a serialized byte array written using Bytes.writeByteArray) </p><div class="orderedlist"><ol class="orderedlist" type="a"><li class="listitem"><p>Key length as a variable-length integer (VInt)
</p></li><li class="listitem"><p>
Key bytes
- </p></li></ol></div></li></ol></div><p>The number of entries in the block index is stored in the fixed file trailer, and has to be passed in to the method that reads the block index. One of the limitations of the block index in version 1 is that it does not provide the compressed size of a block, which turns out to be necessary for decompression. Therefore, the HFile reader has to infer this compressed size from the offset difference between blocks. We fix this limitation in version 2, where we store on-disk block size instead of uncompressed size, and get uncompressed size from the block header.</p></div><div class="footnotes"><br><hr width="100" align="left"><div class="footnote"><p><sup>[<a id="ftn.d2475e12288" href="#d2475e12288" class="para">33</a>] </sup>Image courtesy of Lars George, <a class="link" href="http://www.larsgeorge.com/2009/10/hbase-architecture-101-storage.html" target="_top">hbase-architecture-101-storage.html</a>.</p></div></div></div><
div id="disqus_thread"></div><script type="text/javascript">
+ </p></li></ol></div></li></ol></div><p>The number of entries in the block index is stored in the fixed file trailer, and has to be passed in to the method that reads the block index. One of the limitations of the block index in version 1 is that it does not provide the compressed size of a block, which turns out to be necessary for decompression. Therefore, the HFile reader has to infer this compressed size from the offset difference between blocks. We fix this limitation in version 2, where we store on-disk block size instead of uncompressed size, and get uncompressed size from the block header.</p></div><div class="footnotes"><br><hr width="100" align="left"><div class="footnote"><p><sup>[<a id="ftn.d2519e12494" href="#d2519e12494" class="para">34</a>] </sup>Image courtesy of Lars George, <a class="link" href="http://www.larsgeorge.com/2009/10/hbase-architecture-101-storage.html" target="_top">hbase-architecture-101-storage.html</a>.</p></div></div></div><
div id="disqus_thread"></div><script type="text/javascript">
var disqus_shortname = 'hbase'; // required: replace example with your forum shortname
var disqus_url = 'http://hbase.apache.org/book';
var disqus_identifier = '';
Modified: hbase/hbase.apache.org/trunk/book/apes03.html
URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/apes03.html?rev=1465200&r1=1465199&r2=1465200&view=diff
==============================================================================
--- hbase/hbase.apache.org/trunk/book/apes03.html (original)
+++ hbase/hbase.apache.org/trunk/book/apes03.html Sat Apr 6 06:08:56 2013
@@ -2,12 +2,12 @@
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>E.3. HBase file format with inline blocks (version 2)</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="hfilev2.html" title="Appendix E. HFile format version 2"><link rel="prev" href="apes02.html" title="E.2. HFile format version 1 overview"><link rel="next" href="other.info.html" title="Appendix F. Other Information About HBase"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">E.3.
HBase file format with inline blocks (version 2)
- </th></tr><tr><td width="20%" align="left"><a accesskey="p" href="apes02.html">Prev</a> </td><th width="60%" align="center">Appendix E. HFile format version 2</th><td width="20%" align="right"> <a accesskey="n" href="other.info.html">Next</a></td></tr></table><hr></div><div class="section" title="E.3. HBase file format with inline blocks (version 2)"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="d2475e12319"></a>E.3.
+ </th></tr><tr><td width="20%" align="left"><a accesskey="p" href="apes02.html">Prev</a> </td><th width="60%" align="center">Appendix E. HFile format version 2</th><td width="20%" align="right"> <a accesskey="n" href="other.info.html">Next</a></td></tr></table><hr></div><div class="section" title="E.3. HBase file format with inline blocks (version 2)"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="d2519e12525"></a>E.3.
HBase file format with inline blocks (version 2)
- </h2></div></div></div><div class="section" title="E.3.1. Overview"><div class="titlepage"><div><div><h3 class="title"><a name="d2475e12322"></a>E.3.1. Overview</h3></div></div></div><p>The version of HBase introducing the above features reads both version 1 and 2 HFiles, but only writes version 2 HFiles. A version 2 HFile is structured as follows:
+ </h2></div></div></div><div class="section" title="E.3.1. Overview"><div class="titlepage"><div><div><h3 class="title"><a name="d2519e12528"></a>E.3.1. Overview</h3></div></div></div><p>The version of HBase introducing the above features reads both version 1 and 2 HFiles, but only writes version 2 HFiles. A version 2 HFile is structured as follows:
<span class="inlinemediaobject"><img src="../images/hfilev2.png" align="middle" alt="HFile Version 2"></span>
- </p></div><div class="section" title="E.3.2. Unified version 2 block format"><div class="titlepage"><div><div><h3 class="title"><a name="d2475e12337"></a>E.3.2. Unified version 2 block format</h3></div></div></div><p>In the version 2 every block in the data section contains the following fields: </p><div class="orderedlist"><ol class="orderedlist" type="1"><li class="listitem"><p>8 bytes: Block type, a sequence of bytes equivalent to version 1's "magic records". Supported block types are: </p><div class="orderedlist"><ol class="orderedlist" type="a"><li class="listitem"><p>DATA – data blocks
+ </p></div><div class="section" title="E.3.2. Unified version 2 block format"><div class="titlepage"><div><div><h3 class="title"><a name="d2519e12543"></a>E.3.2. Unified version 2 block format</h3></div></div></div><p>In the version 2 every block in the data section contains the following fields: </p><div class="orderedlist"><ol class="orderedlist" type="1"><li class="listitem"><p>8 bytes: Block type, a sequence of bytes equivalent to version 1's "magic records". Supported block types are: </p><div class="orderedlist"><ol class="orderedlist" type="a"><li class="listitem"><p>DATA – data blocks
</p></li><li class="listitem"><p>
LEAF_INDEX – leaf-level index blocks in a multi-level-block-index
</p></li><li class="listitem"><p>
@@ -35,25 +35,25 @@ Can be used for skipping the current dat
</p></li><li class="listitem"><p>File offset of the previous block of the same type (long)</p><p>
Can be used for seeking to the previous data/index block
</p></li><li class="listitem"><p>Compressed data (or uncompressed data if the compression algorithm is NONE).</p></li></ol></div><p>The above format of blocks is used in the following HFile sections:</p><div class="orderedlist"><ol class="orderedlist" type="1"><li class="listitem"><p>Scanned block section. The section is named so because it contains all data blocks that need to be read when an HFile is scanned sequentially. Also contains leaf block index and Bloom chunk blocks. </p></li><li class="listitem"><p>Non-scanned block section. This section still contains unified-format v2 blocks but it does not have to be read when doing a sequential scan. This section contains “meta” blocks and intermediate-level index blocks.
- </p></li></ol></div><p>We are supporting “meta” blocks in version 2 the same way they were supported in version 1, even though we do not store Bloom filter data in these blocks anymore. </p></div><div class="section" title="E.3.3. Block index in version 2"><div class="titlepage"><div><div><h3 class="title"><a name="d2475e12406"></a>E.3.3. Block index in version 2</h3></div></div></div><p>There are three types of block indexes in HFile version 2, stored in two different formats (root and non-root): </p><div class="orderedlist"><ol class="orderedlist" type="1"><li class="listitem"><p>Data index — version 2 multi-level block index, consisting of:</p><div class="orderedlist"><ol class="orderedlist" type="a"><li class="listitem"><p>
+ </p></li></ol></div><p>We are supporting “meta” blocks in version 2 the same way they were supported in version 1, even though we do not store Bloom filter data in these blocks anymore. </p></div><div class="section" title="E.3.3. Block index in version 2"><div class="titlepage"><div><div><h3 class="title"><a name="d2519e12612"></a>E.3.3. Block index in version 2</h3></div></div></div><p>There are three types of block indexes in HFile version 2, stored in two different formats (root and non-root): </p><div class="orderedlist"><ol class="orderedlist" type="1"><li class="listitem"><p>Data index — version 2 multi-level block index, consisting of:</p><div class="orderedlist"><ol class="orderedlist" type="a"><li class="listitem"><p>
Version 2 root index, stored in the data block index section of the file
</p></li><li class="listitem"><p>
Optionally, version 2 intermediate levels, stored in the non%root format in the data index section of the file. Intermediate levels can only be present if leaf level blocks are present
</p></li><li class="listitem"><p>
Optionally, version 2 leaf levels, stored in the non%root format inline with data blocks
- </p></li></ol></div></li><li class="listitem"><p>Meta index — version 2 root index format only, stored in the meta index section of the file</p></li><li class="listitem"><p>Bloom index — version 2 root index format only, stored in the “load-on-open” section as part of Bloom filter metadata.</p></li></ol></div></div><div class="section" title="E.3.4. Root block index format in version 2"><div class="titlepage"><div><div><h3 class="title"><a name="d2475e12431"></a>E.3.4.
+ </p></li></ol></div></li><li class="listitem"><p>Meta index — version 2 root index format only, stored in the meta index section of the file</p></li><li class="listitem"><p>Bloom index — version 2 root index format only, stored in the “load-on-open” section as part of Bloom filter metadata.</p></li></ol></div></div><div class="section" title="E.3.4. Root block index format in version 2"><div class="titlepage"><div><div><h3 class="title"><a name="d2519e12637"></a>E.3.4.
Root block index format in version 2</h3></div></div></div><p>This format applies to:</p><div class="orderedlist"><ol class="orderedlist" type="1"><li class="listitem"><p>Root level of the version 2 data index</p></li><li class="listitem"><p>Entire meta and Bloom indexes in version 2, which are always single-level. </p></li></ol></div><p>A version 2 root index block is a sequence of entries of the following format, similar to entries of a version 1 block index, but storing on-disk size instead of uncompressed size. </p><div class="orderedlist"><ol class="orderedlist" type="1"><li class="listitem"><p>Offset (long) </p><p>
This offset may point to a data block or to a deeper>level index block.
</p></li><li class="listitem"><p>On-disk size (int) </p></li><li class="listitem"><p>Key (a serialized byte array stored using Bytes.writeByteArray) </p><div class="orderedlist"><ol class="orderedlist" type="a"><li class="listitem"><p>Key (VInt)
</p></li><li class="listitem"><p>Key bytes
- </p></li></ol></div></li></ol></div><p>A single-level version 2 block index consists of just a single root index block. To read a root index block of version 2, one needs to know the number of entries. For the data index and the meta index the number of entries is stored in the trailer, and for the Bloom index it is stored in the compound Bloom filter metadata.</p><p>For a multi-level block index we also store the following fields in the root index block in the load-on-open section of the HFile, in addition to the data structure described above:</p><div class="orderedlist"><ol class="orderedlist" type="1"><li class="listitem"><p>Middle leaf index block offset</p></li><li class="listitem"><p>Middle leaf block on-disk size (meaning the leaf index block containing the reference to the “middle” data block of the file) </p></li><li class="listitem"><p>The index of the mid-key (defined below) in the middle leaf-level block.</p></li></ol></div><p></p><p>The
se additional fields are used to efficiently retrieve the mid-key of the HFile used in HFile splits, which we define as the first key of the block with a zero-based index of (n – 1) / 2, if the total number of blocks in the HFile is n. This definition is consistent with how the mid-key was determined in HFile version 1, and is reasonable in general, because blocks are likely to be the same size on average, but we don’t have any estimates on individual key/value pair sizes. </p><p></p><p>When writing a version 2 HFile, the total number of data blocks pointed to by every leaf-level index block is kept track of. When we finish writing and the total number of leaf-level blocks is determined, it is clear which leaf-level block contains the mid-key, and the fields listed above are computed. When reading the HFile and the mid-key is requested, we retrieve the middle leaf index block (potentially from the block cache) and get the mid-key value from the appropriate
position inside that leaf block.</p></div><div class="section" title="E.3.5. Non-root block index format in version 2"><div class="titlepage"><div><div><h3 class="title"><a name="d2475e12484"></a>E.3.5.
+ </p></li></ol></div></li></ol></div><p>A single-level version 2 block index consists of just a single root index block. To read a root index block of version 2, one needs to know the number of entries. For the data index and the meta index the number of entries is stored in the trailer, and for the Bloom index it is stored in the compound Bloom filter metadata.</p><p>For a multi-level block index we also store the following fields in the root index block in the load-on-open section of the HFile, in addition to the data structure described above:</p><div class="orderedlist"><ol class="orderedlist" type="1"><li class="listitem"><p>Middle leaf index block offset</p></li><li class="listitem"><p>Middle leaf block on-disk size (meaning the leaf index block containing the reference to the “middle” data block of the file) </p></li><li class="listitem"><p>The index of the mid-key (defined below) in the middle leaf-level block.</p></li></ol></div><p></p><p>The
se additional fields are used to efficiently retrieve the mid-key of the HFile used in HFile splits, which we define as the first key of the block with a zero-based index of (n – 1) / 2, if the total number of blocks in the HFile is n. This definition is consistent with how the mid-key was determined in HFile version 1, and is reasonable in general, because blocks are likely to be the same size on average, but we don’t have any estimates on individual key/value pair sizes. </p><p></p><p>When writing a version 2 HFile, the total number of data blocks pointed to by every leaf-level index block is kept track of. When we finish writing and the total number of leaf-level blocks is determined, it is clear which leaf-level block contains the mid-key, and the fields listed above are computed. When reading the HFile and the mid-key is requested, we retrieve the middle leaf index block (potentially from the block cache) and get the mid-key value from the appropriate
position inside that leaf block.</p></div><div class="section" title="E.3.5. Non-root block index format in version 2"><div class="titlepage"><div><div><h3 class="title"><a name="d2519e12690"></a>E.3.5.
Non-root block index format in version 2</h3></div></div></div><p>This format applies to intermediate-level and leaf index blocks of a version 2 multi-level data block index. Every non-root index block is structured as follows. </p><div class="orderedlist"><ol class="orderedlist" type="1"><li class="listitem"><p>numEntries: the number of entries (int). </p></li><li class="listitem"><p>entryOffsets: the “secondary index” of offsets of entries in the block, to facilitate a quick binary search on the key (numEntries + 1 int values). The last value is the total length of all entries in this index block. For example, in a non-root index block with entry sizes 60, 80, 50 the “secondary index” will contain the following int array: {0, 60, 140, 190}.</p></li><li class="listitem"><p>Entries. Each entry contains: </p><div class="orderedlist"><ol class="orderedlist" type="a"><li class="listitem"><p>
Offset of the block referenced by this entry in the file (long)
</p></li><li class="listitem"><p>
On>disk size of the referenced block (int)
</p></li><li class="listitem"><p>
Key. The length can be calculated from entryOffsets.
- </p></li></ol></div></li></ol></div></div><div class="section" title="E.3.6. Bloom filters in version 2"><div class="titlepage"><div><div><h3 class="title"><a name="d2475e12509"></a>E.3.6.
+ </p></li></ol></div></li></ol></div></div><div class="section" title="E.3.6. Bloom filters in version 2"><div class="titlepage"><div><div><h3 class="title"><a name="d2519e12715"></a>E.3.6.
Bloom filters in version 2</h3></div></div></div><p>In contrast with version 1, in a version 2 HFile Bloom filter metadata is stored in the load-on-open section of the HFile for quick startup. </p><div class="orderedlist"><ol class="orderedlist" type="1"><li class="listitem"><p>A compound Bloom filter. </p><div class="orderedlist"><ol class="orderedlist" type="a"><li class="listitem"><p>
Bloom filter version = 3 (int). There used to be a DynamicByteBloomFilter class that had the Bloom filter version number 2
</p></li><li class="listitem"><p>
@@ -72,7 +72,7 @@ The number of chunks (int)
Comparator class used for Bloom filter keys, a UTF>8 encoded string stored using Bytes.writeByteArray
</p></li><li class="listitem"><p>
Bloom block index in the version 2 root block index format
- </p></li></ol></div></li></ol></div></div><div class="section" title="E.3.7. File Info format in versions 1 and 2"><div class="titlepage"><div><div><h3 class="title"><a name="d2475e12546"></a>E.3.7. File Info format in versions 1 and 2</h3></div></div></div><p>The file info block is a serialized <a class="ulink" href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/io/HbaseMapWritable.html" target="_top">HbaseMapWritable</a> (essentially a map from byte arrays to byte arrays) with the following keys, among others. StoreFile-level logic adds more keys to this.</p><div class="informaltable"><table border="1"><colgroup><col><col></colgroup><tbody><tr><td>
+ </p></li></ol></div></li></ol></div></div><div class="section" title="E.3.7. File Info format in versions 1 and 2"><div class="titlepage"><div><div><h3 class="title"><a name="d2519e12752"></a>E.3.7. File Info format in versions 1 and 2</h3></div></div></div><p>The file info block is a serialized <a class="ulink" href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/io/HbaseMapWritable.html" target="_top">HbaseMapWritable</a> (essentially a map from byte arrays to byte arrays) with the following keys, among others. StoreFile-level logic adds more keys to this.</p><div class="informaltable"><table border="1"><colgroup><col><col></colgroup><tbody><tr><td>
<p>hfile.LASTKEY </p>
</td><td>
<p>The last key of the file (byte array) </p>
@@ -84,7 +84,7 @@ Comparator class used for Bloom filter k
<p>hfile.AVG_VALUE_LEN </p>
</td><td>
<p>The average value length in the file (int) </p>
- </td></tr></tbody></table></div><p>File info format did not change in version 2. However, we moved the file info to the final section of the file, which can be loaded as one block at the time the HFile is being opened. Also, we do not store comparator in the version 2 file info anymore. Instead, we store it in the fixed file trailer. This is because we need to know the comparator at the time of parsing the load-on-open section of the HFile.</p></div><div class="section" title="E.3.8. Fixed file trailer format differences between versions 1 and 2"><div class="titlepage"><div><div><h3 class="title"><a name="d2475e12592"></a>E.3.8.
+ </td></tr></tbody></table></div><p>File info format did not change in version 2. However, we moved the file info to the final section of the file, which can be loaded as one block at the time the HFile is being opened. Also, we do not store comparator in the version 2 file info anymore. Instead, we store it in the fixed file trailer. This is because we need to know the comparator at the time of parsing the load-on-open section of the HFile.</p></div><div class="section" title="E.3.8. Fixed file trailer format differences between versions 1 and 2"><div class="titlepage"><div><div><h3 class="title"><a name="d2519e12798"></a>E.3.8.
Fixed file trailer format differences between versions 1 and 2</h3></div></div></div><p>The following table shows common and different fields between fixed file trailers in versions 1 and 2. Note that the size of the trailer is different depending on the version, so it is “fixed” only within one version. However, the version is always stored as the last four-byte integer in the file. </p><p></p><div class="informaltable"><table border="1"><colgroup><col class="c1"><col class="c2"></colgroup><tbody><tr><td>
<p>Version 1 </p>
</td><td>
Modified: hbase/hbase.apache.org/trunk/book/apjs02.html
URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/apjs02.html?rev=1465200&r1=1465199&r2=1465200&view=diff
==============================================================================
--- hbase/hbase.apache.org/trunk/book/apjs02.html (original)
+++ hbase/hbase.apache.org/trunk/book/apjs02.html Sat Apr 6 06:08:56 2013
@@ -1,6 +1,6 @@
<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
- <title>J.2. TODO</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="hbase.rpc.html" title="Appendix J. 0.95 RPC Specification"><link rel="prev" href="hbase.rpc.html" title="Appendix J. 0.95 RPC Specification"><link rel="next" href="apjs03.html" title="J.3. RPC"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">J.2. TODO</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="hbase.rpc.html">Prev</a> </td><th width="60%" align="center">Appendix J. 0.95 RPC Specification</th><td width="20%" align="right"> <a accesskey="n" href="apjs03.html">Next</a></td></tr></table><hr></d
iv><div class="section" title="J.2. TODO"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="d2475e12978"></a>J.2. TODO</h2></div></div></div><p>
+ <title>J.2. TODO</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="hbase.rpc.html" title="Appendix J. 0.95 RPC Specification"><link rel="prev" href="hbase.rpc.html" title="Appendix J. 0.95 RPC Specification"><link rel="next" href="apjs03.html" title="J.3. RPC"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">J.2. TODO</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="hbase.rpc.html">Prev</a> </td><th width="60%" align="center">Appendix J. 0.95 RPC Specification</th><td width="20%" align="right"> <a accesskey="n" href="apjs03.html">Next</a></td></tr></table><hr></d
iv><div class="section" title="J.2. TODO"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="d2519e13184"></a>J.2. TODO</h2></div></div></div><p>
</p><div class="orderedlist"><ol class="orderedlist" type="1"><li class="listitem"><p>List of problems with currently specified format and where
we would like to go in a version2, etc. For example, what would we
have to change if anything to move server async or to support
Modified: hbase/hbase.apache.org/trunk/book/apjs03.html
URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/apjs03.html?rev=1465200&r1=1465199&r2=1465200&view=diff
==============================================================================
--- hbase/hbase.apache.org/trunk/book/apjs03.html (original)
+++ hbase/hbase.apache.org/trunk/book/apjs03.html Sat Apr 6 06:08:56 2013
@@ -1,42 +1,42 @@
<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
- <title>J.3. RPC</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="hbase.rpc.html" title="Appendix J. 0.95 RPC Specification"><link rel="prev" href="apjs02.html" title="J.2. TODO"><link rel="next" href="apjs04.html" title="J.4. Notes"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">J.3. RPC</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="apjs02.html">Prev</a> </td><th width="60%" align="center">Appendix J. 0.95 RPC Specification</th><td width="20%" align="right"> <a accesskey="n" href="apjs04.html">Next</a></td></tr></table><hr></div><div class="section" title="J.3.&
nbsp;RPC"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="d2475e12994"></a>J.3. RPC</h2></div></div></div><p>The client will send setup information on connection establish.
+ <title>J.3. RPC</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="hbase.rpc.html" title="Appendix J. 0.95 RPC Specification"><link rel="prev" href="apjs02.html" title="J.2. TODO"><link rel="next" href="apjs04.html" title="J.4. Notes"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">J.3. RPC</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="apjs02.html">Prev</a> </td><th width="60%" align="center">Appendix J. 0.95 RPC Specification</th><td width="20%" align="right"> <a accesskey="n" href="apjs04.html">Next</a></td></tr></table><hr></div><div class="section" title="J.3.&
nbsp;RPC"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="d2519e13200"></a>J.3. RPC</h2></div></div></div><p>The client will send setup information on connection establish.
Thereafter, the client invokes methods against the remote server sending a protobuf Message and receiving a protobuf Message in response.
Communication is synchronous. All back and forth is preceded by an int that has the total length of the request/response.
Optionally, Cells(KeyValues) can be passed outside of protobufs in follow-behind Cell blocks (because
<a class="link" href="https://docs.google.com/document/d/1WEtrq-JTIUhlnlnvA0oYRLp0F8MKpEBeBSCFcQiacdw/edit#" target="_top">we can’t protobuf megabytes of KeyValues</a> or Cells).
These CellBlocks are encoded and optionally compressed.</p><p></p><p>For more detail on the protobufs involved, see the
- <a class="link" href="http://svn.apache.org/viewvc/hbase/trunk/hbase-protocol/src/main/protobuf/RPC.proto?view=markup" target="_top">RPC.proto</a> file in trunk.</p><div class="section" title="J.3.1. Connection Setup"><div class="titlepage"><div><div><h3 class="title"><a name="d2475e13008"></a>J.3.1. Connection Setup</h3></div></div></div><p>Client initiates connection.</p><div class="section" title="J.3.1.1. Client"><div class="titlepage"><div><div><h4 class="title"><a name="d2475e13013"></a>J.3.1.1. Client</h4></div></div></div><p>On connection setup, client sends a preamble followed by a connection header.
- </p><div class="section" title="J.3.1.1.1. <preamble>"><div class="titlepage"><div><div><h5 class="title"><a name="d2475e13018"></a>J.3.1.1.1. <preamble></h5></div></div></div><pre class="programlisting"><MAGIC 4 byte integer> <1 byte RPC Format Version> <1 byte auth type><sup>[<a name="d2475e13024" href="#ftn.d2475e13024" class="footnote">34</a>]</sup></pre><p>E.g.: HBas0x000x80 -- 4 bytes of MAGIC -- ‘HBas’ -- plus one-byte of version, 0 in this case, and one byte, 0x80 (SIMPLE). of an auth type.</p></div><div class="section" title="J.3.1.1.2. <Protobuf ConnectionHeader Message>"><div class="titlepage"><div><div><h5 class="title"><a name="d2475e13029"></a>J.3.1.1.2. <Protobuf ConnectionHeader Message></h5></div></div></div><p>Has user info, and “protocol”, as well as the encoders and compression the client will use sending CellBlocks.
+ <a class="link" href="http://svn.apache.org/viewvc/hbase/trunk/hbase-protocol/src/main/protobuf/RPC.proto?view=markup" target="_top">RPC.proto</a> file in trunk.</p><div class="section" title="J.3.1. Connection Setup"><div class="titlepage"><div><div><h3 class="title"><a name="d2519e13214"></a>J.3.1. Connection Setup</h3></div></div></div><p>Client initiates connection.</p><div class="section" title="J.3.1.1. Client"><div class="titlepage"><div><div><h4 class="title"><a name="d2519e13219"></a>J.3.1.1. Client</h4></div></div></div><p>On connection setup, client sends a preamble followed by a connection header.
+ </p><div class="section" title="J.3.1.1.1. <preamble>"><div class="titlepage"><div><div><h5 class="title"><a name="d2519e13224"></a>J.3.1.1.1. <preamble></h5></div></div></div><pre class="programlisting"><MAGIC 4 byte integer> <1 byte RPC Format Version> <1 byte auth type><sup>[<a name="d2519e13230" href="#ftn.d2519e13230" class="footnote">35</a>]</sup></pre><p>E.g.: HBas0x000x80 -- 4 bytes of MAGIC -- ‘HBas’ -- plus one-byte of version, 0 in this case, and one byte, 0x80 (SIMPLE). of an auth type.</p></div><div class="section" title="J.3.1.1.2. <Protobuf ConnectionHeader Message>"><div class="titlepage"><div><div><h5 class="title"><a name="d2519e13235"></a>J.3.1.1.2. <Protobuf ConnectionHeader Message></h5></div></div></div><p>Has user info, and “protocol”, as well as the encoders and compression the client will use sending CellBlocks.
CellBlock encoders and compressors are for the life of the connection.
CellBlock encoders implement org.apache.hadoop.hbase.codec.Codec.
CellBlocks may then also be compressed.
Compressors implement org.apache.hadoop.io.compress.CompressionCodec.
This protobuf is written using writeDelimited so is prefaced by a pb varint
- with its serialized length</p></div></div><div class="section" title="J.3.1.2. Server"><div class="titlepage"><div><div><h4 class="title"><a name="d2475e13035"></a>J.3.1.2. Server</h4></div></div></div><p>After client sends preamble and connection header,
+ with its serialized length</p></div></div><div class="section" title="J.3.1.2. Server"><div class="titlepage"><div><div><h4 class="title"><a name="d2519e13241"></a>J.3.1.2. Server</h4></div></div></div><p>After client sends preamble and connection header,
server does NOT respond if successful connection setup.
No response means server is READY to accept requests and to give out response.
If the version or authentication in the preamble is not agreeable or the server has trouble parsing the preamble,
it will throw a org.apache.hadoop.hbase.ipc.FatalConnectionException explaining the error and will then disconnect.
If the client in the connection header -- i.e. the protobuf’d Message that comes after the connection preamble -- asks for for a
- Service the server does not support or a codec the server does not have, again we throw a FatalConnectionException with explanation.</p></div></div><div class="section" title="J.3.2. Request"><div class="titlepage"><div><div><h3 class="title"><a name="d2475e13040"></a>J.3.2. Request</h3></div></div></div><p>After a Connection has been set up, client makes requests. Server responds.</p><p>A request is made up of a protobuf RequestHeader followed by a protobuf Message parameter.
+ Service the server does not support or a codec the server does not have, again we throw a FatalConnectionException with explanation.</p></div></div><div class="section" title="J.3.2. Request"><div class="titlepage"><div><div><h3 class="title"><a name="d2519e13246"></a>J.3.2. Request</h3></div></div></div><p>After a Connection has been set up, client makes requests. Server responds.</p><p>A request is made up of a protobuf RequestHeader followed by a protobuf Message parameter.
The header includes the method name and optionally, metadata on the optional CellBlock that may be following.
The parameter type suits the method being invoked: i.e. if we are doing a getRegionInfo request,
the protobuf Message param will be an instance of GetRegionInfoRequest.
The response will be a GetRegionInfoResponse.
- The CellBlock is optionally used ferrying the bulk of the RPC data: i.e Cells/KeyValues.</p><p></p><div class="section" title="J.3.2.1. Request Parts"><div class="titlepage"><div><div><h4 class="title"><a name="d2475e13048"></a>J.3.2.1. Request Parts</h4></div></div></div><div class="section" title="J.3.2.1.1. <Total Length>"><div class="titlepage"><div><div><h5 class="title"><a name="d2475e13051"></a>J.3.2.1.1. <Total Length></h5></div></div></div><p>The request is prefaced by an int that holds the total length of what follows.</p></div><div class="section" title="J.3.2.1.2. <Protobuf RequestHeader Message>"><div class="titlepage"><div><div><h5 class="title"><a name="d2475e13056"></a>J.3.2.1.2. <Protobuf RequestHeader Message></h5></div></div></div><p>Will have call.id, trace.id, and method name, etc. including optional Metadata on the Cell block IFF one is following.
- Data is protobuf’d inline in this pb Message or optionally comes in the following CellBlock</p></div><div class="section" title="J.3.2.1.3. <Protobuf Param Message>"><div class="titlepage"><div><div><h5 class="title"><a name="d2475e13061"></a>J.3.2.1.3. <Protobuf Param Message></h5></div></div></div><p>If the method being invoked is getRegionInfo, if you study the Service descriptor for the client to regionserver protocol,
- you will find that the request sends a GetRegionInfoRequest protobuf Message param in this position.</p></div><div class="section" title="J.3.2.1.4. <CellBlock>"><div class="titlepage"><div><div><h5 class="title"><a name="d2475e13066"></a>J.3.2.1.4. <CellBlock></h5></div></div></div><p>An encoded and optionally compressed Cell block.</p></div></div></div><div class="section" title="J.3.3. Response"><div class="titlepage"><div><div><h3 class="title"><a name="d2475e13073"></a>J.3.3. Response</h3></div></div></div><p>Same as Request, it is a protobuf ResponseHeader followed by a protobuf Message response where the Message response type suits the method invoked.
- Bulk of the data may come in a following CellBlock.</p><div class="section" title="J.3.3.1. Response Parts"><div class="titlepage"><div><div><h4 class="title"><a name="d2475e13078"></a>J.3.3.1. Response Parts</h4></div></div></div><div class="section" title="J.3.3.1.1. <Total Length>"><div class="titlepage"><div><div><h5 class="title"><a name="d2475e13081"></a>J.3.3.1.1. <Total Length></h5></div></div></div><p>The response is prefaced by an int that holds the total length of what follows.</p></div><div class="section" title="J.3.3.1.2. <Protobuf ResponseHeader Message>"><div class="titlepage"><div><div><h5 class="title"><a name="d2475e13086"></a>J.3.3.1.2. <Protobuf ResponseHeader Message></h5></div></div></div><p>Will have call.id, etc. Will include exception if failed processing. Optionally includes metadata on optional, IFF there is a CellBlock following.</p></div><div class="section" title="J.3.3.1.3. <
Protobuf Response Message>"><div class="titlepage"><div><div><h5 class="title"><a name="d2475e13091"></a>J.3.3.1.3. <Protobuf Response Message></h5></div></div></div><p>Return or may be nothing if exception. If the method being invoked is getRegionInfo, if you study the Service descriptor for the client to regionserver protocol,
- you will find that the response sends a GetRegionInfoResponse protobuf Message param in this position.</p></div><div class="section" title="J.3.3.1.4. <CellBlock>"><div class="titlepage"><div><div><h5 class="title"><a name="d2475e13096"></a>J.3.3.1.4. <CellBlock></h5></div></div></div><p>An encoded and optionally compressed Cell block.</p></div></div></div><div class="section" title="J.3.4. Exceptions"><div class="titlepage"><div><div><h3 class="title"><a name="d2475e13103"></a>J.3.4. Exceptions</h3></div></div></div><p>There are two distinct types.
+ The CellBlock is optionally used ferrying the bulk of the RPC data: i.e Cells/KeyValues.</p><p></p><div class="section" title="J.3.2.1. Request Parts"><div class="titlepage"><div><div><h4 class="title"><a name="d2519e13254"></a>J.3.2.1. Request Parts</h4></div></div></div><div class="section" title="J.3.2.1.1. <Total Length>"><div class="titlepage"><div><div><h5 class="title"><a name="d2519e13257"></a>J.3.2.1.1. <Total Length></h5></div></div></div><p>The request is prefaced by an int that holds the total length of what follows.</p></div><div class="section" title="J.3.2.1.2. <Protobuf RequestHeader Message>"><div class="titlepage"><div><div><h5 class="title"><a name="d2519e13262"></a>J.3.2.1.2. <Protobuf RequestHeader Message></h5></div></div></div><p>Will have call.id, trace.id, and method name, etc. including optional Metadata on the Cell block IFF one is following.
+ Data is protobuf’d inline in this pb Message or optionally comes in the following CellBlock</p></div><div class="section" title="J.3.2.1.3. <Protobuf Param Message>"><div class="titlepage"><div><div><h5 class="title"><a name="d2519e13267"></a>J.3.2.1.3. <Protobuf Param Message></h5></div></div></div><p>If the method being invoked is getRegionInfo, if you study the Service descriptor for the client to regionserver protocol,
+ you will find that the request sends a GetRegionInfoRequest protobuf Message param in this position.</p></div><div class="section" title="J.3.2.1.4. <CellBlock>"><div class="titlepage"><div><div><h5 class="title"><a name="d2519e13272"></a>J.3.2.1.4. <CellBlock></h5></div></div></div><p>An encoded and optionally compressed Cell block.</p></div></div></div><div class="section" title="J.3.3. Response"><div class="titlepage"><div><div><h3 class="title"><a name="d2519e13279"></a>J.3.3. Response</h3></div></div></div><p>Same as Request, it is a protobuf ResponseHeader followed by a protobuf Message response where the Message response type suits the method invoked.
+ Bulk of the data may come in a following CellBlock.</p><div class="section" title="J.3.3.1. Response Parts"><div class="titlepage"><div><div><h4 class="title"><a name="d2519e13284"></a>J.3.3.1. Response Parts</h4></div></div></div><div class="section" title="J.3.3.1.1. <Total Length>"><div class="titlepage"><div><div><h5 class="title"><a name="d2519e13287"></a>J.3.3.1.1. <Total Length></h5></div></div></div><p>The response is prefaced by an int that holds the total length of what follows.</p></div><div class="section" title="J.3.3.1.2. <Protobuf ResponseHeader Message>"><div class="titlepage"><div><div><h5 class="title"><a name="d2519e13292"></a>J.3.3.1.2. <Protobuf ResponseHeader Message></h5></div></div></div><p>Will have call.id, etc. Will include exception if failed processing. Optionally includes metadata on optional, IFF there is a CellBlock following.</p></div><div class="section" title="J.3.3.1.3. <
Protobuf Response Message>"><div class="titlepage"><div><div><h5 class="title"><a name="d2519e13297"></a>J.3.3.1.3. <Protobuf Response Message></h5></div></div></div><p>Return or may be nothing if exception. If the method being invoked is getRegionInfo, if you study the Service descriptor for the client to regionserver protocol,
+ you will find that the response sends a GetRegionInfoResponse protobuf Message param in this position.</p></div><div class="section" title="J.3.3.1.4. <CellBlock>"><div class="titlepage"><div><div><h5 class="title"><a name="d2519e13302"></a>J.3.3.1.4. <CellBlock></h5></div></div></div><p>An encoded and optionally compressed Cell block.</p></div></div></div><div class="section" title="J.3.4. Exceptions"><div class="titlepage"><div><div><h3 class="title"><a name="d2519e13309"></a>J.3.4. Exceptions</h3></div></div></div><p>There are two distinct types.
There is the request failed which is encapsulated inside the response header for the response.
The connection stays open to receive new requests.
The second type, the FatalConnectionException, kills the connection.</p><p>Exceptions can carry extra information.
See the ExceptionResponse protobuf type.
- It has a flag to indicate do-no-retry as well as other miscellaneous payload to help improve client responsiveness.</p></div><div class="section" title="J.3.5. CellBlocks"><div class="titlepage"><div><div><h3 class="title"><a name="d2475e13110"></a>J.3.5. CellBlocks</h3></div></div></div><p>These are not versioned.
+ It has a flag to indicate do-no-retry as well as other miscellaneous payload to help improve client responsiveness.</p></div><div class="section" title="J.3.5. CellBlocks"><div class="titlepage"><div><div><h3 class="title"><a name="d2519e13316"></a>J.3.5. CellBlocks</h3></div></div></div><p>These are not versioned.
Server can do the codec or it cannot.
If new version of a codec with say, tighter encoding, then give it a new class name.
- Codecs will live on the server for all time so old clients can connect.</p></div><div class="footnotes"><br><hr width="100" align="left"><div class="footnote"><p><sup>[<a id="ftn.d2475e13024" href="#d2475e13024" class="para">34</a>] </sup> We need the auth method spec. here so the connection header is encoded if auth enabled.</p></div></div></div><div id="disqus_thread"></div><script type="text/javascript">
+ Codecs will live on the server for all time so old clients can connect.</p></div><div class="footnotes"><br><hr width="100" align="left"><div class="footnote"><p><sup>[<a id="ftn.d2519e13230" href="#d2519e13230" class="para">35</a>] </sup> We need the auth method spec. here so the connection header is encoded if auth enabled.</p></div></div></div><div id="disqus_thread"></div><script type="text/javascript">
var disqus_shortname = 'hbase'; // required: replace example with your forum shortname
var disqus_url = 'http://hbase.apache.org/book';
var disqus_identifier = '';
Modified: hbase/hbase.apache.org/trunk/book/apjs04.html
URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/apjs04.html?rev=1465200&r1=1465199&r2=1465200&view=diff
==============================================================================
--- hbase/hbase.apache.org/trunk/book/apjs04.html (original)
+++ hbase/hbase.apache.org/trunk/book/apjs04.html Sat Apr 6 06:08:56 2013
@@ -1,9 +1,9 @@
<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
- <title>J.4. Notes</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="hbase.rpc.html" title="Appendix J. 0.95 RPC Specification"><link rel="prev" href="apjs03.html" title="J.3. RPC"><link rel="next" href="book_index.html" title="Index"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">J.4. Notes</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="apjs03.html">Prev</a> </td><th width="60%" align="center">Appendix J. 0.95 RPC Specification</th><td width="20%" align="right"> <a accesskey="n" href="book_index.html">Next</a></td></tr></table><hr></div><div class="section" title="J.4.
Notes"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="d2475e13115"></a>J.4. Notes</h2></div></div></div><div class="section" title="J.4.1. Constraints"><div class="titlepage"><div><div><h3 class="title"><a name="d2475e13118"></a>J.4.1. Constraints</h3></div></div></div><p>In some part, current wire-format -- i.e. all requests and responses preceeded by a length -- has been dictated by current server non-async architecture.</p></div><div class="section" title="J.4.2. One fat pb request or header+param"><div class="titlepage"><div><div><h3 class="title"><a name="d2475e13123"></a>J.4.2. One fat pb request or header+param</h3></div></div></div><p>We went with pb header followed by pb param making a request and a pb header followed by pb response for now.
+ <title>J.4. Notes</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="hbase.rpc.html" title="Appendix J. 0.95 RPC Specification"><link rel="prev" href="apjs03.html" title="J.3. RPC"><link rel="next" href="book_index.html" title="Index"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">J.4. Notes</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="apjs03.html">Prev</a> </td><th width="60%" align="center">Appendix J. 0.95 RPC Specification</th><td width="20%" align="right"> <a accesskey="n" href="book_index.html">Next</a></td></tr></table><hr></div><div class="section" title="J.4.
Notes"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="d2519e13321"></a>J.4. Notes</h2></div></div></div><div class="section" title="J.4.1. Constraints"><div class="titlepage"><div><div><h3 class="title"><a name="d2519e13324"></a>J.4.1. Constraints</h3></div></div></div><p>In some part, current wire-format -- i.e. all requests and responses preceeded by a length -- has been dictated by current server non-async architecture.</p></div><div class="section" title="J.4.2. One fat pb request or header+param"><div class="titlepage"><div><div><h3 class="title"><a name="d2519e13329"></a>J.4.2. One fat pb request or header+param</h3></div></div></div><p>We went with pb header followed by pb param making a request and a pb header followed by pb response for now.
Doing header+param rather than a single protobuf Message with both header and param content:</p><p>
</p><div class="orderedlist"><ol class="orderedlist" type="1"><li class="listitem"><p>Is closer to what we currently have</p></li><li class="listitem"><p>Having a single fat pb requires extra copying putting the already pb’d param into the body of the fat request pb (and same making result)</p></li><li class="listitem"><p>We can decide whether to accept the request or not before we read the param; for example, the request might be low priority. As is, we read header+param in one go as server is currently implemented so this is a TODO.</p></li></ol></div><p>
- </p><p>The advantages are minor. If later, fat request has clear advantage, can roll out a v2 later.</p></div><div class="section" title="J.4.3. Compression"><div class="titlepage"><div><div><h3 class="title"><a name="d2475e13143"></a>J.4.3. Compression</h3></div></div></div><p>Uses hadoops compression codecs.</p></div></div><div id="disqus_thread"></div><script type="text/javascript">
+ </p><p>The advantages are minor. If later, fat request has clear advantage, can roll out a v2 later.</p></div><div class="section" title="J.4.3. Compression"><div class="titlepage"><div><div><h3 class="title"><a name="d2519e13349"></a>J.4.3. Compression</h3></div></div></div><p>Uses hadoops compression codecs.</p></div></div><div id="disqus_thread"></div><script type="text/javascript">
var disqus_shortname = 'hbase'; // required: replace example with your forum shortname
var disqus_url = 'http://hbase.apache.org/book';
var disqus_identifier = '';
Modified: hbase/hbase.apache.org/trunk/book/architecture.html
URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/architecture.html?rev=1465200&r1=1465199&r2=1465200&view=diff
==============================================================================
--- hbase/hbase.apache.org/trunk/book/architecture.html (original)
+++ hbase/hbase.apache.org/trunk/book/architecture.html Sat Apr 6 06:08:56 2013
@@ -1,6 +1,6 @@
<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
- <title>Chapter 9. Architecture</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="prev" href="hbase.secure.bulkload.html" title="8.3. Secure Bulk Load"><link rel="next" href="arch.catalog.html" title="9.2. Catalog Tables"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">Chapter 9. Architecture</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="hbase.secure.bulkload.html">Prev</a> </td><th width="60%" align="center"> </th><td width="20%" align="right"> <a accesskey="n" href="arch.catalog.html">Next</a></td></
tr></table><hr></div><div class="chapter" title="Chapter 9. Architecture"><div class="titlepage"><div><div><h2 class="title"><a name="architecture"></a>Chapter 9. Architecture</h2></div></div></div><div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="section"><a href="architecture.html#arch.overview">9.1. Overview</a></span></dt><dd><dl><dt><span class="section"><a href="architecture.html#arch.overview.nosql">9.1.1. NoSQL?</a></span></dt><dt><span class="section"><a href="architecture.html#arch.overview.when">9.1.2. When Should I Use HBase?</a></span></dt><dt><span class="section"><a href="architecture.html#arch.overview.hbasehdfs">9.1.3. What Is The Difference Between HBase and Hadoop/HDFS?</a></span></dt></dl></dd><dt><span class="section"><a href="arch.catalog.html">9.2. Catalog Tables</a></span></dt><dd><dl><dt><span class="section"><a href="arch.catalog.html#arch.catalog.root">9.2.1. ROOT</a></span></dt><dt><span class="section"><a hr
ef="arch.catalog.html#arch.catalog.meta">9.2.2. META</a></span></dt><dt><span class="section"><a href="arch.catalog.html#arch.catalog.startup">9.2.3. Startup Sequencing</a></span></dt></dl></dd><dt><span class="section"><a href="client.html">9.3. Client</a></span></dt><dd><dl><dt><span class="section"><a href="client.html#client.connections">9.3.1. Connections</a></span></dt><dt><span class="section"><a href="client.html#client.writebuffer">9.3.2. WriteBuffer and Batch Methods</a></span></dt><dt><span class="section"><a href="client.html#client.external">9.3.3. External Clients</a></span></dt><dt><span class="section"><a href="client.html#client.rowlocks">9.3.4. RowLocks</a></span></dt></dl></dd><dt><span class="section"><a href="client.filter.html">9.4. Client Request Filters</a></span></dt><dd><dl><dt><span class="section"><a href="client.filter.html#client.filter.structural">9.4.1. Structural</a></span></dt><dt><span class="section"><a href="client.filter.html#client.filt
er.cv">9.4.2. Column Value</a></span></dt><dt><span class="section"><a href="client.filter.html#client.filter.cvp">9.4.3. Column Value Comparators</a></span></dt><dt><span class="section"><a href="client.filter.html#client.filter.kvm">9.4.4. KeyValue Metadata</a></span></dt><dt><span class="section"><a href="client.filter.html#client.filter.row">9.4.5. RowKey</a></span></dt><dt><span class="section"><a href="client.filter.html#client.filter.utility">9.4.6. Utility</a></span></dt></dl></dd><dt><span class="section"><a href="master.html">9.5. Master</a></span></dt><dd><dl><dt><span class="section"><a href="master.html#master.startup">9.5.1. Startup Behavior</a></span></dt><dt><span class="section"><a href="master.html#master.runtime">9.5.2. Runtime Impact</a></span></dt><dt><span class="section"><a href="master.html#master.api">9.5.3. Interface</a></span></dt><dt><span class="section"><a href="master.html#master.processes">9.5.4. Processes</a></span></dt></dl></dd><dt><span cl
ass="section"><a href="regionserver.arch.html">9.6. RegionServer</a></span></dt><dd><dl><dt><span class="section"><a href="regionserver.arch.html#regionserver.arch.api">9.6.1. Interface</a></span></dt><dt><span class="section"><a href="regionserver.arch.html#regionserver.arch.processes">9.6.2. Processes</a></span></dt><dt><span class="section"><a href="regionserver.arch.html#coprocessors">9.6.3. Coprocessors</a></span></dt><dt><span class="section"><a href="regionserver.arch.html#block.cache">9.6.4. Block Cache</a></span></dt><dt><span class="section"><a href="regionserver.arch.html#wal">9.6.5. Write Ahead Log (WAL)</a></span></dt></dl></dd><dt><span class="section"><a href="regions.arch.html">9.7. Regions</a></span></dt><dd><dl><dt><span class="section"><a href="regions.arch.html#arch.regions.size">9.7.1. Region Size</a></span></dt><dt><span class="section"><a href="regions.arch.html#regions.arch.assignment">9.7.2. Region-RegionServer Assignment</a></span></dt><dt><span cla
ss="section"><a href="regions.arch.html#regions.arch.locality">9.7.3. Region-RegionServer Locality</a></span></dt><dt><span class="section"><a href="regions.arch.html#d2475e5763">9.7.4. Region Splits</a></span></dt><dt><span class="section"><a href="regions.arch.html#d2475e5786">9.7.5. Online Region Merges</a></span></dt><dt><span class="section"><a href="regions.arch.html#store">9.7.6. Store</a></span></dt></dl></dd><dt><span class="section"><a href="arch.bulk.load.html">9.8. Bulk Loading</a></span></dt><dd><dl><dt><span class="section"><a href="arch.bulk.load.html#arch.bulk.load.overview">9.8.1. Overview</a></span></dt><dt><span class="section"><a href="arch.bulk.load.html#arch.bulk.load.arch">9.8.2. Bulk Load Architecture</a></span></dt><dt><span class="section"><a href="arch.bulk.load.html#arch.bulk.load.import">9.8.3. Importing the prepared data using the completebulkload tool</a></span></dt><dt><span class="section"><a href="arch.bulk.load.html#arch.bulk.load.also">9.8
.4. See Also</a></span></dt><dt><span class="section"><a href="arch.bulk.load.html#arch.bulk.load.adv">9.8.5. Advanced Usage</a></span></dt></dl></dd><dt><span class="section"><a href="arch.hdfs.html">9.9. HDFS</a></span></dt><dd><dl><dt><span class="section"><a href="arch.hdfs.html#arch.hdfs.nn">9.9.1. NameNode</a></span></dt><dt><span class="section"><a href="arch.hdfs.html#arch.hdfs.dn">9.9.2. DataNode</a></span></dt></dl></dd></dl></div><div class="section" title="9.1. Overview"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="arch.overview"></a>9.1. Overview</h2></div></div></div><div class="section" title="9.1.1. NoSQL?"><div class="titlepage"><div><div><h3 class="title"><a name="arch.overview.nosql"></a>9.1.1. NoSQL?</h3></div></div></div><p>HBase is a type of "NoSQL" database. "NoSQL" is a general term meaning that the database isn't an RDBMS which
+ <title>Chapter 9. Architecture</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="prev" href="hbase.secure.bulkload.html" title="8.3. Secure Bulk Load"><link rel="next" href="arch.catalog.html" title="9.2. Catalog Tables"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">Chapter 9. Architecture</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="hbase.secure.bulkload.html">Prev</a> </td><th width="60%" align="center"> </th><td width="20%" align="right"> <a accesskey="n" href="arch.catalog.html">Next</a></td></
tr></table><hr></div><div class="chapter" title="Chapter 9. Architecture"><div class="titlepage"><div><div><h2 class="title"><a name="architecture"></a>Chapter 9. Architecture</h2></div></div></div><div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="section"><a href="architecture.html#arch.overview">9.1. Overview</a></span></dt><dd><dl><dt><span class="section"><a href="architecture.html#arch.overview.nosql">9.1.1. NoSQL?</a></span></dt><dt><span class="section"><a href="architecture.html#arch.overview.when">9.1.2. When Should I Use HBase?</a></span></dt><dt><span class="section"><a href="architecture.html#arch.overview.hbasehdfs">9.1.3. What Is The Difference Between HBase and Hadoop/HDFS?</a></span></dt></dl></dd><dt><span class="section"><a href="arch.catalog.html">9.2. Catalog Tables</a></span></dt><dd><dl><dt><span class="section"><a href="arch.catalog.html#arch.catalog.root">9.2.1. ROOT</a></span></dt><dt><span class="section"><a hr
ef="arch.catalog.html#arch.catalog.meta">9.2.2. META</a></span></dt><dt><span class="section"><a href="arch.catalog.html#arch.catalog.startup">9.2.3. Startup Sequencing</a></span></dt></dl></dd><dt><span class="section"><a href="client.html">9.3. Client</a></span></dt><dd><dl><dt><span class="section"><a href="client.html#client.connections">9.3.1. Connections</a></span></dt><dt><span class="section"><a href="client.html#client.writebuffer">9.3.2. WriteBuffer and Batch Methods</a></span></dt><dt><span class="section"><a href="client.html#client.external">9.3.3. External Clients</a></span></dt><dt><span class="section"><a href="client.html#client.rowlocks">9.3.4. RowLocks</a></span></dt></dl></dd><dt><span class="section"><a href="client.filter.html">9.4. Client Request Filters</a></span></dt><dd><dl><dt><span class="section"><a href="client.filter.html#client.filter.structural">9.4.1. Structural</a></span></dt><dt><span class="section"><a href="client.filter.html#client.filt
er.cv">9.4.2. Column Value</a></span></dt><dt><span class="section"><a href="client.filter.html#client.filter.cvp">9.4.3. Column Value Comparators</a></span></dt><dt><span class="section"><a href="client.filter.html#client.filter.kvm">9.4.4. KeyValue Metadata</a></span></dt><dt><span class="section"><a href="client.filter.html#client.filter.row">9.4.5. RowKey</a></span></dt><dt><span class="section"><a href="client.filter.html#client.filter.utility">9.4.6. Utility</a></span></dt></dl></dd><dt><span class="section"><a href="master.html">9.5. Master</a></span></dt><dd><dl><dt><span class="section"><a href="master.html#master.startup">9.5.1. Startup Behavior</a></span></dt><dt><span class="section"><a href="master.html#master.runtime">9.5.2. Runtime Impact</a></span></dt><dt><span class="section"><a href="master.html#master.api">9.5.3. Interface</a></span></dt><dt><span class="section"><a href="master.html#master.processes">9.5.4. Processes</a></span></dt></dl></dd><dt><span cl
ass="section"><a href="regionserver.arch.html">9.6. RegionServer</a></span></dt><dd><dl><dt><span class="section"><a href="regionserver.arch.html#regionserver.arch.api">9.6.1. Interface</a></span></dt><dt><span class="section"><a href="regionserver.arch.html#regionserver.arch.processes">9.6.2. Processes</a></span></dt><dt><span class="section"><a href="regionserver.arch.html#coprocessors">9.6.3. Coprocessors</a></span></dt><dt><span class="section"><a href="regionserver.arch.html#block.cache">9.6.4. Block Cache</a></span></dt><dt><span class="section"><a href="regionserver.arch.html#wal">9.6.5. Write Ahead Log (WAL)</a></span></dt></dl></dd><dt><span class="section"><a href="regions.arch.html">9.7. Regions</a></span></dt><dd><dl><dt><span class="section"><a href="regions.arch.html#arch.regions.size">9.7.1. Region Size</a></span></dt><dt><span class="section"><a href="regions.arch.html#regions.arch.assignment">9.7.2. Region-RegionServer Assignment</a></span></dt><dt><span cla
ss="section"><a href="regions.arch.html#regions.arch.locality">9.7.3. Region-RegionServer Locality</a></span></dt><dt><span class="section"><a href="regions.arch.html#d2519e5996">9.7.4. Region Splits</a></span></dt><dt><span class="section"><a href="regions.arch.html#d2519e6019">9.7.5. Online Region Merges</a></span></dt><dt><span class="section"><a href="regions.arch.html#store">9.7.6. Store</a></span></dt></dl></dd><dt><span class="section"><a href="arch.bulk.load.html">9.8. Bulk Loading</a></span></dt><dd><dl><dt><span class="section"><a href="arch.bulk.load.html#arch.bulk.load.overview">9.8.1. Overview</a></span></dt><dt><span class="section"><a href="arch.bulk.load.html#arch.bulk.load.arch">9.8.2. Bulk Load Architecture</a></span></dt><dt><span class="section"><a href="arch.bulk.load.html#arch.bulk.load.import">9.8.3. Importing the prepared data using the completebulkload tool</a></span></dt><dt><span class="section"><a href="arch.bulk.load.html#arch.bulk.load.also">9.8
.4. See Also</a></span></dt><dt><span class="section"><a href="arch.bulk.load.html#arch.bulk.load.adv">9.8.5. Advanced Usage</a></span></dt></dl></dd><dt><span class="section"><a href="arch.hdfs.html">9.9. HDFS</a></span></dt><dd><dl><dt><span class="section"><a href="arch.hdfs.html#arch.hdfs.nn">9.9.1. NameNode</a></span></dt><dt><span class="section"><a href="arch.hdfs.html#arch.hdfs.dn">9.9.2. DataNode</a></span></dt></dl></dd></dl></div><div class="section" title="9.1. Overview"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="arch.overview"></a>9.1. Overview</h2></div></div></div><div class="section" title="9.1.1. NoSQL?"><div class="titlepage"><div><div><h3 class="title"><a name="arch.overview.nosql"></a>9.1.1. NoSQL?</h3></div></div></div><p>HBase is a type of "NoSQL" database. "NoSQL" is a general term meaning that the database isn't an RDBMS which
supports SQL as its primary access language, but there are many types of NoSQL databases: BerkeleyDB is an
example of a local NoSQL database, whereas HBase is very much a distributed database. Technically speaking,
HBase is really more a "Data Store" than "Data Base" because it lacks many of the features you find in an RDBMS,