You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hbase.apache.org by st...@apache.org on 2013/04/02 20:06:21 UTC
svn commit: r1463652 [9/21] - in /hbase/hbase.apache.org/trunk: ./ book/
case_studies/ community/ configuration/ css/ developer/ external_apis/
getting_started/ hbase-assembly/ hbase-assembly/book/ hbase-assembly/xref/
images/ ops_mgt/ performance/ pre...
Modified: hbase/hbase.apache.org/trunk/book/upgrade0.94.html
URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/upgrade0.94.html?rev=1463652&r1=1463651&r2=1463652&view=diff
==============================================================================
--- hbase/hbase.apache.org/trunk/book/upgrade0.94.html (original)
+++ hbase/hbase.apache.org/trunk/book/upgrade0.94.html Tue Apr 2 18:06:19 2013
@@ -1,6 +1,6 @@
<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
- <title>3.2. Upgrading from 0.92.x to 0.94.x</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="upgrading.html" title="Chapter 3. Upgrading"><link rel="prev" href="upgrading.html" title="Chapter 3. Upgrading"><link rel="next" href="upgrade0.92.html" title="3.3. Upgrading from 0.90.x to 0.92.x"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">3.2. Upgrading from 0.92.x to 0.94.x</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="upgrading.html">Prev</a> </td><th width="60%" align="center">Chapter 3. Upgrading</th><td width="20%" align="right"> <a accesskey="n" href="u
pgrade0.92.html">Next</a></td></tr></table><hr></div><div class="section" title="3.2. Upgrading from 0.92.x to 0.94.x"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="upgrade0.94"></a>3.2. Upgrading from 0.92.x to 0.94.x</h2></div></div></div><p>0.92 and 0.94 are interface compatible. You can do a rolling upgrade between these versions.
+ <title>3.3. Upgrading from 0.92.x to 0.94.x</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="upgrading.html" title="Chapter 3. Upgrading"><link rel="prev" href="upgrade0.96.html" title="3.2. Upgrading from 0.94.x to 0.96.x"><link rel="next" href="upgrade0.92.html" title="3.4. Upgrading from 0.90.x to 0.92.x"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">3.3. Upgrading from 0.92.x to 0.94.x</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="upgrade0.96.html">Prev</a> </td><th width="60%" align="center">Chapter 3. Upgrading</th><td width="20%" align="right"> <a access
key="n" href="upgrade0.92.html">Next</a></td></tr></table><hr></div><div class="section" title="3.3. Upgrading from 0.92.x to 0.94.x"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="upgrade0.94"></a>3.3. Upgrading from 0.92.x to 0.94.x</h2></div></div></div><p>0.92 and 0.94 are interface compatible. You can do a rolling upgrade between these versions.
</p></div><div id="disqus_thread"></div><script type="text/javascript">
var disqus_shortname = 'hbase'; // required: replace example with your forum shortname
var disqus_url = 'http://hbase.apache.org/book';
@@ -12,4 +12,4 @@
dsq.src = 'http://' + disqus_shortname + '.disqus.com/embed.js';
(document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
})();
-</script><noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript">comments powered by Disqus.</a></noscript><a href="http://disqus.com" class="dsq-brlink">comments powered by <span class="logo-disqus">Disqus</span></a><div class="navfooter"><hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="upgrading.html">Prev</a> </td><td width="20%" align="center"><a accesskey="u" href="upgrading.html">Up</a></td><td width="40%" align="right"> <a accesskey="n" href="upgrade0.92.html">Next</a></td></tr><tr><td width="40%" align="left" valign="top">Chapter 3. Upgrading </td><td width="20%" align="center"><a accesskey="h" href="book.html">Home</a></td><td width="40%" align="right" valign="top"> 3.3. Upgrading from 0.90.x to 0.92.x</td></tr></table></div></body></html>
\ No newline at end of file
+</script><noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript">comments powered by Disqus.</a></noscript><a href="http://disqus.com" class="dsq-brlink">comments powered by <span class="logo-disqus">Disqus</span></a><div class="navfooter"><hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="upgrade0.96.html">Prev</a> </td><td width="20%" align="center"><a accesskey="u" href="upgrading.html">Up</a></td><td width="40%" align="right"> <a accesskey="n" href="upgrade0.92.html">Next</a></td></tr><tr><td width="40%" align="left" valign="top">3.2. Upgrading from 0.94.x to 0.96.x </td><td width="20%" align="center"><a accesskey="h" href="book.html">Home</a></td><td width="40%" align="right" valign="top"> 3.4. Upgrading from 0.90.x to 0.92.x</td></tr></table></div></body></html>
\ No newline at end of file
Modified: hbase/hbase.apache.org/trunk/book/upgrading.html
URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/upgrading.html?rev=1463652&r1=1463651&r2=1463652&view=diff
==============================================================================
--- hbase/hbase.apache.org/trunk/book/upgrading.html (original)
+++ hbase/hbase.apache.org/trunk/book/upgrading.html Tue Apr 2 18:06:19 2013
@@ -1,28 +1,39 @@
<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
- <title>Chapter 3. Upgrading</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="prev" href="important_configurations.html" title="2.5. The Important Configurations"><link rel="next" href="upgrade0.94.html" title="3.2. Upgrading from 0.92.x to 0.94.x"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">Chapter 3. Upgrading</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="important_configurations.html">Prev</a> </td><th width="60%" align="center"> </th><td width="20%" align="right"> <a accesskey="n" href="upgrad
e0.94.html">Next</a></td></tr></table><hr></div><div class="chapter" title="Chapter 3. Upgrading"><div class="titlepage"><div><div><h2 class="title"><a name="upgrading"></a>Chapter 3. Upgrading</h2></div></div></div><div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="section"><a href="upgrading.html#upgrade0.96">3.1. Upgrading from 0.94.x to 0.96.x</a></span></dt><dt><span class="section"><a href="upgrade0.94.html">3.2. Upgrading from 0.92.x to 0.94.x</a></span></dt><dt><span class="section"><a href="upgrade0.92.html">3.3. Upgrading from 0.90.x to 0.92.x</a></span></dt><dd><dl><dt><span class="section"><a href="upgrade0.92.html#d2279e2710">3.3.1. You can’t go back!
-</a></span></dt><dt><span class="section"><a href="upgrade0.92.html#d2279e2724">3.3.2. MSLAB is ON by default
-</a></span></dt><dt><span class="section"><a href="upgrade0.92.html#d2279e2749">3.3.3. Distributed splitting is on by default
-</a></span></dt><dt><span class="section"><a href="upgrade0.92.html#d2279e2754">3.3.4. Memory accounting is different now
-</a></span></dt><dt><span class="section"><a href="upgrade0.92.html#d2279e2763">3.3.5. On the Hadoop version to use
-</a></span></dt><dt><span class="section"><a href="upgrade0.92.html#d2279e2775">3.3.6. HBase 0.92.0 ships with ZooKeeper 3.4.2
-</a></span></dt><dt><span class="section"><a href="upgrade0.92.html#d2279e2780">3.3.7. Online alter is off by default
-</a></span></dt><dt><span class="section"><a href="upgrade0.92.html#d2279e2787">3.3.8. WebUI
-</a></span></dt><dt><span class="section"><a href="upgrade0.92.html#d2279e2792">3.3.9. Security tarball
-</a></span></dt><dt><span class="section"><a href="upgrade0.92.html#d2279e2797">3.3.10. Experimental off-heap cache
-</a></span></dt><dt><span class="section"><a href="upgrade0.92.html#d2279e2802">3.3.11. Changes in HBase replication
-</a></span></dt><dt><span class="section"><a href="upgrade0.92.html#d2279e2807">3.3.12. RegionServer now aborts if OOME
-</a></span></dt><dt><span class="section"><a href="upgrade0.92.html#d2279e2812">3.3.13. HFile V2 and the “Bigger, Fewer” Tendency
-</a></span></dt></dl></dd><dt><span class="section"><a href="upgrade0.90.html">3.4. Upgrading to HBase 0.90.x from 0.20.x or 0.89.x</a></span></dt></dl></div><p>You cannot skip major verisons upgrading. If you are upgrading from
- version 0.20.x to 0.92.x, you must first go from 0.20.x to 0.90.x and then go
- from 0.90.x to 0.92.x.</p><p>
+ <title>Chapter 3. Upgrading</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="prev" href="important_configurations.html" title="2.5. The Important Configurations"><link rel="next" href="upgrade0.96.html" title="3.2. Upgrading from 0.94.x to 0.96.x"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">Chapter 3. Upgrading</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="important_configurations.html">Prev</a> </td><th width="60%" align="center"> </th><td width="20%" align="right"> <a accesskey="n" href="upgrad
e0.96.html">Next</a></td></tr></table><hr></div><div class="chapter" title="Chapter 3. Upgrading"><div class="titlepage"><div><div><h2 class="title"><a name="upgrading"></a>Chapter 3. Upgrading</h2></div></div></div><div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="section"><a href="upgrading.html#hbase.versioning">3.1. HBase version numbers</a></span></dt><dd><dl><dt><span class="section"><a href="upgrading.html#hbase.development.series">3.1.1. Odd/Even Versioning or "Development"" Series Releases</a></span></dt></dl></dd><dt><span class="section"><a href="upgrade0.96.html">3.2. Upgrading from 0.94.x to 0.96.x</a></span></dt><dt><span class="section"><a href="upgrade0.94.html">3.3. Upgrading from 0.92.x to 0.94.x</a></span></dt><dt><span class="section"><a href="upgrade0.92.html">3.4. Upgrading from 0.90.x to 0.92.x</a></span></dt><dd><dl><dt><span class="section"><a href="upgrade0.92.html#d2475e2714">3.4.1. You can’t go back!
+</a></span></dt><dt><span class="section"><a href="upgrade0.92.html#d2475e2728">3.4.2. MSLAB is ON by default
+</a></span></dt><dt><span class="section"><a href="upgrade0.92.html#d2475e2753">3.4.3. Distributed splitting is on by default
+</a></span></dt><dt><span class="section"><a href="upgrade0.92.html#d2475e2758">3.4.4. Memory accounting is different now
+</a></span></dt><dt><span class="section"><a href="upgrade0.92.html#d2475e2767">3.4.5. On the Hadoop version to use
+</a></span></dt><dt><span class="section"><a href="upgrade0.92.html#d2475e2779">3.4.6. HBase 0.92.0 ships with ZooKeeper 3.4.2
+</a></span></dt><dt><span class="section"><a href="upgrade0.92.html#d2475e2784">3.4.7. Online alter is off by default
+</a></span></dt><dt><span class="section"><a href="upgrade0.92.html#d2475e2791">3.4.8. WebUI
+</a></span></dt><dt><span class="section"><a href="upgrade0.92.html#d2475e2796">3.4.9. Security tarball
+</a></span></dt><dt><span class="section"><a href="upgrade0.92.html#d2475e2801">3.4.10. Experimental off-heap cache
+</a></span></dt><dt><span class="section"><a href="upgrade0.92.html#d2475e2806">3.4.11. Changes in HBase replication
+</a></span></dt><dt><span class="section"><a href="upgrade0.92.html#d2475e2811">3.4.12. RegionServer now aborts if OOME
+</a></span></dt><dt><span class="section"><a href="upgrade0.92.html#d2475e2816">3.4.13. HFile V2 and the “Bigger, Fewer” Tendency
+</a></span></dt></dl></dd><dt><span class="section"><a href="upgrade0.90.html">3.5. Upgrading to HBase 0.90.x from 0.20.x or 0.89.x</a></span></dt></dl></div><p>You cannot skip major verisons upgrading. If you are upgrading from
+ version 0.90.x to 0.94.x, you must first go from 0.90.x to 0.92.x and then go
+ from 0.92.x to 0.94.x.</p><p>
Review <a class="xref" href="configuration.html" title="Chapter 2. Apache HBase (TM) Configuration">Chapter 2, <i>Apache HBase (TM) Configuration</i></a>, in particular the section on Hadoop version.
- </p><div class="section" title="3.1. Upgrading from 0.94.x to 0.96.x"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="upgrade0.96"></a>3.1. Upgrading from 0.94.x to 0.96.x</h2></div><div><h3 class="subtitle">The Singularity</h3></div></div></div><p>You will have to stop your old 0.94 cluster completely to upgrade. If you are replicating
- between clusters, both clusters will have to go down to upgrade. Make sure it is a clean shutdown
- so there are no WAL files laying around (TODO: Can 0.96 read 0.94 WAL files?). Make sure
- zookeeper is cleared of state. All clients must be upgraded to 0.96 too.
- </p><p>The API has changed in a few areas; in particular how you use coprocessors (TODO: MapReduce too?)
- </p><p>TODO: Write about 3.4 zk ensemble and multi support</p></div></div><div id="disqus_thread"></div><script type="text/javascript">
+ </p><div class="section" title="3.1. HBase version numbers"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="hbase.versioning"></a>3.1. HBase version numbers</h2></div></div></div><p>HBase has not walked a straight line where version numbers are concerned.
+ Since we came up out of hadoop itself, we originally tracked hadoop versioning.
+ Later we left hadoop versioning behind because we were moving at a different rate
+ to that of our parent. If you are into the arcane, checkout our old wiki page
+ on <a class="link" href="http://wiki.apache.org/hadoop/Hbase/HBaseVersions" target="_top">HBase Versioning</a>
+ which tries to connect the HBase version dots.</p><div class="section" title="3.1.1. Odd/Even Versioning or "Development"" Series Releases"><div class="titlepage"><div><div><h3 class="title"><a name="hbase.development.series"></a>3.1.1. Odd/Even Versioning or "Development"" Series Releases</h3></div></div></div><p>Ahead of big releases, we have been putting up preview versions to start the
+ feedback cycle turning-over earlier. These "Development" Series releases,
+ always odd-numbered, come with no guarantees, not even regards being able
+ to upgrade between two sequential releases (we reserve the right to break compatibility across
+ "Development" Series releases). Needless to say, these releases are not for
+ production deploys. They are a preview of what is coming in the hope that
+ interested parties will take the release for a test drive and flag us early if we
+ there are issues we've missed ahead of our rolling a production-worthy release.
+ </p><p>Our first "Development" Series was the 0.89 set that came out ahead of
+ HBase 0.90.0. HBase 0.95 is another "Development" Series that portends
+ HBase 0.96.0.
+ </p></div></div></div><div id="disqus_thread"></div><script type="text/javascript">
var disqus_shortname = 'hbase'; // required: replace example with your forum shortname
var disqus_url = 'http://hbase.apache.org/book';
var disqus_identifier = 'upgrading';
@@ -33,4 +44,4 @@
dsq.src = 'http://' + disqus_shortname + '.disqus.com/embed.js';
(document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
})();
-</script><noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript">comments powered by Disqus.</a></noscript><a href="http://disqus.com" class="dsq-brlink">comments powered by <span class="logo-disqus">Disqus</span></a><div class="navfooter"><hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="important_configurations.html">Prev</a> </td><td width="20%" align="center"> </td><td width="40%" align="right"> <a accesskey="n" href="upgrade0.94.html">Next</a></td></tr><tr><td width="40%" align="left" valign="top">2.5. The Important Configurations </td><td width="20%" align="center"><a accesskey="h" href="book.html">Home</a></td><td width="40%" align="right" valign="top"> 3.2. Upgrading from 0.92.x to 0.94.x</td></tr></table></div></body></html>
\ No newline at end of file
+</script><noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript">comments powered by Disqus.</a></noscript><a href="http://disqus.com" class="dsq-brlink">comments powered by <span class="logo-disqus">Disqus</span></a><div class="navfooter"><hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="important_configurations.html">Prev</a> </td><td width="20%" align="center"> </td><td width="40%" align="right"> <a accesskey="n" href="upgrade0.96.html">Next</a></td></tr><tr><td width="40%" align="left" valign="top">2.5. The Important Configurations </td><td width="20%" align="center"><a accesskey="h" href="book.html">Home</a></td><td width="40%" align="right" valign="top"> 3.2. Upgrading from 0.94.x to 0.96.x</td></tr></table></div></body></html>
\ No newline at end of file
Modified: hbase/hbase.apache.org/trunk/book/versions.html
URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/versions.html?rev=1463652&r1=1463651&r2=1463652&view=diff
==============================================================================
--- hbase/hbase.apache.org/trunk/book/versions.html (original)
+++ hbase/hbase.apache.org/trunk/book/versions.html Tue Apr 2 18:06:19 2013
@@ -1,6 +1,6 @@
<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
- <title>5.8. Versions</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="datamodel.html" title="Chapter 5. Data Model"><link rel="prev" href="data_model_operations.html" title="5.7. Data Model Operations"><link rel="next" href="dm.sort.html" title="5.9. Sort Order"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">5.8. Versions</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="data_model_operations.html">Prev</a> </td><th width="60%" align="center">Chapter 5. Data Model</th><td width="20%" align="right"> <a accesskey="n" href="dm.sort.html">Next</a></td></tr></table><hr><
/div><div class="section" title="5.8. Versions"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="versions"></a>5.8. Versions<a class="indexterm" name="d2279e3378"></a></h2></div></div></div><p>A <span class="emphasis"><em>{row, column, version} </em></span>tuple exactly
+ <title>5.8. Versions</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="datamodel.html" title="Chapter 5. Data Model"><link rel="prev" href="data_model_operations.html" title="5.7. Data Model Operations"><link rel="next" href="dm.sort.html" title="5.9. Sort Order"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">5.8. Versions</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="data_model_operations.html">Prev</a> </td><th width="60%" align="center">Chapter 5. Data Model</th><td width="20%" align="right"> <a accesskey="n" href="dm.sort.html">Next</a></td></tr></table><hr><
/div><div class="section" title="5.8. Versions"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="versions"></a>5.8. Versions<a class="indexterm" name="d2475e3382"></a></h2></div></div></div><p>A <span class="emphasis"><em>{row, column, version} </em></span>tuple exactly
specifies a <code class="literal">cell</code> in HBase. It's possible to have an
unbounded number of cells where the row and column are the same but the
cell address differs only in its version dimension.</p><p>While rows and column keys are expressed as bytes, the version is
@@ -14,10 +14,10 @@
first.</p><p>There is a lot of confusion over the semantics of
<code class="literal">cell</code> versions, in HBase. In particular, a couple
questions that often come up are:</p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem"><p>If multiple writes to a cell have the same version, are all
- versions maintained or just the last?<sup>[<a name="d2279e3411" href="#ftn.d2279e3411" class="footnote">13</a>]</sup></p></li><li class="listitem"><p>Is it OK to write cells in a non-increasing version
- order?<sup>[<a name="d2279e3417" href="#ftn.d2279e3417" class="footnote">14</a>]</sup></p></li></ul></div><p>Below we describe how the version dimension in HBase currently
- works<sup>[<a name="d2279e3422" href="#ftn.d2279e3422" class="footnote">15</a>]</sup>.</p><div class="section" title="5.8.1. Versions and HBase Operations"><div class="titlepage"><div><div><h3 class="title"><a name="versions.ops"></a>5.8.1. Versions and HBase Operations</h3></div></div></div><p>In this section we look at the behavior of the version dimension
- for each of the core HBase operations.</p><div class="section" title="5.8.1.1. Get/Scan"><div class="titlepage"><div><div><h4 class="title"><a name="d2279e3440"></a>5.8.1.1. Get/Scan</h4></div></div></div><p>Gets are implemented on top of Scans. The below discussion of
+ versions maintained or just the last?<sup>[<a name="d2475e3415" href="#ftn.d2475e3415" class="footnote">13</a>]</sup></p></li><li class="listitem"><p>Is it OK to write cells in a non-increasing version
+ order?<sup>[<a name="d2475e3421" href="#ftn.d2475e3421" class="footnote">14</a>]</sup></p></li></ul></div><p>Below we describe how the version dimension in HBase currently
+ works<sup>[<a name="d2475e3426" href="#ftn.d2475e3426" class="footnote">15</a>]</sup>.</p><div class="section" title="5.8.1. Versions and HBase Operations"><div class="titlepage"><div><div><h3 class="title"><a name="versions.ops"></a>5.8.1. Versions and HBase Operations</h3></div></div></div><p>In this section we look at the behavior of the version dimension
+ for each of the core HBase operations.</p><div class="section" title="5.8.1.1. Get/Scan"><div class="titlepage"><div><div><h4 class="title"><a name="d2475e3444"></a>5.8.1.1. Get/Scan</h4></div></div></div><p>Gets are implemented on top of Scans. The below discussion of
<a class="link" href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Get.html" target="_top">Get</a> applies equally to <a class="link" href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Scan.html" target="_top">Scans</a>.</p><p>By default, i.e. if you specify no explicit version, when
doing a <code class="literal">get</code>, the cell whose version has the
largest value is returned (which may or may not be the latest one
@@ -45,7 +45,7 @@ Result r = htable.get(get);
byte[] b = r.getValue(CF, ATTR); // returns current version of value
List<KeyValue> kv = r.getColumn(CF, ATTR); // returns all versions of this column
</pre><p>
- </p></div><div class="section" title="5.8.1.4. Put"><div class="titlepage"><div><div><h4 class="title"><a name="d2279e3485"></a>5.8.1.4. Put</h4></div></div></div><p>Doing a put always creates a new version of a
+ </p></div><div class="section" title="5.8.1.4. Put"><div class="titlepage"><div><div><h4 class="title"><a name="d2475e3489"></a>5.8.1.4. Put</h4></div></div></div><p>Doing a put always creates a new version of a
<code class="literal">cell</code>, at a certain timestamp. By default the
system uses the server's <code class="literal">currentTimeMillis</code>, but
you can specify the version (= the long integer) yourself, on a
@@ -75,7 +75,7 @@ htable.put(put);
It's usually best to avoid setting this timestamp yourself. Prefer using a separate
timestamp attribute of the row, or have the timestamp a part of the rowkey, or both.
</p></div></div><div class="section" title="5.8.1.5. Delete"><div class="titlepage"><div><div><h4 class="title"><a name="version.delete"></a>5.8.1.5. Delete</h4></div></div></div><p>There are three different types of internal delete markers
- <sup>[<a name="d2279e3519" href="#ftn.d2279e3519" class="footnote">16</a>]</sup>:
+ <sup>[<a name="d2475e3523" href="#ftn.d2475e3523" class="footnote">16</a>]</sup>:
</p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem"><p>Delete: for a specific version of a column.</p></li><li class="listitem"><p>Delete column: for all versions of a column.</p></li><li class="listitem"><p>Delete family: for all columns of a particular ColumnFamily</p></li></ul></div><p>
When deleting an entire row, HBase will internally create a tombstone for each ColumnFamily (i.e., not each individual column).
</p><p>Deletes work by creating <span class="emphasis"><em>tombstone</em></span>
@@ -87,13 +87,13 @@ htable.put(put);
example a delete will not immediately delete (or mark as deleted)
the entries in the storage file that correspond to the delete
condition. Rather, a so-called <span class="emphasis"><em>tombstone</em></span> is
- written, which will mask the deleted values<sup>[<a name="d2279e3550" href="#ftn.d2279e3550" class="footnote">17</a>]</sup>. If the version you specified when deleting a row is
+ written, which will mask the deleted values<sup>[<a name="d2475e3554" href="#ftn.d2475e3554" class="footnote">17</a>]</sup>. If the version you specified when deleting a row is
larger than the version of any value in the row, then you can
consider the complete row to be deleted.</p><p>For an informative discussion on how deletes and versioning interact, see
the thread <a class="link" href="http://comments.gmane.org/gmane.comp.java.hadoop.hbase.user/28421" target="_top">Put w/ timestamp -> Deleteall -> Put w/ timestamp fails</a>
- up on the user mailing list.</p><p>Also see <a class="xref" href="regions.arch.html#keyvalue" title="9.7.5.4. KeyValue">Section 9.7.5.4, “KeyValue”</a> for more information on the internal KeyValue format.
- </p></div></div><div class="section" title="5.8.2. Current Limitations"><div class="titlepage"><div><div><h3 class="title"><a name="d2279e3563"></a>5.8.2. Current Limitations</h3></div></div></div><div class="section" title="5.8.2.1. Deletes mask Puts"><div class="titlepage"><div><div><h4 class="title"><a name="d2279e3566"></a>5.8.2.1. Deletes mask Puts</h4></div></div></div><p>Deletes mask puts, even puts that happened after the delete
- was entered<sup>[<a name="d2279e3571" href="#ftn.d2279e3571" class="footnote">18</a>]</sup>. Remember that a delete writes a tombstone, which only
+ up on the user mailing list.</p><p>Also see <a class="xref" href="regions.arch.html#keyvalue" title="9.7.6.4. KeyValue">Section 9.7.6.4, “KeyValue”</a> for more information on the internal KeyValue format.
+ </p></div></div><div class="section" title="5.8.2. Current Limitations"><div class="titlepage"><div><div><h3 class="title"><a name="d2475e3567"></a>5.8.2. Current Limitations</h3></div></div></div><div class="section" title="5.8.2.1. Deletes mask Puts"><div class="titlepage"><div><div><h4 class="title"><a name="d2475e3570"></a>5.8.2.1. Deletes mask Puts</h4></div></div></div><p>Deletes mask puts, even puts that happened after the delete
+ was entered<sup>[<a name="d2475e3575" href="#ftn.d2475e3575" class="footnote">18</a>]</sup>. Remember that a delete writes a tombstone, which only
disappears after then next major compaction has run. Suppose you do
a delete of everything <= T. After this you do a new put with a
timestamp <= T. This put, even if it happened after the delete,
@@ -104,22 +104,22 @@ htable.put(put);
always-increasing versions for new puts to a row. But they can occur
even if you do not care about time: just do delete and put
immediately after each other, and there is some chance they happen
- within the same millisecond.</p></div><div class="section" title="5.8.2.2. Major compactions change query results"><div class="titlepage"><div><div><h4 class="title"><a name="d2279e3576"></a>5.8.2.2. Major compactions change query results</h4></div></div></div><p><span class="quote">“<span class="quote">...create three cell versions at t1, t2 and t3, with a
+ within the same millisecond.</p></div><div class="section" title="5.8.2.2. Major compactions change query results"><div class="titlepage"><div><div><h4 class="title"><a name="d2475e3580"></a>5.8.2.2. Major compactions change query results</h4></div></div></div><p><span class="quote">“<span class="quote">...create three cell versions at t1, t2 and t3, with a
maximum-versions setting of 2. So when getting all versions, only
the values at t2 and t3 will be returned. But if you delete the
version at t2 or t3, the one at t1 will appear again. Obviously,
once a major compaction has run, such behavior will not be the case
- anymore...<sup>[<a name="d2279e3582" href="#ftn.d2279e3582" class="footnote">19</a>]</sup></span>”</span></p></div></div><div class="footnotes"><br><hr width="100" align="left"><div class="footnote"><p><sup>[<a id="ftn.d2279e3411" href="#d2279e3411" class="para">13</a>] </sup>Currently, only the last written is fetchable.</p></div><div class="footnote"><p><sup>[<a id="ftn.d2279e3417" href="#d2279e3417" class="para">14</a>] </sup>Yes</p></div><div class="footnote"><p><sup>[<a id="ftn.d2279e3422" href="#d2279e3422" class="para">15</a>] </sup>See <a class="link" href="https://issues.apache.org/jira/browse/HBASE-2406" target="_top">HBASE-2406</a>
+ anymore...<sup>[<a name="d2475e3586" href="#ftn.d2475e3586" class="footnote">19</a>]</sup></span>”</span></p></div></div><div class="footnotes"><br><hr width="100" align="left"><div class="footnote"><p><sup>[<a id="ftn.d2475e3415" href="#d2475e3415" class="para">13</a>] </sup>Currently, only the last written is fetchable.</p></div><div class="footnote"><p><sup>[<a id="ftn.d2475e3421" href="#d2475e3421" class="para">14</a>] </sup>Yes</p></div><div class="footnote"><p><sup>[<a id="ftn.d2475e3426" href="#d2475e3426" class="para">15</a>] </sup>See <a class="link" href="https://issues.apache.org/jira/browse/HBASE-2406" target="_top">HBASE-2406</a>
for discussion of HBase versions. <a class="link" href="http://outerthought.org/blog/417-ot.html" target="_top">Bending time
in HBase</a> makes for a good read on the version, or time,
dimension in HBase. It has more detail on versioning than is
provided here. As of this writing, the limiitation
<span class="emphasis"><em>Overwriting values at existing timestamps</em></span>
mentioned in the article no longer holds in HBase. This section is
- basically a synopsis of this article by Bruno Dumon.</p></div><div class="footnote"><p><sup>[<a id="ftn.d2279e3519" href="#d2279e3519" class="para">16</a>] </sup>See Lars Hofhansl's blog for discussion of his attempt
- adding another, <a class="link" href="http://hadoop-hbase.blogspot.com/2012/01/scanning-in-hbase.html" target="_top">Scanning in HBase: Prefix Delete Marker</a></p></div><div class="footnote"><p><sup>[<a id="ftn.d2279e3550" href="#d2279e3550" class="para">17</a>] </sup>When HBase does a major compaction, the tombstones are
+ basically a synopsis of this article by Bruno Dumon.</p></div><div class="footnote"><p><sup>[<a id="ftn.d2475e3523" href="#d2475e3523" class="para">16</a>] </sup>See Lars Hofhansl's blog for discussion of his attempt
+ adding another, <a class="link" href="http://hadoop-hbase.blogspot.com/2012/01/scanning-in-hbase.html" target="_top">Scanning in HBase: Prefix Delete Marker</a></p></div><div class="footnote"><p><sup>[<a id="ftn.d2475e3554" href="#d2475e3554" class="para">17</a>] </sup>When HBase does a major compaction, the tombstones are
processed to actually remove the dead values, together with the
- tombstones themselves.</p></div><div class="footnote"><p><sup>[<a id="ftn.d2279e3571" href="#d2279e3571" class="para">18</a>] </sup><a class="link" href="https://issues.apache.org/jira/browse/HBASE-2256" target="_top">HBASE-2256</a></p></div><div class="footnote"><p><sup>[<a id="ftn.d2279e3582" href="#d2279e3582" class="para">19</a>] </sup>See <span class="emphasis"><em>Garbage Collection</em></span> in <a class="link" href="http://outerthought.org/blog/417-ot.html" target="_top">Bending
+ tombstones themselves.</p></div><div class="footnote"><p><sup>[<a id="ftn.d2475e3575" href="#d2475e3575" class="para">18</a>] </sup><a class="link" href="https://issues.apache.org/jira/browse/HBASE-2256" target="_top">HBASE-2256</a></p></div><div class="footnote"><p><sup>[<a id="ftn.d2475e3586" href="#d2475e3586" class="para">19</a>] </sup>See <span class="emphasis"><em>Garbage Collection</em></span> in <a class="link" href="http://outerthought.org/blog/417-ot.html" target="_top">Bending
time in HBase</a> </p></div></div></div><div id="disqus_thread"></div><script type="text/javascript">
var disqus_shortname = 'hbase'; // required: replace example with your forum shortname
var disqus_url = 'http://hbase.apache.org/book';
Modified: hbase/hbase.apache.org/trunk/book/zk.sasl.auth.html
URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/zk.sasl.auth.html?rev=1463652&r1=1463651&r2=1463652&view=diff
==============================================================================
--- hbase/hbase.apache.org/trunk/book/zk.sasl.auth.html (original)
+++ hbase/hbase.apache.org/trunk/book/zk.sasl.auth.html Tue Apr 2 18:06:19 2013
@@ -14,7 +14,7 @@
configuration). It's recommended to begin with an
HBase-managed Zookeeper configuration (as opposed to a
standalone Zookeeper quorum) for ease of learning.
- </p><div class="section" title="16.2.1. Operating System Prerequisites"><div class="titlepage"><div><div><h3 class="title"><a name="d2279e11147"></a>16.2.1. Operating System Prerequisites</h3></div></div></div></div><p>
+ </p><div class="section" title="16.2.1. Operating System Prerequisites"><div class="titlepage"><div><div><h3 class="title"><a name="d2475e11402"></a>16.2.1. Operating System Prerequisites</h3></div></div></div></div><p>
You need to have a working Kerberos KDC setup. For
each <code class="code">$HOST</code> that will run a ZooKeeper
server, you should have a principle
@@ -65,7 +65,7 @@
useTicketCache=true;
};
</pre><p>We'll refer to this JAAS configuration file as
- <code class="filename">$CLIENT_CONF</code> below.</p><div class="section" title="16.2.2. HBase-managed Zookeeper Configuration"><div class="titlepage"><div><div><h3 class="title"><a name="d2279e11228"></a>16.2.2. HBase-managed Zookeeper Configuration</h3></div></div></div><p>On each node that will run a zookeeper, a
+ <code class="filename">$CLIENT_CONF</code> below.</p><div class="section" title="16.2.2. HBase-managed Zookeeper Configuration"><div class="titlepage"><div><div><h3 class="title"><a name="d2475e11483"></a>16.2.2. HBase-managed Zookeeper Configuration</h3></div></div></div><p>On each node that will run a zookeeper, a
master, or a regionserver, create a <a class="link" href="http://docs.oracle.com/javase/1.4.2/docs/guide/security/jgss/tutorials/LoginConfigFile.html" target="_top">JAAS</a>
configuration file in the conf directory of the node's
<code class="filename">HBASE_HOME</code> directory that looks like the
@@ -148,7 +148,7 @@
bin/hbase zookeeper start
bin/hbase master start
bin/hbase regionserver start
- </pre></div><div class="section" title="16.2.3. External Zookeeper Configuration"><div class="titlepage"><div><div><h3 class="title"><a name="d2279e11303"></a>16.2.3. External Zookeeper Configuration</h3></div></div></div><p>Add a JAAS configuration file that looks like:
+ </pre></div><div class="section" title="16.2.3. External Zookeeper Configuration"><div class="titlepage"><div><div><h3 class="title"><a name="d2475e11558"></a>16.2.3. External Zookeeper Configuration</h3></div></div></div><p>Add a JAAS configuration file that looks like:
</p><pre class="programlisting">
Client {
@@ -221,7 +221,7 @@
</p><pre class="programlisting">
bin/hbase master start
bin/hbase regionserver start
- </pre></div><div class="section" title="16.2.4. Zookeeper Server Authentication Log Output"><div class="titlepage"><div><div><h3 class="title"><a name="d2279e11362"></a>16.2.4. Zookeeper Server Authentication Log Output</h3></div></div></div><p>If the configuration above is successful,
+ </pre></div><div class="section" title="16.2.4. Zookeeper Server Authentication Log Output"><div class="titlepage"><div><div><h3 class="title"><a name="d2475e11617"></a>16.2.4. Zookeeper Server Authentication Log Output</h3></div></div></div><p>If the configuration above is successful,
you should see something similar to the following in
your Zookeeper server logs:
</p><pre class="programlisting">
@@ -239,7 +239,7 @@
11/12/05 22:43:59 INFO server.ZooKeeperServer: adding SASL authorization for authorizationID: hbase
</pre><p>
- </p></div><div class="section" title="16.2.5. Zookeeper Client Authentication Log Output"><div class="titlepage"><div><div><h3 class="title"><a name="d2279e11370"></a>16.2.5. Zookeeper Client Authentication Log Output</h3></div></div></div><p>On the Zookeeper client side (HBase master or regionserver),
+ </p></div><div class="section" title="16.2.5. Zookeeper Client Authentication Log Output"><div class="titlepage"><div><div><h3 class="title"><a name="d2475e11625"></a>16.2.5. Zookeeper Client Authentication Log Output</h3></div></div></div><p>On the Zookeeper client side (HBase master or regionserver),
you should see something similar to the following:
</p><pre class="programlisting">
@@ -255,7 +255,7 @@
11/12/05 22:43:59 INFO zookeeper.Login: TGT refresh sleeping until: Tue Dec 06 18:30:37 UTC 2011
11/12/05 22:43:59 INFO zookeeper.ClientCnxn: Session establishment complete on server ip-10-166-175-249.us-west-1.compute.internal/10.166.175.249:2181, sessionid = 0x134106594320000, negotiated timeout = 180000
</pre><p>
- </p></div><div class="section" title="16.2.6. Configuration from Scratch"><div class="titlepage"><div><div><h3 class="title"><a name="d2279e11378"></a>16.2.6. Configuration from Scratch</h3></div></div></div>
+ </p></div><div class="section" title="16.2.6. Configuration from Scratch"><div class="titlepage"><div><div><h3 class="title"><a name="d2475e11633"></a>16.2.6. Configuration from Scratch</h3></div></div></div>
This has been tested on the current standard Amazon
Linux AMI. First setup KDC and principals as
@@ -275,7 +275,7 @@
bin/hbase zookeeper &
bin/hbase master &
bin/hbase regionserver &
- </pre></div><div class="section" title="16.2.7. Future improvements"><div class="titlepage"><div><div><h3 class="title"><a name="d2279e11387"></a>16.2.7. Future improvements</h3></div></div></div><div class="section" title="16.2.7.1. Fix target/cached_classpath.txt"><div class="titlepage"><div><div><h4 class="title"><a name="d2279e11390"></a>16.2.7.1. Fix target/cached_classpath.txt</h4></div></div></div><p>
+ </pre></div><div class="section" title="16.2.7. Future improvements"><div class="titlepage"><div><div><h3 class="title"><a name="d2475e11642"></a>16.2.7. Future improvements</h3></div></div></div><div class="section" title="16.2.7.1. Fix target/cached_classpath.txt"><div class="titlepage"><div><div><h4 class="title"><a name="d2475e11645"></a>16.2.7.1. Fix target/cached_classpath.txt</h4></div></div></div><p>
You must override the standard hadoop-core jar file from the
<code class="code">target/cached_classpath.txt</code>
file with the version containing the HADOOP-7070 fix. You can use the following script to do this:
@@ -285,13 +285,13 @@
mv target/tmp.txt target/cached_classpath.txt
</pre><p>
- </p></div><div class="section" title="16.2.7.2. Set JAAS configuration programmatically"><div class="titlepage"><div><div><h4 class="title"><a name="d2279e11401"></a>16.2.7.2. Set JAAS configuration
+ </p></div><div class="section" title="16.2.7.2. Set JAAS configuration programmatically"><div class="titlepage"><div><div><h4 class="title"><a name="d2475e11656"></a>16.2.7.2. Set JAAS configuration
programmatically</h4></div></div></div>
This would avoid the need for a separate Hadoop jar
that fixes <a class="link" href="https://issues.apache.org/jira/browse/HADOOP-7070" target="_top">HADOOP-7070</a>.
- </div><div class="section" title="16.2.7.3. Elimination of kerberos.removeHostFromPrincipal and kerberos.removeRealmFromPrincipal"><div class="titlepage"><div><div><h4 class="title"><a name="d2279e11408"></a>16.2.7.3. Elimination of
+ </div><div class="section" title="16.2.7.3. Elimination of kerberos.removeHostFromPrincipal and kerberos.removeRealmFromPrincipal"><div class="titlepage"><div><div><h4 class="title"><a name="d2475e11663"></a>16.2.7.3. Elimination of
<code class="code">kerberos.removeHostFromPrincipal</code> and
<code class="code">kerberos.removeRealmFromPrincipal</code></h4></div></div></div></div></div></div><div id="disqus_thread"></div><script type="text/javascript">
var disqus_shortname = 'hbase'; // required: replace example with your forum shortname
Modified: hbase/hbase.apache.org/trunk/book/zookeeper.html
URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/book/zookeeper.html?rev=1463652&r1=1463651&r2=1463652&view=diff
==============================================================================
--- hbase/hbase.apache.org/trunk/book/zookeeper.html (original)
+++ hbase/hbase.apache.org/trunk/book/zookeeper.html Tue Apr 2 18:06:19 2013
@@ -1,6 +1,6 @@
<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
- <title>Chapter 16. ZooKeeper</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="prev" href="submitting.patches.html" title="15.11. Submitting Patches"><link rel="next" href="zk.sasl.auth.html" title="16.2. SASL Authentication with ZooKeeper"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">Chapter 16. ZooKeeper</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="submitting.patches.html">Prev</a> </td><th width="60%" align="center"> </th><td width="20%" align="right"> <a accesskey="n" href="zk.sasl.auth.html">
Next</a></td></tr></table><hr></div><div class="chapter" title="Chapter 16. ZooKeeper"><div class="titlepage"><div><div><h2 class="title"><a name="zookeeper"></a>Chapter 16. ZooKeeper<a class="indexterm" name="d2279e10971"></a></h2></div></div></div><div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="section"><a href="zookeeper.html#d2279e11082">16.1. Using existing ZooKeeper ensemble</a></span></dt><dt><span class="section"><a href="zk.sasl.auth.html">16.2. SASL Authentication with ZooKeeper</a></span></dt><dd><dl><dt><span class="section"><a href="zk.sasl.auth.html#d2279e11147">16.2.1. Operating System Prerequisites</a></span></dt><dt><span class="section"><a href="zk.sasl.auth.html#d2279e11228">16.2.2. HBase-managed Zookeeper Configuration</a></span></dt><dt><span class="section"><a href="zk.sasl.auth.html#d2279e11303">16.2.3. External Zookeeper Configuration</a></span></dt><dt><span class="section"><a href="zk.sasl.auth.html#d2279e113
62">16.2.4. Zookeeper Server Authentication Log Output</a></span></dt><dt><span class="section"><a href="zk.sasl.auth.html#d2279e11370">16.2.5. Zookeeper Client Authentication Log Output</a></span></dt><dt><span class="section"><a href="zk.sasl.auth.html#d2279e11378">16.2.6. Configuration from Scratch</a></span></dt><dt><span class="section"><a href="zk.sasl.auth.html#d2279e11387">16.2.7. Future improvements</a></span></dt></dl></dd></dl></div><p>A distributed Apache HBase (TM) installation depends on a running ZooKeeper cluster.
+ <title>Chapter 16. ZooKeeper</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="up" href="book.html" title="The Apache HBase™ Reference Guide"><link rel="prev" href="submitting.patches.html" title="15.11. Submitting Patches"><link rel="next" href="zk.sasl.auth.html" title="16.2. SASL Authentication with ZooKeeper"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">Chapter 16. ZooKeeper</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="submitting.patches.html">Prev</a> </td><th width="60%" align="center"> </th><td width="20%" align="right"> <a accesskey="n" href="zk.sasl.auth.html">
Next</a></td></tr></table><hr></div><div class="chapter" title="Chapter 16. ZooKeeper"><div class="titlepage"><div><div><h2 class="title"><a name="zookeeper"></a>Chapter 16. ZooKeeper<a class="indexterm" name="d2475e11226"></a></h2></div></div></div><div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="section"><a href="zookeeper.html#d2475e11337">16.1. Using existing ZooKeeper ensemble</a></span></dt><dt><span class="section"><a href="zk.sasl.auth.html">16.2. SASL Authentication with ZooKeeper</a></span></dt><dd><dl><dt><span class="section"><a href="zk.sasl.auth.html#d2475e11402">16.2.1. Operating System Prerequisites</a></span></dt><dt><span class="section"><a href="zk.sasl.auth.html#d2475e11483">16.2.2. HBase-managed Zookeeper Configuration</a></span></dt><dt><span class="section"><a href="zk.sasl.auth.html#d2475e11558">16.2.3. External Zookeeper Configuration</a></span></dt><dt><span class="section"><a href="zk.sasl.auth.html#d2475e116
17">16.2.4. Zookeeper Server Authentication Log Output</a></span></dt><dt><span class="section"><a href="zk.sasl.auth.html#d2475e11625">16.2.5. Zookeeper Client Authentication Log Output</a></span></dt><dt><span class="section"><a href="zk.sasl.auth.html#d2475e11633">16.2.6. Configuration from Scratch</a></span></dt><dt><span class="section"><a href="zk.sasl.auth.html#d2475e11642">16.2.7. Future improvements</a></span></dt></dl></dd></dl></div><p>A distributed Apache HBase (TM) installation depends on a running ZooKeeper cluster.
All participating nodes and clients need to be able to access the
running ZooKeeper ensemble. Apache HBase by default manages a ZooKeeper
"cluster" for you. It will start and stop the ZooKeeper ensemble
@@ -25,7 +25,7 @@
<code class="varname">hbase.zookeeper.property.clientPort</code> property.
For all default values used by HBase, including ZooKeeper
configuration, see <a class="xref" href="config.files.html#hbase_default_configurations" title="2.3.1.1. HBase Default Configuration">Section 2.3.1.1, “HBase Default Configuration”</a>. Look for the
- <code class="varname">hbase.zookeeper.property</code> prefix <sup>[<a name="d2279e11010" href="#ftn.d2279e11010" class="footnote">30</a>]</sup></p><p>You must at least list the ensemble servers in
+ <code class="varname">hbase.zookeeper.property</code> prefix <sup>[<a name="d2475e11265" href="#ftn.d2475e11265" class="footnote">30</a>]</sup></p><p>You must at least list the ensemble servers in
<code class="filename">hbase-site.xml</code> using the
<code class="varname">hbase.zookeeper.quorum</code> property. This property
defaults to a single ensemble member at
@@ -93,7 +93,7 @@
have 'interesting' problems a couple of months in; i.e. zookeeper could start
dropping sessions if it has to run through a directory of hundreds of thousands of
logs which is wont to do around leader reelection time -- a process rare but run on
- occasion whether because a machine is dropped or happens to hiccup.</p></div><div class="section" title="16.1. Using existing ZooKeeper ensemble"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="d2279e11082"></a>16.1. Using existing ZooKeeper ensemble</h2></div></div></div><p>To point HBase at an existing ZooKeeper cluster, one that
+ occasion whether because a machine is dropped or happens to hiccup.</p></div><div class="section" title="16.1. Using existing ZooKeeper ensemble"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="d2475e11337"></a>16.1. Using existing ZooKeeper ensemble</h2></div></div></div><p>To point HBase at an existing ZooKeeper cluster, one that
is not managed by HBase, set <code class="varname">HBASE_MANAGES_ZK</code>
in <code class="filename">conf/hbase-env.sh</code> to false
</p><pre class="programlisting">
@@ -119,7 +119,7 @@ ${HBASE_HOME}/bin/hbase-daemons.sh {star
Started Guide</a>. Additionally, see the <a class="link" href="http://wiki.apache.org/hadoop/ZooKeeper/FAQ#A7" target="_top">ZooKeeper Wiki</a> or the
<a class="link" href="http://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin.html#sc_zkMulitServerSetup" target="_top">ZooKeeper documentation</a>
for more information on ZooKeeper sizing.
- </p></div><div class="footnotes"><br><hr width="100" align="left"><div class="footnote"><p><sup>[<a id="ftn.d2279e11010" href="#d2279e11010" class="para">30</a>] </sup>For the full list of ZooKeeper configurations, see
+ </p></div><div class="footnotes"><br><hr width="100" align="left"><div class="footnote"><p><sup>[<a id="ftn.d2475e11265" href="#d2475e11265" class="para">30</a>] </sup>For the full list of ZooKeeper configurations, see
ZooKeeper's <code class="filename">zoo.cfg</code>. HBase does not ship
with a <code class="filename">zoo.cfg</code> so you will need to browse
the <code class="filename">conf</code> directory in an appropriate
Modified: hbase/hbase.apache.org/trunk/bulk-loads.html
URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/bulk-loads.html?rev=1463652&r1=1463651&r2=1463652&view=diff
==============================================================================
--- hbase/hbase.apache.org/trunk/bulk-loads.html (original)
+++ hbase/hbase.apache.org/trunk/bulk-loads.html Tue Apr 2 18:06:19 2013
@@ -1,10 +1,10 @@
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
-<!-- Generated by Apache Maven Doxia at Mar 29, 2013 -->
+<!-- Generated by Apache Maven Doxia at Apr 2, 2013 -->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
- <title>HBase -
+ <title>HBase - Assembly -
Bulk Loads in Apache HBase (TM)
</title>
@@ -15,7 +15,7 @@
</style>
<link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
<link rel="shortcut icon" href="/images/favicon.ico" />
- <meta name="Date-Revision-yyyymmdd" content="20130329" />
+ <meta name="Date-Revision-yyyymmdd" content="20130402" />
<meta http-equiv="Content-Language" content="en" />
<!--Google Analytics-->
<script type="text/javascript">
@@ -34,7 +34,7 @@
</head>
<body class="composite">
<div id="banner">
- <a href="./" id="bannerLeft">
+ <a href="../" id="bannerLeft">
<img src="images/hbase_logo.png" alt="Apache HBase" />
</a>
<!-- Commented out since we do not use it. St.Ack 20110906
@@ -86,6 +86,12 @@
<a href="https://reviews.apache.org" class="externalLink" title="ReviewBoard">ReviewBoard</a>
</li>
<li class="none">
+ <a href="builds.apache.org" title="Apache Build Box">Apache Build Box</a>
+ </li>
+ <li class="none">
+ <a href="http://54.241.6.143/" class="externalLink" title="EC2 Build Box">EC2 Build Box</a>
+ </li>
+ <li class="none">
<a href="team-list.html" title="Team">Team</a>
</li>
<li class="none">
@@ -202,8 +208,8 @@
<div id="footer">
<div class="xright">
- <span id="publishDate">Last Published: 2013-03-29</span>
- | <span id="projectVersion">Version: 0.97-SNAPSHOT</span>
+ <span id="publishDate">Last Published: 2013-04-02</span>
+ | <span id="projectVersion">Version: 0.97.0-SNAPSHOT</span>
</div>
<div class="xright">Copyright ©<a href="http://www.apache.org"> 2013
Modified: hbase/hbase.apache.org/trunk/case_studies.html
URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/case_studies.html?rev=1463652&r1=1463651&r2=1463652&view=diff
==============================================================================
--- hbase/hbase.apache.org/trunk/case_studies.html (original)
+++ hbase/hbase.apache.org/trunk/case_studies.html Tue Apr 2 18:06:19 2013
@@ -110,34 +110,34 @@ disadvantages!) at extremely high throug
really know that would be to try it out. If you don't have time to
build it both ways and compare, my advice would be to start with the
simplest option (one row per user+value). Start simple and iterate! :)
- </p></div></div><div class="section" title="1.3. Performance/Troubleshooting"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="casestudies.perftroub"></a>1.3. Performance/Troubleshooting</h2></div></div></div><div class="section" title="1.3.1. Case Study #1 (Performance Issue On A Single Node)"><div class="titlepage"><div><div><h3 class="title"><a name="casestudies.slownode"></a>1.3.1. Case Study #1 (Performance Issue On A Single Node)</h3></div></div></div><div class="section" title="1.3.1.1. Scenario"><div class="titlepage"><div><div><h4 class="title"><a name="d2207e82"></a>1.3.1.1. Scenario</h4></div></div></div><p>Following a scheduled reboot, one data node began exhibiting unusual behavior. Routine MapReduce
+ </p></div></div><div class="section" title="1.3. Performance/Troubleshooting"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="casestudies.perftroub"></a>1.3. Performance/Troubleshooting</h2></div></div></div><div class="section" title="1.3.1. Case Study #1 (Performance Issue On A Single Node)"><div class="titlepage"><div><div><h3 class="title"><a name="casestudies.slownode"></a>1.3.1. Case Study #1 (Performance Issue On A Single Node)</h3></div></div></div><div class="section" title="1.3.1.1. Scenario"><div class="titlepage"><div><div><h4 class="title"><a name="d2403e82"></a>1.3.1.1. Scenario</h4></div></div></div><p>Following a scheduled reboot, one data node began exhibiting unusual behavior. Routine MapReduce
jobs run against HBase tables which regularly completed in five or six minutes began taking 30 or 40 minutes
to finish. These jobs were consistently found to be waiting on map and reduce tasks assigned to the troubled data node
(e.g., the slow map tasks all had the same Input Split).
The situation came to a head during a distributed copy, when the copy was severely prolonged by the lagging node.
- </p></div><div class="section" title="1.3.1.2. Hardware"><div class="titlepage"><div><div><h4 class="title"><a name="d2207e87"></a>1.3.1.2. Hardware</h4></div></div></div><p>Datanodes:
+ </p></div><div class="section" title="1.3.1.2. Hardware"><div class="titlepage"><div><div><h4 class="title"><a name="d2403e87"></a>1.3.1.2. Hardware</h4></div></div></div><p>Datanodes:
</p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem">Two 12-core processors</li><li class="listitem">Six Enerprise SATA disks</li><li class="listitem">24GB of RAM</li><li class="listitem">Two bonded gigabit NICs</li></ul></div><p>
</p><p>Network:
</p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem">10 Gigabit top-of-rack switches</li><li class="listitem">20 Gigabit bonded interconnects between racks.</li></ul></div><p>
- </p></div><div class="section" title="1.3.1.3. Hypotheses"><div class="titlepage"><div><div><h4 class="title"><a name="d2207e110"></a>1.3.1.3. Hypotheses</h4></div></div></div><div class="section" title="1.3.1.3.1. HBase "Hot Spot" Region"><div class="titlepage"><div><div><h5 class="title"><a name="d2207e113"></a>1.3.1.3.1. HBase "Hot Spot" Region</h5></div></div></div><p>We hypothesized that we were experiencing a familiar point of pain: a "hot spot" region in an HBase table,
+ </p></div><div class="section" title="1.3.1.3. Hypotheses"><div class="titlepage"><div><div><h4 class="title"><a name="d2403e110"></a>1.3.1.3. Hypotheses</h4></div></div></div><div class="section" title="1.3.1.3.1. HBase "Hot Spot" Region"><div class="titlepage"><div><div><h5 class="title"><a name="d2403e113"></a>1.3.1.3.1. HBase "Hot Spot" Region</h5></div></div></div><p>We hypothesized that we were experiencing a familiar point of pain: a "hot spot" region in an HBase table,
where uneven key-space distribution can funnel a huge number of requests to a single HBase region, bombarding the RegionServer
process and cause slow response time. Examination of the HBase Master status page showed that the number of HBase requests to the
troubled node was almost zero. Further, examination of the HBase logs showed that there were no region splits, compactions, or other region transitions
in progress. This effectively ruled out a "hot spot" as the root cause of the observed slowness.
- </p></div><div class="section" title="1.3.1.3.2. HBase Region With Non-Local Data"><div class="titlepage"><div><div><h5 class="title"><a name="d2207e118"></a>1.3.1.3.2. HBase Region With Non-Local Data</h5></div></div></div><p>Our next hypothesis was that one of the MapReduce tasks was requesting data from HBase that was not local to the datanode, thus
+ </p></div><div class="section" title="1.3.1.3.2. HBase Region With Non-Local Data"><div class="titlepage"><div><div><h5 class="title"><a name="d2403e118"></a>1.3.1.3.2. HBase Region With Non-Local Data</h5></div></div></div><p>Our next hypothesis was that one of the MapReduce tasks was requesting data from HBase that was not local to the datanode, thus
forcing HDFS to request data blocks from other servers over the network. Examination of the datanode logs showed that there were very
few blocks being requested over the network, indicating that the HBase region was correctly assigned, and that the majority of the necessary
data was located on the node. This ruled out the possibility of non-local data causing a slowdown.
- </p></div><div class="section" title="1.3.1.3.3. Excessive I/O Wait Due To Swapping Or An Over-Worked Or Failing Hard Disk"><div class="titlepage"><div><div><h5 class="title"><a name="d2207e123"></a>1.3.1.3.3. Excessive I/O Wait Due To Swapping Or An Over-Worked Or Failing Hard Disk</h5></div></div></div><p>After concluding that the Hadoop and HBase were not likely to be the culprits, we moved on to troubleshooting the datanode's hardware.
+ </p></div><div class="section" title="1.3.1.3.3. Excessive I/O Wait Due To Swapping Or An Over-Worked Or Failing Hard Disk"><div class="titlepage"><div><div><h5 class="title"><a name="d2403e123"></a>1.3.1.3.3. Excessive I/O Wait Due To Swapping Or An Over-Worked Or Failing Hard Disk</h5></div></div></div><p>After concluding that the Hadoop and HBase were not likely to be the culprits, we moved on to troubleshooting the datanode's hardware.
Java, by design, will periodically scan its entire memory space to do garbage collection. If system memory is heavily overcommitted, the Linux
kernel may enter a vicious cycle, using up all of its resources swapping Java heap back and forth from disk to RAM as Java tries to run garbage
collection. Further, a failing hard disk will often retry reads and/or writes many times before giving up and returning an error. This can manifest
as high iowait, as running processes wait for reads and writes to complete. Finally, a disk nearing the upper edge of its performance envelope will
begin to cause iowait as it informs the kernel that it cannot accept any more data, and the kernel queues incoming data into the dirty write pool in memory.
However, using <code class="code">vmstat(1)</code> and <code class="code">free(1)</code>, we could see that no swap was being used, and the amount of disk IO was only a few kilobytes per second.
- </p></div><div class="section" title="1.3.1.3.4. Slowness Due To High Processor Usage"><div class="titlepage"><div><div><h5 class="title"><a name="d2207e134"></a>1.3.1.3.4. Slowness Due To High Processor Usage</h5></div></div></div><p>Next, we checked to see whether the system was performing slowly simply due to very high computational load. <code class="code">top(1)</code> showed that the system load
+ </p></div><div class="section" title="1.3.1.3.4. Slowness Due To High Processor Usage"><div class="titlepage"><div><div><h5 class="title"><a name="d2403e134"></a>1.3.1.3.4. Slowness Due To High Processor Usage</h5></div></div></div><p>Next, we checked to see whether the system was performing slowly simply due to very high computational load. <code class="code">top(1)</code> showed that the system load
was higher than normal, but <code class="code">vmstat(1)</code> and <code class="code">mpstat(1)</code> showed that the amount of processor being used for actual computation was low.
- </p></div><div class="section" title="1.3.1.3.5. Network Saturation (The Winner)"><div class="titlepage"><div><div><h5 class="title"><a name="d2207e148"></a>1.3.1.3.5. Network Saturation (The Winner)</h5></div></div></div><p>Since neither the disks nor the processors were being utilized heavily, we moved on to the performance of the network interfaces. The datanode had two
+ </p></div><div class="section" title="1.3.1.3.5. Network Saturation (The Winner)"><div class="titlepage"><div><div><h5 class="title"><a name="d2403e148"></a>1.3.1.3.5. Network Saturation (The Winner)</h5></div></div></div><p>Since neither the disks nor the processors were being utilized heavily, we moved on to the performance of the network interfaces. The datanode had two
gigabit ethernet adapters, bonded to form an active-standby interface. <code class="code">ifconfig(8)</code> showed some unusual anomalies, namely interface errors, overruns, framing errors.
While not unheard of, these kinds of errors are exceedingly rare on modern hardware which is operating as it should:
</p><pre class="programlisting">
@@ -182,7 +182,7 @@ Current message level: 0x00000003 (3)
Link detected: yes
</pre><p>
</p><p>In normal operation, the ICMP ping round trip time should be around 20ms, and the interface speed and duplex should read, "1000MB/s", and, "Full", respectively.
- </p></div></div><div class="section" title="1.3.1.4. Resolution"><div class="titlepage"><div><div><h4 class="title"><a name="d2207e169"></a>1.3.1.4. Resolution</h4></div></div></div><p>After determining that the active ethernet adapter was at the incorrect speed, we used the <code class="code">ifenslave(8)</code> command to make the standby interface
+ </p></div></div><div class="section" title="1.3.1.4. Resolution"><div class="titlepage"><div><div><h4 class="title"><a name="d2403e169"></a>1.3.1.4. Resolution</h4></div></div></div><p>After determining that the active ethernet adapter was at the incorrect speed, we used the <code class="code">ifenslave(8)</code> command to make the standby interface
the active interface, which yielded an immediate improvement in MapReduce performance, and a 10 times improvement in network throughput:
</p><p>On the next trip to the datacenter, we determined that the line speed issue was ultimately caused by a bad network cable, which was replaced.
</p></div></div><div class="section" title="1.3.2. Case Study #2 (Performance Research 2012)"><div class="titlepage"><div><div><h3 class="title"><a name="casestudies.perf.1"></a>1.3.2. Case Study #2 (Performance Research 2012)</h3></div></div></div><p>Investigation results of a self-described "we're not sure what's wrong, but it seems slow" problem.
Modified: hbase/hbase.apache.org/trunk/case_studies/casestudies.perftroub.html
URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/case_studies/casestudies.perftroub.html?rev=1463652&r1=1463651&r2=1463652&view=diff
==============================================================================
--- hbase/hbase.apache.org/trunk/case_studies/casestudies.perftroub.html (original)
+++ hbase/hbase.apache.org/trunk/case_studies/casestudies.perftroub.html Tue Apr 2 18:06:19 2013
@@ -1,33 +1,33 @@
<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
- <title>1.3. Performance/Troubleshooting</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="case_studies.html" title="Chapter 1. Apache HBase (TM) Case Studies"><link rel="up" href="case_studies.html" title="Chapter 1. Apache HBase (TM) Case Studies"><link rel="prev" href="casestudies.schema.html" title="1.2. Schema Design"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">1.3. Performance/Troubleshooting</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="casestudies.schema.html">Prev</a> </td><th width="60%" align="center"> </th><td width="20%" align="right"> </td></tr></table><hr></div><div class="section" title="1.3. Performance/Troubleshooti
ng"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="casestudies.perftroub"></a>1.3. Performance/Troubleshooting</h2></div></div></div><div class="section" title="1.3.1. Case Study #1 (Performance Issue On A Single Node)"><div class="titlepage"><div><div><h3 class="title"><a name="casestudies.slownode"></a>1.3.1. Case Study #1 (Performance Issue On A Single Node)</h3></div></div></div><div class="section" title="1.3.1.1. Scenario"><div class="titlepage"><div><div><h4 class="title"><a name="d2207e82"></a>1.3.1.1. Scenario</h4></div></div></div><p>Following a scheduled reboot, one data node began exhibiting unusual behavior. Routine MapReduce
+ <title>1.3. Performance/Troubleshooting</title><link rel="stylesheet" type="text/css" href="../css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"><link rel="home" href="case_studies.html" title="Chapter 1. Apache HBase (TM) Case Studies"><link rel="up" href="case_studies.html" title="Chapter 1. Apache HBase (TM) Case Studies"><link rel="prev" href="casestudies.schema.html" title="1.2. Schema Design"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">1.3. Performance/Troubleshooting</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="casestudies.schema.html">Prev</a> </td><th width="60%" align="center"> </th><td width="20%" align="right"> </td></tr></table><hr></div><div class="section" title="1.3. Performance/Troubleshooti
ng"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="casestudies.perftroub"></a>1.3. Performance/Troubleshooting</h2></div></div></div><div class="section" title="1.3.1. Case Study #1 (Performance Issue On A Single Node)"><div class="titlepage"><div><div><h3 class="title"><a name="casestudies.slownode"></a>1.3.1. Case Study #1 (Performance Issue On A Single Node)</h3></div></div></div><div class="section" title="1.3.1.1. Scenario"><div class="titlepage"><div><div><h4 class="title"><a name="d2403e82"></a>1.3.1.1. Scenario</h4></div></div></div><p>Following a scheduled reboot, one data node began exhibiting unusual behavior. Routine MapReduce
jobs run against HBase tables which regularly completed in five or six minutes began taking 30 or 40 minutes
to finish. These jobs were consistently found to be waiting on map and reduce tasks assigned to the troubled data node
(e.g., the slow map tasks all had the same Input Split).
The situation came to a head during a distributed copy, when the copy was severely prolonged by the lagging node.
- </p></div><div class="section" title="1.3.1.2. Hardware"><div class="titlepage"><div><div><h4 class="title"><a name="d2207e87"></a>1.3.1.2. Hardware</h4></div></div></div><p>Datanodes:
+ </p></div><div class="section" title="1.3.1.2. Hardware"><div class="titlepage"><div><div><h4 class="title"><a name="d2403e87"></a>1.3.1.2. Hardware</h4></div></div></div><p>Datanodes:
</p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem">Two 12-core processors</li><li class="listitem">Six Enerprise SATA disks</li><li class="listitem">24GB of RAM</li><li class="listitem">Two bonded gigabit NICs</li></ul></div><p>
</p><p>Network:
</p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem">10 Gigabit top-of-rack switches</li><li class="listitem">20 Gigabit bonded interconnects between racks.</li></ul></div><p>
- </p></div><div class="section" title="1.3.1.3. Hypotheses"><div class="titlepage"><div><div><h4 class="title"><a name="d2207e110"></a>1.3.1.3. Hypotheses</h4></div></div></div><div class="section" title="1.3.1.3.1. HBase "Hot Spot" Region"><div class="titlepage"><div><div><h5 class="title"><a name="d2207e113"></a>1.3.1.3.1. HBase "Hot Spot" Region</h5></div></div></div><p>We hypothesized that we were experiencing a familiar point of pain: a "hot spot" region in an HBase table,
+ </p></div><div class="section" title="1.3.1.3. Hypotheses"><div class="titlepage"><div><div><h4 class="title"><a name="d2403e110"></a>1.3.1.3. Hypotheses</h4></div></div></div><div class="section" title="1.3.1.3.1. HBase "Hot Spot" Region"><div class="titlepage"><div><div><h5 class="title"><a name="d2403e113"></a>1.3.1.3.1. HBase "Hot Spot" Region</h5></div></div></div><p>We hypothesized that we were experiencing a familiar point of pain: a "hot spot" region in an HBase table,
where uneven key-space distribution can funnel a huge number of requests to a single HBase region, bombarding the RegionServer
process and cause slow response time. Examination of the HBase Master status page showed that the number of HBase requests to the
troubled node was almost zero. Further, examination of the HBase logs showed that there were no region splits, compactions, or other region transitions
in progress. This effectively ruled out a "hot spot" as the root cause of the observed slowness.
- </p></div><div class="section" title="1.3.1.3.2. HBase Region With Non-Local Data"><div class="titlepage"><div><div><h5 class="title"><a name="d2207e118"></a>1.3.1.3.2. HBase Region With Non-Local Data</h5></div></div></div><p>Our next hypothesis was that one of the MapReduce tasks was requesting data from HBase that was not local to the datanode, thus
+ </p></div><div class="section" title="1.3.1.3.2. HBase Region With Non-Local Data"><div class="titlepage"><div><div><h5 class="title"><a name="d2403e118"></a>1.3.1.3.2. HBase Region With Non-Local Data</h5></div></div></div><p>Our next hypothesis was that one of the MapReduce tasks was requesting data from HBase that was not local to the datanode, thus
forcing HDFS to request data blocks from other servers over the network. Examination of the datanode logs showed that there were very
few blocks being requested over the network, indicating that the HBase region was correctly assigned, and that the majority of the necessary
data was located on the node. This ruled out the possibility of non-local data causing a slowdown.
- </p></div><div class="section" title="1.3.1.3.3. Excessive I/O Wait Due To Swapping Or An Over-Worked Or Failing Hard Disk"><div class="titlepage"><div><div><h5 class="title"><a name="d2207e123"></a>1.3.1.3.3. Excessive I/O Wait Due To Swapping Or An Over-Worked Or Failing Hard Disk</h5></div></div></div><p>After concluding that the Hadoop and HBase were not likely to be the culprits, we moved on to troubleshooting the datanode's hardware.
+ </p></div><div class="section" title="1.3.1.3.3. Excessive I/O Wait Due To Swapping Or An Over-Worked Or Failing Hard Disk"><div class="titlepage"><div><div><h5 class="title"><a name="d2403e123"></a>1.3.1.3.3. Excessive I/O Wait Due To Swapping Or An Over-Worked Or Failing Hard Disk</h5></div></div></div><p>After concluding that the Hadoop and HBase were not likely to be the culprits, we moved on to troubleshooting the datanode's hardware.
Java, by design, will periodically scan its entire memory space to do garbage collection. If system memory is heavily overcommitted, the Linux
kernel may enter a vicious cycle, using up all of its resources swapping Java heap back and forth from disk to RAM as Java tries to run garbage
collection. Further, a failing hard disk will often retry reads and/or writes many times before giving up and returning an error. This can manifest
as high iowait, as running processes wait for reads and writes to complete. Finally, a disk nearing the upper edge of its performance envelope will
begin to cause iowait as it informs the kernel that it cannot accept any more data, and the kernel queues incoming data into the dirty write pool in memory.
However, using <code class="code">vmstat(1)</code> and <code class="code">free(1)</code>, we could see that no swap was being used, and the amount of disk IO was only a few kilobytes per second.
- </p></div><div class="section" title="1.3.1.3.4. Slowness Due To High Processor Usage"><div class="titlepage"><div><div><h5 class="title"><a name="d2207e134"></a>1.3.1.3.4. Slowness Due To High Processor Usage</h5></div></div></div><p>Next, we checked to see whether the system was performing slowly simply due to very high computational load. <code class="code">top(1)</code> showed that the system load
+ </p></div><div class="section" title="1.3.1.3.4. Slowness Due To High Processor Usage"><div class="titlepage"><div><div><h5 class="title"><a name="d2403e134"></a>1.3.1.3.4. Slowness Due To High Processor Usage</h5></div></div></div><p>Next, we checked to see whether the system was performing slowly simply due to very high computational load. <code class="code">top(1)</code> showed that the system load
was higher than normal, but <code class="code">vmstat(1)</code> and <code class="code">mpstat(1)</code> showed that the amount of processor being used for actual computation was low.
- </p></div><div class="section" title="1.3.1.3.5. Network Saturation (The Winner)"><div class="titlepage"><div><div><h5 class="title"><a name="d2207e148"></a>1.3.1.3.5. Network Saturation (The Winner)</h5></div></div></div><p>Since neither the disks nor the processors were being utilized heavily, we moved on to the performance of the network interfaces. The datanode had two
+ </p></div><div class="section" title="1.3.1.3.5. Network Saturation (The Winner)"><div class="titlepage"><div><div><h5 class="title"><a name="d2403e148"></a>1.3.1.3.5. Network Saturation (The Winner)</h5></div></div></div><p>Since neither the disks nor the processors were being utilized heavily, we moved on to the performance of the network interfaces. The datanode had two
gigabit ethernet adapters, bonded to form an active-standby interface. <code class="code">ifconfig(8)</code> showed some unusual anomalies, namely interface errors, overruns, framing errors.
While not unheard of, these kinds of errors are exceedingly rare on modern hardware which is operating as it should:
</p><pre class="programlisting">
@@ -72,7 +72,7 @@ Current message level: 0x00000003 (3)
Link detected: yes
</pre><p>
</p><p>In normal operation, the ICMP ping round trip time should be around 20ms, and the interface speed and duplex should read, "1000MB/s", and, "Full", respectively.
- </p></div></div><div class="section" title="1.3.1.4. Resolution"><div class="titlepage"><div><div><h4 class="title"><a name="d2207e169"></a>1.3.1.4. Resolution</h4></div></div></div><p>After determining that the active ethernet adapter was at the incorrect speed, we used the <code class="code">ifenslave(8)</code> command to make the standby interface
+ </p></div></div><div class="section" title="1.3.1.4. Resolution"><div class="titlepage"><div><div><h4 class="title"><a name="d2403e169"></a>1.3.1.4. Resolution</h4></div></div></div><p>After determining that the active ethernet adapter was at the incorrect speed, we used the <code class="code">ifenslave(8)</code> command to make the standby interface
the active interface, which yielded an immediate improvement in MapReduce performance, and a 10 times improvement in network throughput:
</p><p>On the next trip to the datacenter, we determined that the line speed issue was ultimately caused by a bad network cable, which was replaced.
</p></div></div><div class="section" title="1.3.2. Case Study #2 (Performance Research 2012)"><div class="titlepage"><div><div><h3 class="title"><a name="casestudies.perf.1"></a>1.3.2. Case Study #2 (Performance Research 2012)</h3></div></div></div><p>Investigation results of a self-described "we're not sure what's wrong, but it seems slow" problem.
Modified: hbase/hbase.apache.org/trunk/community.html
URL: http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/community.html?rev=1463652&r1=1463651&r2=1463652&view=diff
==============================================================================
--- hbase/hbase.apache.org/trunk/community.html (original)
+++ hbase/hbase.apache.org/trunk/community.html Tue Apr 2 18:06:19 2013
@@ -3,7 +3,7 @@
<title>Chapter 1. Community</title><link rel="stylesheet" type="text/css" href="css/freebsd_docbook.css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="chapter" title="Chapter 1. Community"><div class="titlepage"><div><div><h2 class="title"><a name="community"></a>Chapter 1. Community</h2></div></div></div><div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="section"><a href="#decisions">1.1. Decisions</a></span></dt><dd><dl><dt><span class="section"><a href="#feature_branches">1.1.1. Feature Branches</a></span></dt><dt><span class="section"><a href="#patchplusonepolicy">1.1.2. Patch +1 Policy</a></span></dt></dl></dd><dt><span class="section"><a href="#community.roles">1.2. Community Roles</a></span></dt><dd><dl><dt><span class="section"><a href="#OWNER">1.2.1. Component Owner</a></span></dt></dl></dd></dl
></div><div class="section" title="1.1. Decisions"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="decisions"></a>1.1. Decisions</h2></div></div></div><div class="section" title="1.1.1. Feature Branches"><div class="titlepage"><div><div><h3 class="title"><a name="feature_branches"></a>1.1.1. Feature Branches</h3></div></div></div><p>Feature Branches are easy to make. You do not have to be a committer to make one. Just request the name of your branch be added to JIRA up on the
developer's mailing list and a committer will add it for you. Thereafter you can file issues against your feature branch in Apache HBase (TM) JIRA. Your code you
keep elsewhere -- it should be public so it can be observed -- and you can update dev mailing list on progress. When the feature is ready for commit,
- 3 +1s from committers will get your feature merged<sup>[<a name="d2181e13" href="#ftn.d2181e13" class="footnote">1</a>]</sup>
+ 3 +1s from committers will get your feature merged<sup>[<a name="d2377e13" href="#ftn.d2377e13" class="footnote">1</a>]</sup>
</p></div><div class="section" title="1.1.2. Patch +1 Policy"><div class="titlepage"><div><div><h3 class="title"><a name="patchplusonepolicy"></a>1.1.2. Patch +1 Policy</h3></div></div></div><p>
The below policy is something we put in place 09/2012. It is a
suggested policy rather than a hard requirement. We want to try it
@@ -47,7 +47,7 @@ goals or the design toward which they ar
</p><p>
If you would like to be volunteer as a component owner, just write the
dev list and we'll sign you up. Owners do not need to be committers.
-</p></div></div><div class="footnotes"><br><hr width="100" align="left"><div class="footnote"><p><sup>[<a id="ftn.d2181e13" href="#d2181e13" class="para">1</a>] </sup>See <a class="link" href="http://search-hadoop.com/m/asM982C5FkS1" target="_top">HBase, mail # dev - Thoughts about large feature dev branches</a></p></div></div></div><div id="disqus_thread"></div><script type="text/javascript">
+</p></div></div><div class="footnotes"><br><hr width="100" align="left"><div class="footnote"><p><sup>[<a id="ftn.d2377e13" href="#d2377e13" class="para">1</a>] </sup>See <a class="link" href="http://search-hadoop.com/m/asM982C5FkS1" target="_top">HBase, mail # dev - Thoughts about large feature dev branches</a></p></div></div></div><div id="disqus_thread"></div><script type="text/javascript">
var disqus_shortname = 'hbase'; // required: replace example with your forum shortname
var disqus_url = 'http://hbase.apache.org/book';
var disqus_identifier = 'community';