You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hbase.apache.org by jd...@apache.org on 2010/04/08 23:03:02 UTC
svn commit: r932112 - in /hadoop/hbase/branches/0.20: CHANGES.txt
src/java/overview.html
Author: jdcryans
Date: Thu Apr 8 21:03:02 2010
New Revision: 932112
URL: http://svn.apache.org/viewvc?rev=932112&view=rev
Log:
HBASE-2423 Update 'Getting Started' for 0.20.4 including making
"important configurations more visiable"
Modified:
hadoop/hbase/branches/0.20/CHANGES.txt
hadoop/hbase/branches/0.20/src/java/overview.html
Modified: hadoop/hbase/branches/0.20/CHANGES.txt
URL: http://svn.apache.org/viewvc/hadoop/hbase/branches/0.20/CHANGES.txt?rev=932112&r1=932111&r2=932112&view=diff
==============================================================================
--- hadoop/hbase/branches/0.20/CHANGES.txt (original)
+++ hadoop/hbase/branches/0.20/CHANGES.txt Thu Apr 8 21:03:02 2010
@@ -121,6 +121,8 @@ Release 0.20.4 - Unreleased
HBASE-2415 Disable META splitting in 0.20 (Todd Lipcon via Stack)
HBASE-2360 Make sure we have all the hadoop fixes in our our copy of its rpc
(Todd Lipcon via Stack)
+ HBASE-2423 Update 'Getting Started' for 0.20.4 including making
+ "important configurations more visiable"
NEW FEATURES
HBASE-2257 [stargate] multiuser mode
Modified: hadoop/hbase/branches/0.20/src/java/overview.html
URL: http://svn.apache.org/viewvc/hadoop/hbase/branches/0.20/src/java/overview.html?rev=932112&r1=932111&r2=932112&view=diff
==============================================================================
--- hadoop/hbase/branches/0.20/src/java/overview.html (original)
+++ hadoop/hbase/branches/0.20/src/java/overview.html Thu Apr 8 21:03:02 2010
@@ -53,7 +53,7 @@
<h2><a name="requirements">Requirements</a></h2>
<ul>
- <li>Java 1.6.x, preferably from <a href="http://www.java.com/download/">Sun</a>. Use the latest version available.</li>
+ <li>Java 1.6.x, preferably from <a href="http://www.java.com/download/">Sun</a>. Use the latest version available except u18 (u19 is fine).</li>
<li>This version of HBase will only run on <a href="http://hadoop.apache.org/common/releases.html">Hadoop 0.20.x</a>.</li>
<li>
<em>ssh</em> must be installed and <em>sshd</em> must be running to use Hadoop's scripts to manage remote Hadoop daemons.
@@ -72,31 +72,13 @@
</li>
<li>Hosts must be able to resolve the fully-qualified domain name of the master.</li>
<li>
- HBase currently is a file handle hog. The usual default of 1024 on *nix systems is insufficient
- if you are loading any significant amount of data into regionservers.
- See the <a href="http://wiki.apache.org/hadoop/Hbase/FAQ#A6">FAQ: Why do I see "java.io.IOException...(Too many open files)" in my logs?</a>
- for how to up the limit. Also, as of 0.18.x Hadoop DataNodes have an upper-bound on the number of threads they will
- support (<code>dfs.datanode.max.xcievers</code>). The default is 256 threads. Up this limit on your hadoop cluster.
- </li>
- <li>
The clocks on cluster members should be in basic alignments. Some skew is tolerable but
wild skew could generate odd behaviors. Run <a href="http://en.wikipedia.org/wiki/Network_Time_Protocol">NTP</a>
on your cluster, or an equivalent.
</li>
<li>
- HBase servers put up 10 listeners for incoming connections by default.
- Up this number if you have a dataset of any substance by setting <code>hbase.regionserver.handler.count</code>
- in your <code>hbase-site.xml</code>.
- </li>
- <li>
This is the current list of patches we recommend you apply to your running Hadoop cluster:
<ul>
- <li><a hef="https://issues.apache.org/jira/browse/HADOOP-4681">HADOOP-4681/HDFS-127 <i>"DFSClient block read failures cause open DFSInputStream to become unusable"</i></a>. This patch will help with the ever-popular, "No live nodes contain current block".
- The hadoop version bundled with hbase has this patch applied. Its an HDFS client
- fix so this should do for usual usage but if your cluster is missing the patch,
- and in particular if calling hbase from a mapreduce job, you may run into this
- issue. If running with hadoop 0.20.2, it has this patch in place already.
- </li>
<li>
<a href="https://issues.apache.org/jira/browse/HDFS-630">HDFS-630: <em>"In DFSOutputStream.nextBlockOutputStream(), the client can exclude specific datanodes when locating the next block"</em></a>.
Dead DataNodes take ten minutes to timeout at NameNode.
@@ -106,6 +88,28 @@
</li>
</ul>
</li>
+ <li>
+ HBase is a database, it uses a lot of files at the same time. The default <b>ulimit -n</b> of 1024 on *nix systems is insufficient.
+ Any significant amount of loading will lead you to
+ <a href="http://wiki.apache.org/hadoop/Hbase/FAQ#A6">FAQ: Why do I see "java.io.IOException...(Too many open files)" in my logs?</a>.
+ You will also notice errors like:
+ <pre>
+2010-04-06 03:04:37,542 INFO org.apache.hadoop.hdfs.DFSClient: Exception increateBlockOutputStream java.io.EOFException
+2010-04-06 03:04:37,542 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_-6935524980745310745_1391901
+ </pre>
+ Do yourself a favor and change this to more than 10k using the FAQ.
+ Also, HDFS has an upper bound of files that it can serve at the same time, called xcievers (yes, this is <em>misspelled</em>). Again, before doing any loading,
+ make sure you configured Hadoop's conf/hdfs-site.xml with this:
+ <pre>
+<property>
+ <name>dfs.datanode.max.xcievers</name>
+ <value>2047</value>
+</property>
+ </pre>
+ See the background of this issue here: <a href="http://wiki.apache.org/hadoop/Hbase/Troubleshooting#A5">Problem: "xceiverCount 258 exceeds the limit of concurrent xcievers 256"</a>.
+ Failure to follow these instructions will result in <b>data loss</b>.
+ </li>
+
</ul>
<h3><a name="windows">Windows</a></h3>