[Hadoop Wiki] Update of "Hbase/FAQ_Operations" by DougMeil

[Apache Wiki <wi...@apache.org>]

Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.

The "Hbase/FAQ_Operations" page has been changed by DougMeil:
http://wiki.apache.org/hadoop/Hbase/FAQ_Operations

New page:
Operations and Troubleshooting

== Questions ==
 1. [[#4|Can I fix OutOfMemoryExceptions in hbase?]]
 1. [[#5|How do I enable hbase DEBUG-level logging?]]
 1. [[#6|Why do I see "java.io.IOException...(Too many open files)" in my logs?]]
 1. [[#7|What can I do to improve hbase performance?]]
 1. [[#9|What ports does HBase use?]]
 1. [[#10|Why is HBase ignoring HDFS client configuration such as dfs.replication?]]
 1. [[#12|Can I safely move the master from node A to node B?]]
 1. [[#13|Can I safely move the hbase rootdir in hdfs?]]
 1. [[#19|How do I add/remove a node?]]
 1. [[#20|Why do servers have start codes?]]

== Answers ==


'''4. <<Anchor(4)>> Can I fix OutOfMemoryExceptions in hbase?'''

Out-of-the-box, hbase uses a default of 1G heap size.  Set the ''HBASE_HEAPSIZE'' environment variable in ''${HBASE_HOME}/conf/hbase-env.sh'' if your install needs to run with a larger heap.  ''HBASE_HEAPSIZE'' is like ''HADOOP_HEAPSIZE'' in that its value is the desired heap size in MB.  The surrounding '-Xmx' and 'm' needed to make up the maximum heap size java option are added by the hbase start script (See how ''HBASE_HEAPSIZE'' is used in the ''${HBASE_HOME}/bin/hbase'' script for clarification).

'''5. <<Anchor(5)>> How do I enable hbase DEBUG-level logging?'''

Either add the following line to your log4j.properties file -- ''log4j.logger.org.apache.hadoop.hbase=DEBUG'' -- and restart your cluster or, if running a post-0.15.x version, you can set DEBUG via the UI by clicking on the 'Log Level' link (but you need set 'org.apache.hadoop.hbase' to DEBUG without the 'log4j.logger' prefix).

'''6. <<Anchor(6)>> Why do I see "java.io.IOException...(Too many open files)" in my logs?'''

See the Troubleshooting section in the HBase Book http://hbase.apache.org/book.html#trouble

'''7. <<Anchor(7)>> What can I do to improve hbase performance?'''

See the Performance section in the HBase book http://hbase.apache.org/book.html#performance
Also, see [[PerformanceTuning|Performance Tuning]] on the wiki home page

'''9. <<Anchor(9)>> What ports does HBase use?'''

Not counting the ports used by hadoop -- hdfs and mapreduce -- by default, hbase runs the master and its informational http server at 60000 and 60010 respectively and regionservers at 60020 and their informational http server at 60030.  ''${HBASE_HOME}/conf/hbase-default.xml'' lists the default values of all ports used.  Also check ''${HBASE_HOME}/conf/hbase-site.xml'' for site-specific overrides.


'''10. <<Anchor(10)>> Why is HBase ignoring HDFS client configuration such as dfs.replication?'''

If you have made HDFS client configuration on your hadoop cluster, HBase will not see this configuration unless you do one of the following:

 * Add a pointer to your ''HADOOP_CONF_DIR'' to ''CLASSPATH'' in ''hbase-env.sh'' or symlink your hadoop-site.xml from the hbase conf directory.
 * Add a copy of ''hadoop-site.xml'' to ''${HBASE_HOME}/conf'', or
 * If only a small set of HDFS client configurations, add them to ''hbase-site.xml''

The first option is the better of the three since it avoids duplication.

'''12. <<Anchor(12)>> Can I safely move the master from node A to node B?'''
  Yes.  HBase must be shutdown.  Edit your hbase-site.xml configuration across the cluster setting hbase.master to point at the new location.

'''13. <<Anchor(13)>> Can I safely move the hbase rootdir in hdfs?'''
  Yes.  HBase must be down for the move.  After the move, update the hbase-site.xml across the cluster and restart.


'''19. <<Anchor(19)>> How do I add/remove a node?'''

For removing nodes, see the section on decommissioning nodes in the HBase Book http://hbase.apache.org/book.html#decommission

Adding and removing nodes works the same way in HBase and Hadoop. To add a new node, do the following steps:

 1. Edit $HBASE_HOME/conf/regionservers on the Master node and add the new address.
 2. Setup the new node with needed software, permissions.
 3. On that node run $HBASE_HOME/bin/hbase-daemon.sh start regionserver
 4. Confirm it worked by looking at the Master's web UI or in that region server's log.

Removing a node is as easy, first issue "stop" instead of start then remove the address from the regionservers file. 

For Hadoop, use the same kind of script (starts with hadoop-*), their process names (datanode, tasktracker), and edit the slaves file. Removing datanodes is tricky, please review the dfsadmin command before doing it.

'''20. <<Anchor(20)>> Why do servers have start codes?'''

If a region server crashes and recovers, it cannot be given work until its lease times out. If the lease is identified only by an IP address and port number, then that server can't do any progress until the lease times out. A start code is added so that the restarted server can begin doing work immediately upon recovery. For more, see https://issues.apache.org/jira/browse/HBASE-1156.