[jira] Commented: (HBASE-2098) [EC2] Build a HA cluster

["Berk D. Demir (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-2098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12798157#action_12798157 ] 

Berk D. Demir commented on HBASE-2098:
--------------------------------------

Until HDFS comes up with a solution to eliminate NN SPoF, old fashioned HA measures are required to keep NameNode available.

So far, best and seemingly reliable bet on Linux is to have network replicated block device, a heart beat providing messaging connection between HA nodes and a cluster resource manager software to keep track of infrastructural resource dependencies and moving them between machines in the requirement order.

All in all, HBase's tolerance window for NN unavailability mostly depends on particular load at the time of failover and RSs requirements to create new files.

Failing over to another node where a healthy replica of NN store exists and starting an NN instance will cause the NN to collect block information from every "new" and "unknown" DataNode for the first time. Additionally, default value for extension of safe mode after threshold reach is 30 seconds. (property: {{dfs.namenode.safemode.extension}}). This prolonged unavailability window can/will have bad effects on RSs. (jdcryans will comment his observations).

We implemented a NameNode HA cluster with open source tools like OpenAIS, Pacemaker, Heartbeat and DRBD.

* NameNode disk storage is replicated between two machines _(adding a 3rd machine is possible with new DRBD)_.
* OpenAIS provides intra-cluster messaging and heart beat availability layer.
* Pacemaker is used to manage Cluster Resources. _(DRBD disks, filesystem mount, NN service IP, NN daemon)_
* An OCF script to start, stop, validate and monitor _(periodic calls)_ the subsystem _(NN, JT, SNN)_.

At the end of the day, this is applicable to not only NameNode but also to JobTracker and SecondaryNameNode.

For a starting point, ClusterLabs _(creators of Pacemaker)_ offer an e-book on creating clusters with DRBD, Pacemaker, OpenAIS called [Clusters From Scratch|http://www.clusterlabs.org/mediawiki/images/9/9d/Clusters_from_Scratch_-_Apache_on_Fedora11.pdf] ^(PDF Link)^


> [EC2] Build a HA cluster
> ------------------------
>
>                 Key: HBASE-2098
>                 URL: https://issues.apache.org/jira/browse/HBASE-2098
>             Project: Hadoop HBase
>          Issue Type: Sub-task
>            Reporter: Andrew Purtell
>            Assignee: Andrew Purtell
>            Priority: Minor
>             Fix For: 0.21.0
>
>
> The Hadoop NameNode is a single point of failure. If the master instance fails, HDFS is down; therefore, HBase as well. So we do not try to deploy HBase in a multimaster configuration for that reason. Instead we colocate the HDFS NameNode and HBase HMaster on the same instance and run with its failure as a known risk. As these EC2 scripts are starter scripts which can (and should) be customized, this is ok, but we can do better. We should deploy a fully fault tolerant Hadoop+HBase cluster as a worked example of how to do it. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.