You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Andrew Purtell (JIRA)" <ji...@apache.org> on 2010/01/08 20:07:54 UTC
[jira] Created: (HBASE-2098) [EC2] Build a HA cluster
[EC2] Build a HA cluster
------------------------
Key: HBASE-2098
URL: https://issues.apache.org/jira/browse/HBASE-2098
Project: Hadoop HBase
Issue Type: Sub-task
Reporter: Andrew Purtell
Assignee: Andrew Purtell
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2098) [EC2] Build a HA cluster
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-2098:
----------------------------------
Description: The Hadoop NameNode is a single point of failure. If the master instance fails, HDFS is down; therefore, HBase as well. So we do not try to deploy HBase in a multimaster configuration for that reason. Instead we colocate the HDFS NameNode and HBase HMaster on the same instance and run with its failure as a known risk. As these EC2 scripts are starter scripts which can (and should) be customized, this is ok, but we can do better. We should deploy a fully fault tolerant Hadoop+HBase cluster as a worked example of how to do it.
Priority: Minor (was: Major)
Fix Version/s: 0.21.0
> [EC2] Build a HA cluster
> ------------------------
>
> Key: HBASE-2098
> URL: https://issues.apache.org/jira/browse/HBASE-2098
> Project: Hadoop HBase
> Issue Type: Sub-task
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Priority: Minor
> Fix For: 0.21.0
>
>
> The Hadoop NameNode is a single point of failure. If the master instance fails, HDFS is down; therefore, HBase as well. So we do not try to deploy HBase in a multimaster configuration for that reason. Instead we colocate the HDFS NameNode and HBase HMaster on the same instance and run with its failure as a known risk. As these EC2 scripts are starter scripts which can (and should) be customized, this is ok, but we can do better. We should deploy a fully fault tolerant Hadoop+HBase cluster as a worked example of how to do it.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-2098) [EC2] Build a HA cluster
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell resolved HBASE-2098.
-----------------------------------
Resolution: Later
Closed per HBASE-2543.
> [EC2] Build a HA cluster
> ------------------------
>
> Key: HBASE-2098
> URL: https://issues.apache.org/jira/browse/HBASE-2098
> Project: Hadoop HBase
> Issue Type: Sub-task
> Reporter: Andrew Purtell
> Priority: Minor
> Fix For: 0.22.0
>
>
> The Hadoop NameNode is a single point of failure. If the master instance fails, HDFS is down; therefore, HBase as well. So we do not try to deploy HBase in a multimaster configuration for that reason. Instead we colocate the HDFS NameNode and HBase HMaster on the same instance and run with its failure as a known risk. As these EC2 scripts are starter scripts which can (and should) be customized, this is ok, but we can do better. We should deploy a fully fault tolerant Hadoop+HBase cluster as a worked example of how to do it.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2098) [EC2] Build a HA cluster
Posted by "Berk D. Demir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12798157#action_12798157 ]
Berk D. Demir commented on HBASE-2098:
--------------------------------------
Until HDFS comes up with a solution to eliminate NN SPoF, old fashioned HA measures are required to keep NameNode available.
So far, best and seemingly reliable bet on Linux is to have network replicated block device, a heart beat providing messaging connection between HA nodes and a cluster resource manager software to keep track of infrastructural resource dependencies and moving them between machines in the requirement order.
All in all, HBase's tolerance window for NN unavailability mostly depends on particular load at the time of failover and RSs requirements to create new files.
Failing over to another node where a healthy replica of NN store exists and starting an NN instance will cause the NN to collect block information from every "new" and "unknown" DataNode for the first time. Additionally, default value for extension of safe mode after threshold reach is 30 seconds. (property: {{dfs.namenode.safemode.extension}}). This prolonged unavailability window can/will have bad effects on RSs. (jdcryans will comment his observations).
We implemented a NameNode HA cluster with open source tools like OpenAIS, Pacemaker, Heartbeat and DRBD.
* NameNode disk storage is replicated between two machines _(adding a 3rd machine is possible with new DRBD)_.
* OpenAIS provides intra-cluster messaging and heart beat availability layer.
* Pacemaker is used to manage Cluster Resources. _(DRBD disks, filesystem mount, NN service IP, NN daemon)_
* An OCF script to start, stop, validate and monitor _(periodic calls)_ the subsystem _(NN, JT, SNN)_.
At the end of the day, this is applicable to not only NameNode but also to JobTracker and SecondaryNameNode.
For a starting point, ClusterLabs _(creators of Pacemaker)_ offer an e-book on creating clusters with DRBD, Pacemaker, OpenAIS called [Clusters From Scratch|http://www.clusterlabs.org/mediawiki/images/9/9d/Clusters_from_Scratch_-_Apache_on_Fedora11.pdf] ^(PDF Link)^
> [EC2] Build a HA cluster
> ------------------------
>
> Key: HBASE-2098
> URL: https://issues.apache.org/jira/browse/HBASE-2098
> Project: Hadoop HBase
> Issue Type: Sub-task
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Priority: Minor
> Fix For: 0.21.0
>
>
> The Hadoop NameNode is a single point of failure. If the master instance fails, HDFS is down; therefore, HBase as well. So we do not try to deploy HBase in a multimaster configuration for that reason. Instead we colocate the HDFS NameNode and HBase HMaster on the same instance and run with its failure as a known risk. As these EC2 scripts are starter scripts which can (and should) be customized, this is ok, but we can do better. We should deploy a fully fault tolerant Hadoop+HBase cluster as a worked example of how to do it.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2098) [EC2] Build a HA cluster
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12798412#action_12798412 ]
Eli Collins commented on HBASE-2098:
------------------------------------
bq. Failing over to another node where a healthy replica of NN store exists and starting an NN instance will cause the NN to collect block information from every "new" and "unknown" DataNode for the first time.
Check out HDFS-839 (NN forwards block reports to the BNN). Enabling high availability via fast automatic fail over to the backup name node is something HDFS developers are working on. You also might find [Dhruba's recent post on HA|http://hadoopblog.blogspot.com/2009/11/hdfs-high-availability.html] of interest.
> [EC2] Build a HA cluster
> ------------------------
>
> Key: HBASE-2098
> URL: https://issues.apache.org/jira/browse/HBASE-2098
> Project: Hadoop HBase
> Issue Type: Sub-task
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Priority: Minor
> Fix For: 0.21.0
>
>
> The Hadoop NameNode is a single point of failure. If the master instance fails, HDFS is down; therefore, HBase as well. So we do not try to deploy HBase in a multimaster configuration for that reason. Instead we colocate the HDFS NameNode and HBase HMaster on the same instance and run with its failure as a known risk. As these EC2 scripts are starter scripts which can (and should) be customized, this is ok, but we can do better. We should deploy a fully fault tolerant Hadoop+HBase cluster as a worked example of how to do it.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2098) [EC2] Build a HA cluster
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12798864#action_12798864 ]
Andrew Purtell commented on HBASE-2098:
---------------------------------------
Link to HBASE-2108.
> [EC2] Build a HA cluster
> ------------------------
>
> Key: HBASE-2098
> URL: https://issues.apache.org/jira/browse/HBASE-2098
> Project: Hadoop HBase
> Issue Type: Sub-task
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Priority: Minor
> Fix For: 0.21.0
>
>
> The Hadoop NameNode is a single point of failure. If the master instance fails, HDFS is down; therefore, HBase as well. So we do not try to deploy HBase in a multimaster configuration for that reason. Instead we colocate the HDFS NameNode and HBase HMaster on the same instance and run with its failure as a known risk. As these EC2 scripts are starter scripts which can (and should) be customized, this is ok, but we can do better. We should deploy a fully fault tolerant Hadoop+HBase cluster as a worked example of how to do it.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2098) [EC2] Build a HA cluster
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12798274#action_12798274 ]
Andrew Purtell commented on HBASE-2098:
---------------------------------------
@Berk: Thank you for all of the very helpful comment.
> [EC2] Build a HA cluster
> ------------------------
>
> Key: HBASE-2098
> URL: https://issues.apache.org/jira/browse/HBASE-2098
> Project: Hadoop HBase
> Issue Type: Sub-task
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Priority: Minor
> Fix For: 0.21.0
>
>
> The Hadoop NameNode is a single point of failure. If the master instance fails, HDFS is down; therefore, HBase as well. So we do not try to deploy HBase in a multimaster configuration for that reason. Instead we colocate the HDFS NameNode and HBase HMaster on the same instance and run with its failure as a known risk. As these EC2 scripts are starter scripts which can (and should) be customized, this is ok, but we can do better. We should deploy a fully fault tolerant Hadoop+HBase cluster as a worked example of how to do it.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.