You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by "Alejandro Fernandez (JIRA)" <ji...@apache.org> on 2015/06/06 02:23:00 UTC
[jira] [Updated] (AMBARI-11743) NameNode is forced to leave
safemode, which causes HBMaster master to crash if done too quickly
[ https://issues.apache.org/jira/browse/AMBARI-11743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alejandro Fernandez updated AMBARI-11743:
-----------------------------------------
Description:
1. Install cluster with Ambari 2.1 and HDP 2.3
2. Add services HDFS, YARN, MR, ZK, and HBaste
3. Perform several Stop All and Start All on HDFS service
4. Periodically, HBase Master will crash
This was a non-HA cluster.
{code}
2015-06-02 09:34:24,865 WARN [ip-172-31-33-225:16000.activeMasterManager] hdfs.DFSClient: Could not obtain block: BP-925466282-172.31.33.226-1433234647051:blk_1073741829_1005 file=/apps/hbase/data/hbase.id No live nodes contain current block Block locations: Dead nodes: . Throwing a BlockMissingException
2015-06-02 09:34:24,866 WARN [ip-172-31-33-225:16000.activeMasterManager] hdfs.DFSClient: DFS Read
org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-925466282-172.31.33.226-1433234647051:blk_1073741829_1005 file=/apps/hbase/data/hbase.id
at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:945)
at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:604)
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:844)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:896)
at java.io.DataInputStream.readFully(DataInputStream.java:195)
at java.io.DataInputStream.readFully(DataInputStream.java:169)
at org.apache.hadoop.hbase.util.FSUtils.getClusterId(FSUtils.java:816)
at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:474)
at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:146)
at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:126)
at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:649)
at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:182)
at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1646)
at java.lang.Thread.run(Thread.java:745)
2015-06-02 09:34:24,870 FATAL [ip-172-31-33-225:16000.activeMasterManager] master.HMaster: Failed to become active master
org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-925466282-172.31.33.226-1433234647051:blk_1073741829_1005 file=/apps/hbase/data/hbase.id
at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:945)
at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:604)
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:844)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:896)
at java.io.DataInputStream.readFully(DataInputStream.java:195)
at java.io.DataInputStream.readFully(DataInputStream.java:169)
at org.apache.hadoop.hbase.util.FSUtils.getClusterId(FSUtils.java:816)
at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:474)
at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:146)
at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:126)
at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:649)
at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:182)
at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1646)
at java.lang.Thread.run(Thread.java:745)
{code}
was:
1. Install cluster with Ambari 2.1 and HDP 2.3
2. Add services HDFS, YARN, MR, ZK, and HBaste
3. Perform several Stop All and Start All on HDFS service
4. Periodically, HBase Master will crash
This was a non-HA cluster.
> NameNode is forced to leave safemode, which causes HBMaster master to crash if done too quickly
> -----------------------------------------------------------------------------------------------
>
> Key: AMBARI-11743
> URL: https://issues.apache.org/jira/browse/AMBARI-11743
> Project: Ambari
> Issue Type: Bug
> Reporter: Alejandro Fernandez
> Assignee: Alejandro Fernandez
>
> 1. Install cluster with Ambari 2.1 and HDP 2.3
> 2. Add services HDFS, YARN, MR, ZK, and HBaste
> 3. Perform several Stop All and Start All on HDFS service
> 4. Periodically, HBase Master will crash
> This was a non-HA cluster.
> {code}
> 2015-06-02 09:34:24,865 WARN [ip-172-31-33-225:16000.activeMasterManager] hdfs.DFSClient: Could not obtain block: BP-925466282-172.31.33.226-1433234647051:blk_1073741829_1005 file=/apps/hbase/data/hbase.id No live nodes contain current block Block locations: Dead nodes: . Throwing a BlockMissingException
> 2015-06-02 09:34:24,866 WARN [ip-172-31-33-225:16000.activeMasterManager] hdfs.DFSClient: DFS Read
> org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-925466282-172.31.33.226-1433234647051:blk_1073741829_1005 file=/apps/hbase/data/hbase.id
> at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:945)
> at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:604)
> at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:844)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:896)
> at java.io.DataInputStream.readFully(DataInputStream.java:195)
> at java.io.DataInputStream.readFully(DataInputStream.java:169)
> at org.apache.hadoop.hbase.util.FSUtils.getClusterId(FSUtils.java:816)
> at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:474)
> at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:146)
> at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:126)
> at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:649)
> at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:182)
> at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1646)
> at java.lang.Thread.run(Thread.java:745)
> 2015-06-02 09:34:24,870 FATAL [ip-172-31-33-225:16000.activeMasterManager] master.HMaster: Failed to become active master
> org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-925466282-172.31.33.226-1433234647051:blk_1073741829_1005 file=/apps/hbase/data/hbase.id
> at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:945)
> at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:604)
> at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:844)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:896)
> at java.io.DataInputStream.readFully(DataInputStream.java:195)
> at java.io.DataInputStream.readFully(DataInputStream.java:169)
> at org.apache.hadoop.hbase.util.FSUtils.getClusterId(FSUtils.java:816)
> at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:474)
> at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:146)
> at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:126)
> at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:649)
> at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:182)
> at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1646)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)