You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Andrew Purtell (JIRA)" <ji...@apache.org> on 2011/03/08 18:54:59 UTC
[jira] Reopened: (HBASE-1960) Master should wait for DFS to come up
when creating hbase.version
[ https://issues.apache.org/jira/browse/HBASE-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell reopened HBASE-1960:
-----------------------------------
Got this again with 0.90:
{noformat}
2011-03-08 08:41:44,428 INFO org.apache.hadoop.hbase.master.ActiveMasterManager:
Master=ip-10-160-166-143.us-west-1.compute.internal:60000
2011-03-08 08:41:44,780 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exce
ption: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /hbase/h
base.version could only be replicated to 0 nodes, instead of 1
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBloc
k(FSNamesystem.java:1416)
at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.jav
a:643)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1416)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1412)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma
tion.java:1116)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1410)
at org.apache.hadoop.ipc.Client.call(Client.java:1104)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226)
at $Proxy7.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryI
nvocationHandler.java:82)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocat
ionHandler.java:59)
at $Proxy7.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock
(DFSClient.java:3241)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStrea
m(DFSClient.java:3110)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1900(DFSClien
t.java:2360)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFS
Client.java:2555)
{noformat}
> Master should wait for DFS to come up when creating hbase.version
> -----------------------------------------------------------------
>
> Key: HBASE-1960
> URL: https://issues.apache.org/jira/browse/HBASE-1960
> Project: HBase
> Issue Type: Bug
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Priority: Minor
> Fix For: 0.20.3, 0.90.0
>
> Attachments: HBASE-1960.patch
>
>
> The master does not wait for DFS to come up in the circumstance where the DFS master is started for the first time after format and no datanodes have been started yet.
> {noformat}
> 2009-11-07 11:47:28,115 INFO org.apache.hadoop.hbase.master.HMaster: vmName=Java HotSpot(TM) 64-Bit Server VM, vmVendor=Sun Microsystems Inc., vmVersion=14.2-b01
> 2009-11-07 11:47:28,116 INFO org.apache.hadoop.hbase.master.HMaster: vmInputArguments=[-Xmx1000m, -XX:+HeapDumpOnOutOfMemoryError, -XX:+UseConcMarkSweepGC, -XX:+CMSIncrementalMode, -Dhbase.log.dir=/mnt/hbase/logs, -Dhbase.log.file=hbase-root-master-ip-10-242-15-159.log, -Dhbase.home.dir=/usr/local/hbase-0.20.1/bin/.., -Dhbase.id.str=root, -Dhbase.root.logger=INFO,DRFA, -Djava.library.path=/usr/local/hbase-0.20.1/bin/../lib/native/Linux-amd64-64]
> 2009-11-07 11:47:28,247 INFO org.apache.hadoop.hbase.master.HMaster: My address is ip-10-242-15-159.ec2.internal:60000
> 2009-11-07 11:47:28,728 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /hbase/hbase.version could only be replicated to 0 nodes, instead of 1
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
> [...]
> 2009-11-07 11:47:28,728 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block null bad datanode[0] nodes == null
> 2009-11-07 11:47:28,728 WARN org.apache.hadoop.hdfs.DFSClient: Could not get block locations. Source file "/hbase/hbase.version" - Aborting...
> 2009-11-07 11:47:28,729 FATAL org.apache.hadoop.hbase.master.HMaster: Not starting HMaster because:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /hbase/hbase.version could only be replicated to 0 nodes, instead of 1
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
> {noformat}
> Should probably sleep and retry the write a few times.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira