You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Karthik Manimaran <ka...@gmail.com> on 2011/02/07 21:58:39 UTC
HBase startup issues
Hi,
My HBase setup was running fine for a couple of months and all of a sudden
the following issues has cropped up. The master will shutdown immediately
after startup. The Hadoop datanode is running fine and hdfs status is
Healthy. Any ideas on what could be happening and steps on how to fix/bypass
these errors and get HBase running again? Here are the relevant errors from
logs:
2011-02-07 14:39:45,643 WARN org.apache.hadoop.hdfs.DFSClient:
DataStreamer Exception: org.apache.hadoop.ipc.RemoteException:
java.io.IOException: File
/hbase/.META./1028785192/info/89629418893503418 could only be
replicated to 0 nodes, instead of 1
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:739)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy0.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy0.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2906)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2788)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2078)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2264)
2011-02-07 14:39:45,643 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block null bad datanode[0] nodes == null
2011-02-07 14:39:45,643 WARN org.apache.hadoop.hdfs.DFSClient: Could
not get block locations. Source file
"/hbase/.META./1028785192/info/89629418893503418" - Aborting...
2011-02-07 14:39:45,644 FATAL
org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Replay of hlog
required. Forcing server shutdown
org.apache.hadoop.hbase.DroppedSnapshotException: region: .META.,,1
at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:946)
at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:839)
at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:241)
at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:149)
Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException:
File /hbase/.META./1028785192/info/89629418893503418 could only be
replicated to 0 nodes, instead of 1
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:739)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy0.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy0.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2906)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2788)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2078)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2264)
2011-02-07 14:39:45,662 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
request=0.0, regions=73, stores=227, storefiles=184,
storefileIndexSize=1, memstoreSize=0, usedHeap=37, maxHeap=1993,
blockCacheSize=1962504, blockCacheFree=416130968, blockCacheCount=1,
blockCacheHitRatio=75
2011-02-07 14:39:45,662 INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher:
RegionServer:0.cacheFlusher exiting
2011-02-07 14:39:48,481 INFO org.apache.hadoop.ipc.HBaseServer:
Stopping server on 37024
2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer: IPC
Server handler 3 on 37024: exiting
2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer:
Stopping IPC Server listener on 37024
2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer: IPC
Server handler 0 on 37024: exiting
2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer: IPC
Server handler 1 on 37024: exiting
2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer: IPC
Server handler 2 on 37024: exiting
2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer: IPC
Server handler 5 on 37024: exiting
2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer: IPC
Server handler 7 on 37024: exiting
2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer: IPC
Server handler 9 on 37024: exiting
2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer: IPC
Server handler 4 on 37024: exiting
2011-02-07 14:39:48,483 INFO org.apache.hadoop.ipc.HBaseServer: IPC
Server handler 6 on 37024: exiting
2011-02-07 14:39:48,483 INFO org.apache.hadoop.ipc.HBaseServer: IPC
Server handler 8 on 37024: exiting
2011-02-07 14:39:48,488 INFO org.apache.hadoop.ipc.HBaseServer:
Stopping IPC Server Responder
2011-02-07 14:39:48,488 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping
infoServer
2011-02-07 14:39:48,490 DEBUG
org.apache.hadoop.hbase.regionserver.HLog: closing hlog writer in
hdfs://ec2-75-101-139-144.compute-1.amazonaws.com:9000/hbase/.logs/ip-10-212-78-95.ec2.internal,37024,1297107521809
2011-02-07 14:39:48,490 INFO
org.apache.hadoop.hbase.regionserver.LogFlusher:
RegionServer:0.logFlusher exiting
2011-02-07 14:39:48,490 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread:
RegionServer:0.compactor exiting
2011-02-07 14:39:48,490 INFO
org.apache.hadoop.hbase.regionserver.LogRoller: LogRoller exiting.
2011-02-07 14:39:48,490 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer$MajorCompactionChecker:
RegionServer:0.majorCompactionChecker exiting
2011-02-07 14:39:48,492 WARN org.apache.hadoop.hdfs.DFSClient:
DataStreamer Exception: org.apache.hadoop.ipc.RemoteException:
java.io.IOException: File
/hbase/.logs/ip-10-212-78-95.ec2.internal,37024,1297107521809/hlog.dat.1297107521949
could only be replicated to 0 nodes, instead of 1
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:739)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy0.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy0.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2906)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2788)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2078)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2264)
2011-02-07 14:39:48,492 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block null bad datanode[0] nodes == null
2011-02-07 14:39:48,492 WARN org.apache.hadoop.hdfs.DFSClient: Could
not get block locations. Source file
"/hbase/.logs/ip-10-212-78-95.ec2.internal,37024,1297107521809/hlog.dat.1297107521949"
- Aborting...
2011-02-07 14:39:48,501 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to close
log in abort
java.io.IOException: java.io.IOException: File
/hbase/.logs/ip-10-212-78-95.ec2.internal,37024,1297107521809/hlog.dat.1297107521949
could only be replicated to 0 nodes, instead of 1
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
at org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:631)
at java.lang.Thread.run(Thread.java:619)
2011-02-07 14:39:48,502 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: closing region
user_activity,,1288475114682
2011-02-07 14:39:48,502 DEBUG
org.apache.hadoop.hbase.regionserver.HRegion: Closing
user_activity,,1288475114682: compactions & flushes disabled
2011-02-07 14:39:48,502 DEBUG
org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for
region, no outstanding scanners on user_activity,,1288475114682
2011-02-07 14:39:48,502 DEBUG
org.apache.hadoop.hbase.regionserver.HRegion: No more row locks
outstanding on region user_activity,,1288475114682
2011-02-07 14:39:48,503 DEBUG
org.apache.hadoop.hbase.regionserver.Store: closed comments
2011-02-07 14:39:48,503 DEBUG
org.apache.hadoop.hbase.regionserver.Store: closed company_ratings
2011-02-07 14:39:48,504 DEBUG
org.apache.hadoop.hbase.regionserver.Store: closed details
2011-02-07 14:39:48,504 DEBUG
org.apache.hadoop.hbase.regionserver.Store: closed event
2011-02-07 14:39:48,505 DEBUG
org.apache.hadoop.hbase.regionserver.Store: closed event_details
2011-02-07 14:39:48,505 DEBUG
org.apache.hadoop.hbase.regionserver.Store: closed likes
2011-02-07 14:39:48,505 DEBUG
org.apache.hadoop.hbase.regionserver.Store: closed tags
2011-02-07 14:39:48,506 DEBUG
org.apache.hadoop.hbase.regionserver.Store: closed user_ratings
2011-02-07 14:39:48,506 INFO
org.apache.hadoop.hbase.regionserver.HRegion: Closed
user_activity,,1288475114682
2011-02-07 14:39:48,506 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: closing region
company_analytics_post_a_job_tab,,1288475078918
2011-02-07 14:39:48,506 DEBUG
org.apache.hadoop.hbase.regionserver.HRegion: Closing
company_analytics_post_a_job_tab,,1288475078918: compactions & flushes
disabled
2011-02-07 14:39:48,506 DEBUG
org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for
region, no outstanding scanners on
company_analytics_post_a_job_tab,,1288475078918
2011-02-07 14:39:48,506 DEBUG
org.apache.hadoop.hbase.regionserver.HRegion: No more row locks
outstanding on region company_analytics_post_a_job_tab,,1288475078918
2011-02-07 14:39:48,506 DEBUG
org.apache.hadoop.hbase.regionserver.Store: closed system_details
2011-02-07 14:39:48,506 DEBUG
org.apache.hadoop.hbase.regionserver.Store: closed viewer_details
Thanks,
Karthik Manimaran.
Re: HBase startup issues
Posted by Karthik Manimaran <ka...@gmail.com>.
Thanks for the hint J-D. The filesystem got filled up causing this
issue. All clear now.
regards,
Karthik Manimaran.
On 02/07/2011 01:00 PM, Jean-Daniel Cryans wrote:
> Your HDFS doesn't look healthy to me, it doesn't accept new blocks!
> Check the datanode / namenode logs and see if any exception in there.
>
> J-D
>
> On Mon, Feb 7, 2011 at 12:58 PM, Karthik Manimaran
> <ka...@gmail.com> wrote:
>
>> Hi,
>>
>> My HBase setup was running fine for a couple of months and all of a sudden
>> the following issues has cropped up. The master will shutdown immediately
>> after startup. The Hadoop datanode is running fine and hdfs status is
>> Healthy. Any ideas on what could be happening and steps on how to fix/bypass
>> these errors and get HBase running again? Here are the relevant errors from
>> logs:
>>
>> 2011-02-07 14:39:45,643 WARN org.apache.hadoop.hdfs.DFSClient:
>> DataStreamer Exception: org.apache.hadoop.ipc.RemoteException:
>> java.io.IOException: File
>> /hbase/.META./1028785192/info/89629418893503418 could only be
>> replicated to 0 nodes, instead of 1
>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
>> at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
>> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>>
>> at org.apache.hadoop.ipc.Client.call(Client.java:739)
>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
>> at $Proxy0.addBlock(Unknown Source)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> at $Proxy0.addBlock(Unknown Source)
>> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2906)
>> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2788)
>> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2078)
>> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2264)
>>
>> 2011-02-07 14:39:45,643 WARN org.apache.hadoop.hdfs.DFSClient: Error
>> Recovery for block null bad datanode[0] nodes == null
>> 2011-02-07 14:39:45,643 WARN org.apache.hadoop.hdfs.DFSClient: Could
>> not get block locations. Source file
>> "/hbase/.META./1028785192/info/89629418893503418" - Aborting...
>> 2011-02-07 14:39:45,644 FATAL
>> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Replay of hlog
>> required. Forcing server shutdown
>> org.apache.hadoop.hbase.DroppedSnapshotException: region: .META.,,1
>> at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:946)
>> at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:839)
>> at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:241)
>> at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:149)
>> Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException:
>> File /hbase/.META./1028785192/info/89629418893503418 could only be
>> replicated to 0 nodes, instead of 1
>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
>> at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
>> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>>
>> at org.apache.hadoop.ipc.Client.call(Client.java:739)
>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
>> at $Proxy0.addBlock(Unknown Source)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> at $Proxy0.addBlock(Unknown Source)
>> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2906)
>> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2788)
>> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2078)
>> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2264)
>> 2011-02-07 14:39:45,662 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
>> request=0.0, regions=73, stores=227, storefiles=184,
>> storefileIndexSize=1, memstoreSize=0, usedHeap=37, maxHeap=1993,
>> blockCacheSize=1962504, blockCacheFree=416130968, blockCacheCount=1,
>> blockCacheHitRatio=75
>> 2011-02-07 14:39:45,662 INFO
>> org.apache.hadoop.hbase.regionserver.MemStoreFlusher:
>> RegionServer:0.cacheFlusher exiting
>> 2011-02-07 14:39:48,481 INFO org.apache.hadoop.ipc.HBaseServer:
>> Stopping server on 37024
>> 2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>> Server handler 3 on 37024: exiting
>> 2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer:
>> Stopping IPC Server listener on 37024
>> 2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>> Server handler 0 on 37024: exiting
>> 2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>> Server handler 1 on 37024: exiting
>> 2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>> Server handler 2 on 37024: exiting
>> 2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>> Server handler 5 on 37024: exiting
>> 2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>> Server handler 7 on 37024: exiting
>> 2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>> Server handler 9 on 37024: exiting
>> 2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>> Server handler 4 on 37024: exiting
>> 2011-02-07 14:39:48,483 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>> Server handler 6 on 37024: exiting
>> 2011-02-07 14:39:48,483 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>> Server handler 8 on 37024: exiting
>> 2011-02-07 14:39:48,488 INFO org.apache.hadoop.ipc.HBaseServer:
>> Stopping IPC Server Responder
>> 2011-02-07 14:39:48,488 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping
>> infoServer
>> 2011-02-07 14:39:48,490 DEBUG
>> org.apache.hadoop.hbase.regionserver.HLog: closing hlog writer in
>> hdfs://ec2-75-101-139-144.compute-1.amazonaws.com:9000/hbase/.logs/ip-10-212-78-95.ec2.internal,37024,1297107521809
>> 2011-02-07 14:39:48,490 INFO
>> org.apache.hadoop.hbase.regionserver.LogFlusher:
>> RegionServer:0.logFlusher exiting
>> 2011-02-07 14:39:48,490 INFO
>> org.apache.hadoop.hbase.regionserver.CompactSplitThread:
>> RegionServer:0.compactor exiting
>> 2011-02-07 14:39:48,490 INFO
>> org.apache.hadoop.hbase.regionserver.LogRoller: LogRoller exiting.
>> 2011-02-07 14:39:48,490 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer$MajorCompactionChecker:
>> RegionServer:0.majorCompactionChecker exiting
>> 2011-02-07 14:39:48,492 WARN org.apache.hadoop.hdfs.DFSClient:
>> DataStreamer Exception: org.apache.hadoop.ipc.RemoteException:
>> java.io.IOException: File
>> /hbase/.logs/ip-10-212-78-95.ec2.internal,37024,1297107521809/hlog.dat.1297107521949
>> could only be replicated to 0 nodes, instead of 1
>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
>> at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
>> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>>
>> at org.apache.hadoop.ipc.Client.call(Client.java:739)
>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
>> at $Proxy0.addBlock(Unknown Source)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> at $Proxy0.addBlock(Unknown Source)
>> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2906)
>> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2788)
>> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2078)
>> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2264)
>>
>> 2011-02-07 14:39:48,492 WARN org.apache.hadoop.hdfs.DFSClient: Error
>> Recovery for block null bad datanode[0] nodes == null
>> 2011-02-07 14:39:48,492 WARN org.apache.hadoop.hdfs.DFSClient: Could
>> not get block locations. Source file
>> "/hbase/.logs/ip-10-212-78-95.ec2.internal,37024,1297107521809/hlog.dat.1297107521949"
>> - Aborting...
>> 2011-02-07 14:39:48,501 ERROR
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to close
>> log in abort
>> java.io.IOException: java.io.IOException: File
>> /hbase/.logs/ip-10-212-78-95.ec2.internal,37024,1297107521809/hlog.dat.1297107521949
>> could only be replicated to 0 nodes, instead of 1
>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
>> at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
>> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>>
>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>> at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
>> at org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
>> at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:631)
>> at java.lang.Thread.run(Thread.java:619)
>> 2011-02-07 14:39:48,502 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegionServer: closing region
>> user_activity,,1288475114682
>> 2011-02-07 14:39:48,502 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: Closing
>> user_activity,,1288475114682: compactions& flushes disabled
>> 2011-02-07 14:39:48,502 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for
>> region, no outstanding scanners on user_activity,,1288475114682
>> 2011-02-07 14:39:48,502 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: No more row locks
>> outstanding on region user_activity,,1288475114682
>> 2011-02-07 14:39:48,503 DEBUG
>> org.apache.hadoop.hbase.regionserver.Store: closed comments
>> 2011-02-07 14:39:48,503 DEBUG
>> org.apache.hadoop.hbase.regionserver.Store: closed company_ratings
>> 2011-02-07 14:39:48,504 DEBUG
>> org.apache.hadoop.hbase.regionserver.Store: closed details
>> 2011-02-07 14:39:48,504 DEBUG
>> org.apache.hadoop.hbase.regionserver.Store: closed event
>> 2011-02-07 14:39:48,505 DEBUG
>> org.apache.hadoop.hbase.regionserver.Store: closed event_details
>> 2011-02-07 14:39:48,505 DEBUG
>> org.apache.hadoop.hbase.regionserver.Store: closed likes
>> 2011-02-07 14:39:48,505 DEBUG
>> org.apache.hadoop.hbase.regionserver.Store: closed tags
>> 2011-02-07 14:39:48,506 DEBUG
>> org.apache.hadoop.hbase.regionserver.Store: closed user_ratings
>> 2011-02-07 14:39:48,506 INFO
>> org.apache.hadoop.hbase.regionserver.HRegion: Closed
>> user_activity,,1288475114682
>> 2011-02-07 14:39:48,506 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegionServer: closing region
>> company_analytics_post_a_job_tab,,1288475078918
>> 2011-02-07 14:39:48,506 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: Closing
>> company_analytics_post_a_job_tab,,1288475078918: compactions& flushes
>> disabled
>> 2011-02-07 14:39:48,506 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for
>> region, no outstanding scanners on
>> company_analytics_post_a_job_tab,,1288475078918
>> 2011-02-07 14:39:48,506 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: No more row locks
>> outstanding on region company_analytics_post_a_job_tab,,1288475078918
>> 2011-02-07 14:39:48,506 DEBUG
>> org.apache.hadoop.hbase.regionserver.Store: closed system_details
>> 2011-02-07 14:39:48,506 DEBUG
>> org.apache.hadoop.hbase.regionserver.Store: closed viewer_details
>>
>>
>> Thanks,
>> Karthik Manimaran.
>>
>>
Re: HBase startup issues
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Your HDFS doesn't look healthy to me, it doesn't accept new blocks!
Check the datanode / namenode logs and see if any exception in there.
J-D
On Mon, Feb 7, 2011 at 12:58 PM, Karthik Manimaran
<ka...@gmail.com> wrote:
> Hi,
>
> My HBase setup was running fine for a couple of months and all of a sudden
> the following issues has cropped up. The master will shutdown immediately
> after startup. The Hadoop datanode is running fine and hdfs status is
> Healthy. Any ideas on what could be happening and steps on how to fix/bypass
> these errors and get HBase running again? Here are the relevant errors from
> logs:
>
> 2011-02-07 14:39:45,643 WARN org.apache.hadoop.hdfs.DFSClient:
> DataStreamer Exception: org.apache.hadoop.ipc.RemoteException:
> java.io.IOException: File
> /hbase/.META./1028785192/info/89629418893503418 could only be
> replicated to 0 nodes, instead of 1
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:739)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at $Proxy0.addBlock(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at $Proxy0.addBlock(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2906)
> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2788)
> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2078)
> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2264)
>
> 2011-02-07 14:39:45,643 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block null bad datanode[0] nodes == null
> 2011-02-07 14:39:45,643 WARN org.apache.hadoop.hdfs.DFSClient: Could
> not get block locations. Source file
> "/hbase/.META./1028785192/info/89629418893503418" - Aborting...
> 2011-02-07 14:39:45,644 FATAL
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Replay of hlog
> required. Forcing server shutdown
> org.apache.hadoop.hbase.DroppedSnapshotException: region: .META.,,1
> at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:946)
> at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:839)
> at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:241)
> at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:149)
> Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException:
> File /hbase/.META./1028785192/info/89629418893503418 could only be
> replicated to 0 nodes, instead of 1
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:739)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at $Proxy0.addBlock(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at $Proxy0.addBlock(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2906)
> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2788)
> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2078)
> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2264)
> 2011-02-07 14:39:45,662 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
> request=0.0, regions=73, stores=227, storefiles=184,
> storefileIndexSize=1, memstoreSize=0, usedHeap=37, maxHeap=1993,
> blockCacheSize=1962504, blockCacheFree=416130968, blockCacheCount=1,
> blockCacheHitRatio=75
> 2011-02-07 14:39:45,662 INFO
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher:
> RegionServer:0.cacheFlusher exiting
> 2011-02-07 14:39:48,481 INFO org.apache.hadoop.ipc.HBaseServer:
> Stopping server on 37024
> 2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server handler 3 on 37024: exiting
> 2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer:
> Stopping IPC Server listener on 37024
> 2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server handler 0 on 37024: exiting
> 2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server handler 1 on 37024: exiting
> 2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server handler 2 on 37024: exiting
> 2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server handler 5 on 37024: exiting
> 2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server handler 7 on 37024: exiting
> 2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server handler 9 on 37024: exiting
> 2011-02-07 14:39:48,482 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server handler 4 on 37024: exiting
> 2011-02-07 14:39:48,483 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server handler 6 on 37024: exiting
> 2011-02-07 14:39:48,483 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server handler 8 on 37024: exiting
> 2011-02-07 14:39:48,488 INFO org.apache.hadoop.ipc.HBaseServer:
> Stopping IPC Server Responder
> 2011-02-07 14:39:48,488 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping
> infoServer
> 2011-02-07 14:39:48,490 DEBUG
> org.apache.hadoop.hbase.regionserver.HLog: closing hlog writer in
> hdfs://ec2-75-101-139-144.compute-1.amazonaws.com:9000/hbase/.logs/ip-10-212-78-95.ec2.internal,37024,1297107521809
> 2011-02-07 14:39:48,490 INFO
> org.apache.hadoop.hbase.regionserver.LogFlusher:
> RegionServer:0.logFlusher exiting
> 2011-02-07 14:39:48,490 INFO
> org.apache.hadoop.hbase.regionserver.CompactSplitThread:
> RegionServer:0.compactor exiting
> 2011-02-07 14:39:48,490 INFO
> org.apache.hadoop.hbase.regionserver.LogRoller: LogRoller exiting.
> 2011-02-07 14:39:48,490 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer$MajorCompactionChecker:
> RegionServer:0.majorCompactionChecker exiting
> 2011-02-07 14:39:48,492 WARN org.apache.hadoop.hdfs.DFSClient:
> DataStreamer Exception: org.apache.hadoop.ipc.RemoteException:
> java.io.IOException: File
> /hbase/.logs/ip-10-212-78-95.ec2.internal,37024,1297107521809/hlog.dat.1297107521949
> could only be replicated to 0 nodes, instead of 1
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:739)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at $Proxy0.addBlock(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at $Proxy0.addBlock(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2906)
> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2788)
> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2078)
> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2264)
>
> 2011-02-07 14:39:48,492 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block null bad datanode[0] nodes == null
> 2011-02-07 14:39:48,492 WARN org.apache.hadoop.hdfs.DFSClient: Could
> not get block locations. Source file
> "/hbase/.logs/ip-10-212-78-95.ec2.internal,37024,1297107521809/hlog.dat.1297107521949"
> - Aborting...
> 2011-02-07 14:39:48,501 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to close
> log in abort
> java.io.IOException: java.io.IOException: File
> /hbase/.logs/ip-10-212-78-95.ec2.internal,37024,1297107521809/hlog.dat.1297107521949
> could only be replicated to 0 nodes, instead of 1
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
> at org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:631)
> at java.lang.Thread.run(Thread.java:619)
> 2011-02-07 14:39:48,502 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegionServer: closing region
> user_activity,,1288475114682
> 2011-02-07 14:39:48,502 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegion: Closing
> user_activity,,1288475114682: compactions & flushes disabled
> 2011-02-07 14:39:48,502 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for
> region, no outstanding scanners on user_activity,,1288475114682
> 2011-02-07 14:39:48,502 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegion: No more row locks
> outstanding on region user_activity,,1288475114682
> 2011-02-07 14:39:48,503 DEBUG
> org.apache.hadoop.hbase.regionserver.Store: closed comments
> 2011-02-07 14:39:48,503 DEBUG
> org.apache.hadoop.hbase.regionserver.Store: closed company_ratings
> 2011-02-07 14:39:48,504 DEBUG
> org.apache.hadoop.hbase.regionserver.Store: closed details
> 2011-02-07 14:39:48,504 DEBUG
> org.apache.hadoop.hbase.regionserver.Store: closed event
> 2011-02-07 14:39:48,505 DEBUG
> org.apache.hadoop.hbase.regionserver.Store: closed event_details
> 2011-02-07 14:39:48,505 DEBUG
> org.apache.hadoop.hbase.regionserver.Store: closed likes
> 2011-02-07 14:39:48,505 DEBUG
> org.apache.hadoop.hbase.regionserver.Store: closed tags
> 2011-02-07 14:39:48,506 DEBUG
> org.apache.hadoop.hbase.regionserver.Store: closed user_ratings
> 2011-02-07 14:39:48,506 INFO
> org.apache.hadoop.hbase.regionserver.HRegion: Closed
> user_activity,,1288475114682
> 2011-02-07 14:39:48,506 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegionServer: closing region
> company_analytics_post_a_job_tab,,1288475078918
> 2011-02-07 14:39:48,506 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegion: Closing
> company_analytics_post_a_job_tab,,1288475078918: compactions & flushes
> disabled
> 2011-02-07 14:39:48,506 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for
> region, no outstanding scanners on
> company_analytics_post_a_job_tab,,1288475078918
> 2011-02-07 14:39:48,506 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegion: No more row locks
> outstanding on region company_analytics_post_a_job_tab,,1288475078918
> 2011-02-07 14:39:48,506 DEBUG
> org.apache.hadoop.hbase.regionserver.Store: closed system_details
> 2011-02-07 14:39:48,506 DEBUG
> org.apache.hadoop.hbase.regionserver.Store: closed viewer_details
>
>
> Thanks,
> Karthik Manimaran.
>