You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Lance Riedel <la...@gmail.com> on 2010/12/09 20:27:42 UTC
Crashing Region Servers
We have a 6 node cluster, 5 with region serves. 2 of the region servers have been stable for days, but 3 of them keep crashing. Here are the logs around around when the crash occurs. (btw, we are shoving approximately the twitter firehose into hbase via flume) I'm an hbase newbie, but I have been reading. Not sure what else is needed to help debug. When the problem occurs, it is concurrent with issues that appear all the way down in the data node. Also, interesting to note, all 3 servers seem to fail differently somewhat:
Servers (6):
00.hadoop is the master
01.hadoop, 03.hadoop (there is no 02.hadoop - flaky machine that got taken out) - are stable Region Servers
04-06.hadoop crash at different times, very different logs, Region Servers
Master Attributes:
HBase Version 0.89.20100924+28, r
Hadoop Version 0.20.2+737, r98c55c28258aa6f42250569bd7fa431ac657bdbd
HBase Root Directory hdfs://00.hadoop.****:54310/hbase
Load average
103.0
Average number of regions per regionserver. Naive computation.
Regions On FS
577
(This is after recently restarting 04-06)
Address Start Code Load
01.hadoop:60030 1291268910638 requests=223, regions=118, usedHeap=519, maxHeap=987
03.hadoop:60030 1291269219610 requests=23, regions=111, usedHeap=862, maxHeap=987
04.hadoop:60030 1291910665912 requests=169, regions=82, usedHeap=194, maxHeap=888
05.hadoop:60030 1291909584060 requests=232, regions=110, usedHeap=477, maxHeap=888
06.hadoop60030 1291909723787 requests=99, regions=94, usedHeap=394, maxHeap=888
Total:
servers: 5
requests=746, regions=515
Following are various logs around the time of the failures for 04-06
************************************************************************************
04.hadoop
************************************************************************************
REGION SERVER ->
2010-12-09 05:01:58,640 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region article,a83858a08f2270d319f75a7b43c756c2453988e7,1291811929868.58f6d9fc80
c78f3ca490b0280b4f1226. in 0sec
2010-12-09 05:04:30,898 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.46 MB, free=176.33 MB, max=177.79 MB, blocks=0, accesses=200583, hits=0, hitRatio=
0.00%%, evictions=0, evicted=0, evictedPerRun=NaN
2010-12-09 05:04:59,612 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs -- HDFS-200
2010-12-09 05:04:59,618 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Roll /hbase/.logs/04.hadoop.****,60020,1291858770125/10.100.154.103%3A60020.1291870585253, entries=5318
6, filesize=63776311. New hlog /hbase/.logs/04.hadoop.****,60020,1291858770125/10.100.154.103%3A60020.1291871099582
2010-12-09 05:08:02,033 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: Found 1 hlogs to remove out of total 6; oldest outstanding sequenceid is 32891801 from region article
,4b2039b791e894dd479b90661ca97087f61645d3,1291813201039.7dff47c63bb5648bb1ee6670c60553a5.
2010-12-09 05:08:02,034 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler 13 on 60020 took 182416ms appending an edit to hlog; editcount=0
2010-12-09 05:08:02,066 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: moving old hlog file /hbase/.logs/04.hadoop.****,60020,1291858770125/10.100.154.103%3A60020.12918679476
65 whose highest sequenceid is 32880227 to /hbase/.oldlogs/10.100.154.103%3A60020.1291867947665
2010-12-09 05:08:02,066 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 182936ms for sessionid 0x12ca58c08ff0048, closing socket conn
ection and attempting reconnect
2010-12-09 05:08:02,089 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: Aborting region server serverName=04.hadoop.****,60020,1291858770125, load=(requests=24, regions=
64, usedHeap=548, maxHeap=888): Unhandled exception
org.apache.hadoop.hbase.YouAreDeadException: org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing 04.hadoop.****,60020,1291858770125 as dead s
erver
at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:217)
at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:271)
at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:744)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:561)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
at org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
at org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
at org.apache.hadoop.hbase.regionserver.HRegionServer.doRun(HRegionServer.java:635)
at org.apache.hadoop.hbase.regionserver.HRegionServer.access$000(HRegionServer.java:126)
at org.apache.hadoop.hbase.regionserver.HRegionServer$1.run(HRegionServer.java:518)
at org.apache.hadoop.hbase.regionserver.HRegionServer$1.run(HRegionServer.java:516)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:337)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1046)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:516)
at java.lang.Thread.run(Thread.java:662)
2010-12-09 05:08:02,090 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: request=0.0, regions=64, stores=64, storefiles=136, storefileIndexSize=27, memsto
reSize=353, compactionQueueSize=0, usedHeap=549, maxHeap=888, blockCacheSize=1530552, blockCacheFree=184893160, blockCacheCount=0, blockCacheHitRatio=0
2010-12-09 05:08:02,090 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60020
2010-12-09 05:08:02,090 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 0 on 60020: exiting2010-12-09 05:08:02,090 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server listener on 60020
MASTER HBASE ->
2010-12-09 05:05:06,072 INFO org.apache.hadoop.hbase.master.BaseScanner: All 1 .META. region(s) scanned
2010-12-09 05:05:40,004 INFO org.apache.hadoop.hbase.master.ServerManager: 04.hadoop.jive,60020,1291858770125 znode expired
2010-12-09 05:05:40,005 DEBUG org.apache.hadoop.hbase.master.ServerManager: Added=04.hadoop.jive,60020,1291858770125 to dead servers, added shutdown processing operation
2010-12-09 05:05:40,005 DEBUG org.apache.hadoop.hbase.master.RegionServerOperationQueue: Processing todo: ProcessServerShutdown of 04.hadoop.jive,60020,12918587701252010-12-09 05:05:40,005 INFO org.apache.hadoop.hbase.master.RegionServerOperation: Process shutdown of server 04.hadoop.jive,60020,1291858770125: logSplit: false, rootRescanned: f
alse, numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
2010-12-09 05:05:40,008 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Splitting 7 hlog(s) in hdfs://00.hadoop.jive:54310/hbase/.logs/04.hadoop.jive,60020,12918587701252010-12-09 05:05:40,008 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: Splitting hlog 1 of 7: hdfs://00.hadoop.jive:54310/hbase/.logs/04.hadoop.jive,60020,1291858770125/10.1
00.154.103%3A60020.1291867947665, length=637577092010-12-09 05:05:40,008 INFO org.apache.hadoop.hbase.util.FSUtils: Recovering filehdfs://00.hadoop.jive:54310/hbase/.logs/04.hadoop.jive,60020,1291858770125/10.100.154.103%3A60020
.1291867947665
NAMENODE ->
2010-12-09 05:08:02,471 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 10.100.159.13:50010 is added to blk_1531008743226086399_251615 size 63757709
2010-12-09 05:08:02,473 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 54310, call complete(/hbase/.logs/04.hadoop.jive,60020,1291858770125/10.100.154.103%3A60020.1291867947665, DFSClient_hb_m_10.194.194.79:60000_1291788452343) from 10.194.194.79:44117: error: org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /hbase/.logs/04.hadoop.jive,60020,1291858770125/10.100.154.103%3A60020.1291867947665 File does not exist. [Lease. Holder: DFSClient_hb_m_10.194.194.79:60000_1291788452343, pendingcreates: 1]
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /hbase/.logs/04.hadoop.jive,60020,1291858770125/10.100.154.103%3A60020.1291867947665 File does not exist. [Lease. Holder: DFSClient_hb_m_10.194.194.79:60000_1291788452343, pendingcreates: 1]
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1488)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1479)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:1534)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:1522)
at org.apache.hadoop.hdfs.server.namenode.NameNode.complete(NameNode.java:610)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:528)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1319)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1315)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1313)
2010-12-09 05:08:04,206 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 10.99.97.106:50010 is added to blk_-734280257049179934_251614 size 7330
DATANODE ->
2010-12-09 05:08:02,212 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in BlockReceiver.lastNodeRun: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100) at sun.nio.ch.IOUtil.write(IOUtil.java:71)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142) at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107) at java.io.DataOutputStream.writeLong(DataOutputStream.java:207)
at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.write(DataTransferProtocol.java:133) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.lastDataNodeRun(BlockReceiver.java:840)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:870)
at java.lang.Thread.run(Thread.java:662)
2010-12-09 05:08:02,213 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: checkDiskError: exception:
2010-12-09 05:08:02,213 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: checkDiskError: exception: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100)
at sun.nio.ch.IOUtil.write(IOUtil.java:71)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
at java.io.DataOutputStream.writeLong(DataOutputStream.java:207)
at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.write(DataTransferProtocol.java:133)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.lastDataNodeRun(BlockReceiver.java:840)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:870)
at java.lang.Thread.run(Thread.java:662)
2010-12-09 05:08:02,221 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder blk_-8817504198034990390_251613 0 Exception java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100)
at sun.nio.ch.IOUtil.write(IOUtil.java:71)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
at java.io.DataOutputStream.writeLong(DataOutputStream.java:207)
at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.write(DataTransferProtocol.java:133)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.lastDataNodeRun(BlockReceiver.java:840)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:870)
at java.lang.Thread.run(Thread.java:662)
2010-12-09 05:08:02,221 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block blk_-8817504198034990390_251613 terminating
2010-12-09 05:08:02,224 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock blk_-8817504198034990390_251613 received exception java.io.IOException: Connection reset by peer
2010-12-09 05:08:02,224 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_4505306716668305567_250752 file /dist1/data/hadoop-data/current/subdir19/blk_4505306716668305567 for deletion
2010-12-09 05:08:02,226 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(10.100.154.103:50010, storageID=DS-866555524-10.100.154.103-50010-1291262739506, infoPort=50075, ipcPort=50020):DataXceiver
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:237)
at sun.nio.ch.IOUtil.read(IOUtil.java:210)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:55)
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at java.io.DataInputStream.read(DataInputStream.java:132)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:267)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:357)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:378)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:534)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:417)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:122)
2010-12-09 05:08:02,226 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block blk_1422641301942366074_250732 at file /dist1/data/hadoop-data/current/subdir19/blk_1422641301942366074
2010-12-09 05:08:02,227 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block blk_4505306716668305567_250752 at file /dist1/data/hadoop-data/current/subdir19/blk_4505306716668305567
************************************************************************************
05.hadoop
************************************************************************************
REGION SERVER ->
2010-12-09 04:29:25,047 DEBUG org.apache.hadoop.hbase.regionserver.Store: Compaction size of data: 250.5m; Skipped 1 file(s), size: 206896367
2010-12-09 04:29:25,047 INFO org.apache.hadoop.hbase.regionserver.Store: Started compaction of 2 file(s) in data of article,060c4b75067378c045eead9edec2d4fb7d8f3e5e,1291812563166.95e92e97823e11e57850720a63433a62. into hdfs://00.hadoop.****:54310/hbase/article/95e92e97823e11e57850720a63433a62/.tmp, sequenceid=33050221
2010-12-09 04:29:26,328 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 2 file(s) in data of article,060c4b75067378c045eead9edec2d4fb7d8f3e5e,1291812563166.95e92e97823e11e57850720a63433a62.; new storefile is hdfs://00.hadoop.****:54310/hbase/article/95e92e97823e11e57850720a63433a62/data/2787768957794924068; store size is 250.5m
2010-12-09 04:29:26,328 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region article,060c4b75067378c045eead9edec2d4fb7d8f3e5e,1291812563166.95e92e97823e11e57850720a63433a62. in 1sec
2010-12-09 04:29:26,328 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on region article,ed0cdc7ae6db73432e2ca47c973d545bee11ddd8,1291813400028.85a8d31d099f409dd36f3bcc4be1eba6.
2010-12-09 04:29:26,330 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region article,ed0cdc7ae6db73432e2ca47c973d545bee11ddd8,1291813400028.85a8d31d099f409dd36f3bcc4be1eba6. in 0sec
2010-12-09 04:32:33,081 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGIONSERVER_STOP
2010-12-09 04:32:33,081 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60020
2010-12-09 04:32:33,089 WARN org.apache.hadoop.hdfs.DFSClient: DFSOutputStream ResponseProcessor exception for block blk_2012253533858803341_250248java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:180)
at java.io.DataInputStream.readLong(DataInputStream.java:399)
at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:120)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2621)
2010-12-09 04:32:33,089 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x12ca58c08ff0049, likely server has closed socket, closing sock
et connection and attempting reconnect
2010-12-09 04:32:33,089 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler 5 on 60020 took 75446ms appending an edit to hlog; editcount=50505
2010-12-09 04:32:33,090 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 18 on 60020: exiting
2010-12-09 04:32:33,139 INFO org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_2012253533858803341_250248 waiting for responder to exit.
2010-12-09 04:32:33,139 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 8 on 60020: exiting
2010-12-09 04:32:33,139 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 1 on 60020: exiting
2010-12-09 04:32:33,139 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 10 on 60020: exiting
2010-12-09 04:32:33,140 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 0 on 60020: exiting2010-12-09 04:32:33,140 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 5 on 60020: exiting
2010-12-09 04:32:33,139 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 8 on 60020: exiting2010-12-09 04:32:33,140 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 4 on 60020: exiting
2010-12-09 04:32:33,140 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 3 on 60020: exiting2010-12-09 04:32:33,142 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 7 on 60020: exiting
2010-12-09 04:32:33,142 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 9 on 60020: exiting2010-12-09 04:32:33,143 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
2010-12-09 04:32:33,143 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 2 on 60020: exiting
2010-12-09 04:32:33,143 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server Responder
2010-12-09 04:32:33,143 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 6 on 60020: exiting
2010-12-09 04:32:33,144 INFO org.mortbay.log: Stopped SelectChannelConnector@0.0.0.0:60030
NAMENODE ->
2010-12-09 04:32:08,408 WARN org.apache.hadoop.hdfs.StateChange: DIR* NameSystem.startFile: failed to create file /hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555 for DFSClient_hb_m_10.194.194.79:60000_1291788452343 on client 10.194.194.79, because this file is already being created by NN_Recovery on 10.100.159.13
2010-12-09 04:32:08,408 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 54310, call append(/hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555, DFSClient_hb_m_10.194.194.79:60000_1291788452343) from 10.194.194.79:59118: error: org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to create file /hba
se/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555 for DFSClient_hb_m_10.194.194.79:60000_1291788452343 on client 10.194.194.79, because this file is already being created by NN_Recovery on 10.100.159.13
org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to create file /hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555 for DFSClient_hb_m_10.194.194.79:60000_1291788452343 on client 10.194.194.79, because this file is already being created by NN_Recovery on 10.100.159.13
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1194) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:1282)
at org.apache.hadoop.hdfs.server.namenode.NameNode.append(NameNode.java:541) at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:528) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1319)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1315) at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1313)2010-12-09 04:32:09,411 WARN org.apache.hadoop.hdfs.StateChange: DIR* NameSystem.startFile: failed to create file /hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A6
0020.1291868481555 for DFSClient_hb_m_10.194.194.79:60000_1291788452343 on client 10.194.194.79, because this file is already being created by NN_Recovery on 10.100.159.132010-12-09 04:32:09,411 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 54310, call append(/hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868
481555, DFSClient_hb_m_10.194.194.79:60000_1291788452343) from 10.194.194.79:59118: error: org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to create file /hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555 for DFSClient_hb_m_10.194.194.79:60000_1291788452343 on client 10.194.194.79, because this file is
already being created by NN_Recovery on 10.100.159.13org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to create file /hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555 for DFSClie
nt_hb_m_10.194.194.79:60000_1291788452343 on client 10.194.194.79, because this file is already being created by NN_Recovery on 10.100.159.13 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1194)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:1282) at org.apache.hadoop.hdfs.server.namenode.NameNode.append(NameNode.java:541)
at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:528)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1319) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1315)
DATANODE
2010-12-09 04:32:09,534 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder blk_2012253533858803341_250248 1 : Thread is interrupted.
2010-12-09 04:32:09,534 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(10.100.159.13:50010, storageID=DS-145997007-10.100.159.13-50010-1291262743182, infoPort=50075, ipcPort=50020):DataXceiver
java.io.IOException: Interrupted receiveBlock
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:579)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:417)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:122)
2010-12-09 04:32:09,534 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 1 for block blk_2012253533858803341_250248 terminating
************************************************************************************
06.hadoop
************************************************************************************
REGION SERVER ->
2010-12-09 06:17:47,215 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
2010-12-09 06:17:47,430 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
2010-12-09 06:17:47,549 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
2010-12-09 06:17:47,557 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
2010-12-09 06:17:47,563 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
2010-12-09 06:17:48,619 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
2010-12-09 06:17:48,619 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
2010-12-09 06:17:48,619 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
2010-12-09 06:17:48,619 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
2010-12-09 06:17:48,619 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
2010-12-09 06:17:49,703 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
2010-12-09 06:17:49,703 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
2010-12-09 06:17:49,703 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
2010-12-09 06:17:49,705 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
2010-12-09 06:17:49,729 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
A bunch of these ^
Then a little later::
....
2010-12-09 06:23:23,561 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction requested for region article,993143fca3613014990cc6975cacdd62f4147e03,12918133247
53.19413f33fc660536d9d4f315a1283440. because: regionserver60020.cacheFlusher
2010-12-09 06:23:27,058 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs -- HDFS-200
2010-12-09 06:23:27,080 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Roll /hbase/.logs/06.hadoop.****,60020,1291864620851/10.99.97.106%3A60020.1291875047947, entries=53315,
filesize=63781925. New hlog /hbase/.logs/06.hadoop.****,60020,1291864620851/10.99.97.106%3A60020.1291875807006
2010-12-09 06:23:27,080 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: Found 1 hlogs to remove out of total 8; oldest outstanding sequenceid is 33526881 from region article
,1221b0013c61fd7446cb822f698d0bcafa487342,1291872610788.bb2aa6ae8c694a096a00e7bb91fc46c3.
2010-12-09 06:23:27,080 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: moving old hlog file /hbase/.logs/06.hadoop.****,60020,1291864620851/10.99.97.106%3A60020.1291872064424
whose highest sequenceid is 33506639 to /hbase/.oldlogs/10.99.97.106%3A60020.1291872064424
2010-12-09 06:25:39,162 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler 12 on 60020 took 119034ms appending an edit to hlog; editcount=3683
2010-12-09 06:25:39,162 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100)
at sun.nio.ch.IOUtil.write(IOUtil.java:71)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2509)
2010-12-09 06:25:39,163 WARN org.apache.hadoop.hdfs.DFSClient: DFSOutputStream ResponseProcessor exception for block blk_1141147913164105086_258331java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:180)
at java.io.DataInputStream.readLong(DataInputStream.java:399)
at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:120)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2621)
2010-12-09 06:25:39,163 INFO org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_1141147913164105086_258331 waiting for responder to exit.
2010-12-09 06:25:39,164 WARN org.apache.hadoop.hdfs.DFSClient: DFSOutputStream ResponseProcessor exception for block blk_-7076574401277381592_258324java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:180)
at java.io.DataInputStream.readLong(DataInputStream.java:399)
at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:120)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2621)
2010-12-09 06:25:39,164 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_-7076574401277381592_258324 bad datanode[0] 10.99.97.106:50010
2010-12-09 06:25:39,164 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x12ca58c08ff004b, likely server has closed socket, closing socket connection and attempting reconnect
2010-12-09 06:25:39,164 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_-7076574401277381592_258324 in pipeline 10.99.97.106:50010, 10.100.154.103:50010: bad datanode 10.99.97.106:50010
2010-12-09 06:25:39,165 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_1141147913164105086_258331 bad datanode[0] 10.99.97.106:50010
2010-12-09 06:25:39,167 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x12ca58c08ff004c, likely server has closed socket, closing socket connection and attempting reconnect
2010-12-09 06:25:39,169 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.100.154.103:50020. Already tried 0 time(s).
2010-12-09 06:25:39,169 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_1141147913164105086_258331 in pipeline 10.99.97.106:50010, 10.100.159.13:50010: bad datanode 10.99.97.106:50010
2010-12-09 06:25:39,209 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGIONSERVER_STOP
2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60020
2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 3 on 60020: exiting
2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 9 on 60020: exiting
2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 24 on 60020: exiting
2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 1 on 60020: exiting
2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 15 on 60020: exiting
2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 0 on 60020: exiting
2010-12-09 06:25:39,214 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 3 on 60020: exiting
2010-12-09 06:25:39,214 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server listener on 60020
2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 2 on 60020: exiting
2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 7 on 60020: exiting
2010-12-09 06:25:39,213 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 21 on 60020: exiting
2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 4 on 60020: exiting
2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server Responder
2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 5 on 60020: exiting
2010-12-09 06:25:39,216 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 8 on 60020: exiting
2010-12-09 06:25:39,216 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 9 on 60020: exiting
2010-12-09 06:25:39,216 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 6 on 60020: exiting
2010-12-09 06:25:39,216 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
2010-12-09 06:25:39,223 WARN org.apache.hadoop.hdfs.DFSClient: Failed recovery attempt #0 from primary datanode 10.100.159.13:50010
Re: Crashing Region Servers
Posted by Lance Riedel <la...@gmail.com>.
I've been really watching swap/GC and not seeing anything. I'll look more into it. I know this sounds out there, but I'm wondering if there is any JVM/OS issues? I'm seeing a very weird spike at the time of just one of the CPUs where it is pegged at 100%, with 75% of it all Sys. In the mean time, it looks like all IO is hung up.. nothing is happening, and load goes up to like 20.
I can't get to the machine during this time, but ganglia keeps reporting. And we are no where near memory/swap at the time (and i in fact never hit swap). Out of the 6 gigs allocated to the JVM, maybe 3-4 are being used.
During this time the rest of the cpus go idle, and the network goes to 0.. this is for the 15 minutes!
I also found this happen during a normal mapreduce job when not using hbase, just the cluster (heavily loaded mind you, but not justifying the weird behavior).
I'm using Ubuntu lucid (the AMI recommended by Cloudera) and have tried u20 and u22 for the JVMs
I'm now installing the exact same setup with CentOS with u16, to see if i see this same behavior.
We haven't gotten to the rolling of 30 days yet, will get to that partitioning when we come to it, so far we only are testing with a week's worth of data.
On Dec 21, 2010, at 11:53 AM, Stack wrote:
> On Mon, Dec 20, 2010 at 11:13 AM, Lance Riedel <la...@gmail.com> wrote:
>> Still having issues, so taking a step back here for some scale questions:
>> We are doing:
>> - inserts at the rate of up to 1k/sec with each entry just under 1k (roughly)
>> - roughly 60-100 megs/min - 5-10 gigs/day/ 100 million records per day
>> - rolling 30 days of data (so will kill off a days worth of data per day)
>
> How you do the rolling?
>
>> ~ 300gigs of raw data per month (maybe some more, but not orders more)
>>
>> What size of machines / number do you think we need?
>>
>
>> Here is what we have been using with lots of issues for just 1 weeks worth of data:
>> -4 compute.xlarge = each with 4 cpus, 16 gigs ram
>> - Each regionserver is getting 6 gigs allocated
>>
>
> Multiply all hardware requirements by N (where N is 2-4?) if running
> on EC2. It looks like you are trying to do test loading so you'll be
> best situated regards how much HW is needed.
>
>
>> Problem we are having:
>> Region server crashes
>>
>> Summary of problem:
>> We get a region server crashing after about 12 hours of runtime. The only load really is the inserts as described above. All the logs (master, region, datanode) are below during the hangup.
>>
>
> FYI, its tough reading logs pasted into email. Stick them on pastebin
> or up in an issue.
>
>
>> Where it gets weird:
>> The only way to startup the region server is to turn off concurrent sweep GC, otherwise it just dies on startup (see 2nd set of logs at bottom)
>>
>
> Looking at 2nd set of logs I see this:
>
>> 2010-12-20 17:26:37,543 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 56739ms for sessionid 0x12ce6e6154d00b3, closing socket connection and attempting reconnect
>
>
> That the zk client running in regionserver saying that it hasn't
> talked to the zk ensemble in near 60 seconds. Given that it pings
> every two or three seconds, you probably had a big GC pause in there
> -- were you swapping at the time? -- and since the zk timeout is 60
> seconds, the server lost its lease and exited.
>
> Can you figure whats going on here on the machine when it goes down?
> Its just starting up doing a few compactions. Doesn't look
> particularly unusual.
>
> For the first crash, it looks like you are using default JVM and then
> as you note, there is the 14minute outage... then it says:
>
>
>> 2010-12-20 12:12:55,175 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: Aborting region server serverName=01.hbase.xxx,60020,1292805088631, load=(requests=486, regions=317, usedHeap=2185, maxHeap=5333): Unhandled exception
>
>
> So, are your machines swapping?
>
> St.Ack
>
>
>>
>> Below are 2 sets of logs, (Dies after 12 hours, and Dies on startup) - only difference is GC params
>>
>> **********************************************************
>> DIES AFTER 12 HOURS LOGS:
>> Non Concurrent Sweep Death Logs
>> Runs for up to 12 hours, then here are the various logs when it dies
>> export HBASE_OPTS="$HBASE_OPTS -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:$HBASE_HOME/logs/gc-hbase.log"
>> **********************************************************
>>
>>
>> ///////////REGION SERVER
>> 2010-12-20 11:58:32,252 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Roll /hbase/.logs/01.hbase.xxxx,60020,1292805088631/10.101.25.203%3A6002
>> 0.1292845076146, entries=53328, filesize=63945980. New hlog /hbase/.logs/01.hbase.xxxx,60020,1292805088631/10.101.25.203%3A60020.1292846312039
>> 2010-12-20 11:58:32,252 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: Found 2 hlogs to remove out of total 34; oldest outstanding sequenceid
>> is 57439916 from region article,6addb2ace7f12175eb8ec92d16faa26f7592c64e,1292731695023.ec7bc70e76c6f4fc3b0779c412680375.
>> 2010-12-20 11:58:32,252 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: moving old hlog file /hbase/.logs/01.hbase.xxxx,60020,1292805088631/10.1
>> 01.25.203%3A60020.1292826985598 whose highest sequenceid is 57337628 to /hbase/.oldlogs/10.101.25.203%3A60020.1292826985598
>> 2010-12-20 11:58:32,253 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: moving old hlog file /hbase/.logs/01.hbase.xxxx,60020,1292805088631/10.1
>> 01.25.203%3A60020.1292827095006 whose highest sequenceid is 57390617 to /hbase/.oldlogs/10.101.25.203%3A60020.1292827095006
>> 2010-12-20 11:59:24,061 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler 3 on 60020 took 5172ms appending an edit to hlog; edi
>> tcount=29066
>> ****NOTE 14 minute gap here!!!! *********
>> 2010-12-20 12:12:55,016 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=32.86 MB, free=1.01 GB, max=1.04 GB, blocks=1836, accesses=1250468, hits=775218, hitRatio=61.99%%, evictions=0, evicted=0, evictedPerRun=NaN
>> 2010-12-20 12:12:55,016 WARN org.apache.hadoop.hdfs.DFSClient: DFSOutputStream ResponseProcessor exception for block blk_-6892804755545789592_78455java.net.SocketTimeoutException: 66000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.101.25.203:57216 remote=/10.101.25.203:50010]
>> 2010-12-20 12:12:55,017 INFO org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_-6892804755545789592_78455 waiting for responder to exit.
>> 2010-12-20 12:12:55,044 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 784310ms for sessionid 0x12ce6e6154d00a3, closing socket connection and attempting reconnect
>> 2010-12-20 12:12:55,062 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_-6892804755545789592_78455 bad datanode[0] 10.101.25.203:50010
>> 2010-12-20 12:12:55,062 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_-6892804755545789592_78455 in pipeline 10.101.25.203:50010, 10.124.189.106:50010: bad datanode 10.101.25.203:50010
>> 2010-12-20 12:12:55,089 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: HDFS pipeline error detected. Found 1 replicas but expecting 2 replicas. Requesting close of hlog.
>> 2010-12-20 12:12:55,148 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs -- HDFS-200
>> 2010-12-20 12:12:55,155 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event, state: Disconnected, type: None, path: null
>> 2010-12-20 12:12:55,155 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event, state: Disconnected, type: None, path: null
>> 2010-12-20 12:12:55,175 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: Aborting region server serverName=01.hbase.xxxx,60020,1292805088631, load=(requests=486, regions=317, usedHeap=2185, maxHeap=5333): Unhandled exception
>> 2010-12-20 12:12:55,176 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: request=0.0, regions=317, stores=318, storefiles=679, storefileIndexSize=88, memstoreSize=2096, compactionQueueSize=0, usedHeap=2192, maxHeap=5333, blockCacheSize=34460320, blockCacheFree=1084029536, blockCacheCount=1836, blockCacheHitRatio=61
>> 2010-12-20 12:12:55,176 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60020
>>
>> ///////////////////MASTER
>> 2010-12-20 12:12:55,022 DEBUG org.apache.hadoop.hbase.master.ServerManager: Server REPORT rejected; currently processing 01.hbase.xxxx,60020,1292805088631 as dead server
>> 2010-12-20 12:12:55,761 INFO org.apache.hadoop.hbase.util.FSUtils: Finished lease recover attempt for hdfs://00.hbase.xxxx:54310/hbase/.logs/01.hbase.xxxx,60020,1292805088631/10.101.25.203%3A60020.1292827812330
>> 2010-12-20 12:12:56,824 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: Pushed=52893 entries from hdfs://00.hbase.xxxx:54310/hbase/.logs/01.hbase.xxxx,60020,1292805088631/10.101.25.203%3A60020.1292827812330
>> 2010-12-20 12:12:56,825 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: Splitting hlog 3 of 33: hdfs://00.hbase.xxxx:54310/hbase/.logs/01.hbase.xxxx,60020,1292805088631/10.101.25.203%3A60020.1292828346328, length=63757023
>> 2010-12-20 12:12:56,825 INFO org.apache.hadoop.hbase.util.FSUtils: Recovering filehdfs://00.hbase.xxxx:54310/hbase/.logs/01.hbase.xxxx,60020,1292805088631/10.101.25.203%3A60020.1292828346328
>> 2010-12-20 12:12:57,279 INFO org.apache.hadoop.hbase.util.FSUtils: Finished lease recover attempt for hdfs://00.hbase.xxxx:54310/hbase/.logs/01.hbase.xxxx,60020,1292805088631/10.101.25.203%3A60020.1292828346328
>> 2010-12-20 12:12:58,735 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: Pushed=52918 entries from hdfs://00.hbase.xxxx:54310/hbase/.logs/01.hbase.xxxx,60020,1292805088631/10.101.25.203%3A60020.1292828346328
>>
>>
>> ////////////////DATANODE
>>
>> 2010-12-20 12:12:55,067 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in receiveBlock for block blk_-6892804755545789592_78455 java.io.EOFException: while trying to read 4377 bytes
>> 2010-12-20 12:12:55,068 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder blk_-6892804755545789592_78455 1 : Thread is interrupted.
>> 2010-12-20 12:12:55,068 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 1 for block blk_-6892804755545789592_78455 terminating
>> 2010-12-20 12:12:55,068 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock blk_-6892804755545789592_78455 received exception java.io.EOFException: while trying to read 4377 bytes
>> 2010-12-20 12:12:55,069 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(10.101.25.203:50010, storageID=DS-1813182679-10.101.25.203-50010-1292355638641, infoPort=50075, ipcPort=50020):DataXceiver
>> 2010-12-20 12:12:55,414 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_-702706422692129802_78497 src: /10.101.25.249:55244 dest: /10.101.25.203:50010
>> 2010-12-20 12:12:55,462 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_7987291438907198920_78499 src: /10.101.25.203:33387 dest: /10.101.25.203:50010
>> 2010-12-20 12:12:55,464 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_-2741441115648159702_78500 src: /10.101.25.203:33388 dest: /10.101.25.203:50010
>> 2010-12-20 12:12:55,486 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected: wildcard ongoingCreates exists for block blk_-702706422692129802_78496
>> 2010-12-20 12:12:55,486 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Reopen Block for append blk_-702706422692129802_78496
>> 2010-12-20 12:12:55,487 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in receiveBlock for block blk_-702706422692129802_78496 java.io.EOFException: while trying to read 65557 bytes
>> 2010-12-20 12:12:55,487 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block blk_-702706422692129802_78496 Interrupted.
>> 2010-12-20 12:12:55,487 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block blk_-702706422692129802_78496 terminating
>> 2010-12-20 12:12:55,487 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock blk_-702706422692129802_78496 received exception java.io.EOFException: while trying to read 65557 bytes
>> 2010-12-20 12:12:55,487 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(10.101.25.203:50010, storageID=DS-1813182679-10.101.25.203-50010-1292355638641, infoPort=50075, ipcPort=50020):DataXceiver
>> 2010-12-20 12:12:55,489 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: oldblock=blk_287777955370148653_76327(length=63761157), newblock=blk_287777955370148653_78501(length=63761157), datanode=10.101.25.203:50010
>> 2010-12-20 12:12:55,501 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in receiveBlock for block blk_-702706422692129802_78497 java.io.EOFException: while trying to read 65536 bytes
>> 2010-12-20 12:12:55,501 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Block blk_-702706422692129802_78497 unfinalized and removed.
>> 2010-12-20 12:12:55,501 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock blk_-702706422692129802_78497 received exception java.io.EOFException: while trying to read 65536 bytes
>> 2010-12-20 12:12:55,501 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(10.101.25.203:50010, storageID=DS-1813182679-10.101.25.203-50010-1292355638641, infoPort=50075, ipcPort=50020):DataXceiver
>> 2010-12-20 12:12:55,610 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.203:33388, dest: /10.101.25.203:50010, bytes: 5736187, op: HDFS_WRITE, cliID: DFSClient_-1650596123, offset: 0, srvID: DS-1813182679-10.101.25.203-50010-1292355638641, blockid: blk_-2741441115648159702_78500, duration: 119469848
>>
>> ////////////////////GC
>> 2010-12-20T11:48:25.248+0000: 40617.905: [GC [PSYoungGen: 312017K->31408K(354880K)] 2396181K->2115572K(2563648K), 0.0727710 secs] [Times: user=0.18 sys=0.00, real=0.07 secs]
>> 2010-12-20T11:57:47.651+0000: 41180.308: [GC [PSYoungGen: 312048K->33351K(307712K)] 2396212K->2117515K(2516480K), 0.0643350 secs] [Times: user=0.16 sys=0.01, real=0.06 secs]
>> 2010-12-20T11:57:54.525+0000: 41187.182: [GC [PSYoungGen: 307655K->37385K(341504K)] 2391819K->2121549K(2550272K), 0.0710110 secs] [Times: user=0.17 sys=0.00, real=0.07 secs]
>> 2010-12-20T11:58:00.157+0000: 41192.814: [GC [PSYoungGen: 305545K->26719K(289024K)] 2389709K->2127955K(2497792K), 0.0779900 secs] [Times: user=0.18 sys=0.01, real=0.08 secs]
>> 2010-12-20T11:58:04.530+0000: 41197.188: [GC [PSYoungGen: 288991K->11195K(339968K)] 2390227K->2127423K(2548736K), 0.0531710 secs] [Times: user=0.12 sys=0.00, real=0.06 secs]
>> 2010-12-20T11:58:15.908+0000: 41208.565: [GC [PSYoungGen: 275835K->10768K(269696K)] 2392063K->2131139K(2478464K), 0.0256330 secs] [Times: user=0.06 sys=0.00, real=0.02 secs]
>> 2010-12-20T11:58:20.325+0000: 41212.982: [GC [PSYoungGen: 269648K->14982K(311744K)] 2390019K->2142213K(2520512K), 0.0394690 secs] [Times: user=0.07 sys=0.00, real=0.04 secs]
>> 2010-12-20T11:58:29.751+0000: 41222.409: [GC [PSYoungGen: 268258K->7168K(255104K)] 2395489K->2141035K(2463872K), 0.0367260 secs] [Times: user=0.09 sys=0.00, real=0.03 secs]
>> 2010-12-20T11:58:34.476+0000: 41227.133: [GC [PSYoungGen: 255067K->7424K(298624K)] 2388934K->2148539K(2507392K), 0.0379800 secs] [Times: user=0.12 sys=0.00, real=0.03 secs]
>> 2010-12-20T11:58:38.866+0000: 41231.523: [GC [PSYoungGen: 250094K->7104K(244800K)] 2391210K->2155495K(2453568K), 0.0414990 secs] [Times: user=0.09 sys=0.00, real=0.04 secs]
>> 2010-12-20T11:58:43.223+0000: 41235.880: [GC [PSYoungGen: 244694K->7072K(301120K)] 2393086K->2162599K(2509888K), 0.0404940 secs] [Times: user=0.08 sy
>> s=0.00, real=0.04 secs]
>> 2010-12-20T11:58:47.589+0000: 41240.246: [GC [PSYoungGen: 247054K->7264K(298752K)] 2402582K->2169607K(2507520K), 0.0322390 secs] [Times: user=0.08 sy
>> s=0.00, real=0.04 secs]
>> 2010-12-20T11:58:52.521+0000: 41245.178: [GC [PSYoungGen: 247208K->7200K(326016K)] 2409552K->2176615K(2534784K), 0.0366060 secs] [Times: user=0.07 sy
>> s=0.00, real=0.03 secs]
>> 2010-12-20T11:58:57.931+0000: 41250.588: [GC [PSYoungGen: 281759K->8128K(328448K)] 2451174K->2184519K(2537216K), 0.0359190 secs] [Times: user=0.09 sy
>> s=0.00, real=0.03 secs]
>> 2010-12-20T11:59:09.681+0000: 41262.338: [GC [PSYoungGen: 282649K->8000K(276480K)] 2459041K->2192471K(2485248K), 0.0432830 secs] [Times: user=0.11 sys=0.01, real=0.05 secs]
>> 2010-12-20T11:59:14.136+0000: 41266.793: [GC [PSYoungGen: 276405K->7872K(324416K)] 2460876K->2200295K(2533184K), 0.0565900 secs] [Times: user=0.11 sys=0.01, real=0.06 secs]
>> 2010-12-20T11:59:18.889+0000: 41271.546: [GC [PSYoungGen: 283525K->8288K(356544K)] 2475949K->2208559K(2565312K), 0.0390340 secs] [Times: user=0.10 sys=0.01, real=0.04 secs]
>> 2010-12-20T11:59:18.928+0000: 41271.585: [Full GC [PSYoungGen: 8288K->0K(356544K)] [PSOldGen: 2200271K->1991701K(2358656K)] 2208559K->1991701K(2715200K) [PSPermGen: 18540K->18540K(21248K)], 5.1322070 secs] [Times: user=5.07 sys=0.00, real=5.14 secs]
>> 2010-12-20T11:59:29.253+0000: 41281.910: [GC [PSYoungGen: 310221K->9152K(312256K)] 2301922K->2000853K(2670912K), 0.0268860 secs] [Times: user=0.06 sys=0.00, real=0.04 secs]
>> 2010-12-20T11:59:34.610+0000: 41287.267: [GC [PSYoungGen: 312172K->8928K(325888K)] 2303873K->2009821K(2684544K), 0.0403310 secs] [Times: user=0.09 sys=0.00, real=0.05 secs]
>> 2010-12-20T11:59:41.074+0000: 41293.731: [GC [PSYoungGen: 304982K->8544K(297920K)] 2305875K->2018269K(2656576K), 0.0468460 secs] [Times: user=0.07 sys=0.00, real=0.05 secs]
>> 2010-12-20T11:59:47.708+0000: 41300.365: [GC [PSYoungGen: 297888K->8448K(309504K)] 2307613K->2026869K(2668160K), 0.0409660 secs] [Times: user=0.09 sys=0.00, real=0.04 secs]
>> Heap
>> PSYoungGen total 309504K, used 243942K [0x00007f5556d50000, 0x00007f556c610000, 0x00007f55d3d50000)
>> eden space 282880K, 83% used [0x00007f5556d50000,0x00007f55653498f0,0x00007f5568190000)
>> from space 26624K, 31% used [0x00007f556ac10000,0x00007f556b4500c0,0x00007f556c610000)
>> to space 35072K, 0% used [0x00007f5568190000,0x00007f5568190000,0x00007f556a3d0000)
>> PSOldGen total 2358656K, used 2018421K [0x00007f545cd50000, 0x00007f54eccb0000, 0x00007f5556d50000)
>> object space 2358656K, 85% used [0x00007f545cd50000,0x00007f54d806d480,0x00007f54eccb0000)
>> PSPermGen total 21248K, used 18595K [0x00007f5457950000, 0x00007f5458e10000, 0x00007f545cd50000)
>> object space 21248K, 87% used [0x00007f5457950000,0x00007f5458b78c50,0x00007f5458e10000)
>>
>>
>> **********************************************************
>> DIES ON STARTUP LOGS:
>> Non Concurrent Sweep
>> export HBASE_OPTS="$HBASE_OPTS -verbose:gc -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:$HBASE_HOME/logs/gc-hbase.log"
>> **********************************************************
>>
>> ///////////REGION SERVER
>>
>> 2010-12-20 17:25:49,020 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region article,26b5ee07c80cf31d88719b1a510726591f2b01da,12927082
>> 91003.aae953527ae2ce697fda39699e29f70e. in 0sec
>> 2010-12-20 17:25:49,020 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on region article,c050050bf22e8f276a8a6bd3c900b34d6600d6d0,129273169
>> 0903.0460d68f5f4fd423750dd6bf67937b7e.2010-12-20 17:25:49,022 DEBUG org.apache.hadoop.hbase.regionserver.Store: Compaction size of data: 186.0m; Skipped 2 file(s), size: 1894568042010-12-20 17:25:49,023 INFO org.apache.hadoop.hbase.regionserver.Store: Started compaction of 2 file(s) in data of article,c050050bf22e8f276a8a6bd3c900b34d6600d6d
>> 0,1292731690903.0460d68f5f4fd423750dd6bf67937b7e. into hdfs://00.hbase.xxxx:54310/hbase/article/0460d68f5f4fd423750dd6bf67937b7e/.tmp, sequenceid=63434988
>> 2010-12-20 17:25:49,324 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Applied 0, skipped 10953, firstSequenceidInLog=52141208, maxSequenceidInLog=63429846
>> 2010-12-20 17:25:49,327 INFO org.apache.hadoop.hbase.regionserver.HRegion: Onlined article,150bfc2bd7a0effc9f3dfa22312878bf916d68df,1292597687450.f213d3923dca3a7e3
>> d020cf7dbf51918.; next sequenceid=63429847
>> 2010-12-20 17:25:49,328 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 2 file(s) in data of article,c050050bf22e8f276a8a6bd3c900b34d6600d
>> 6d0,1292731690903.0460d68f5f4fd423750dd6bf67937b7e.; new storefile is hdfs://00.hbase.xxxx:54310/hbase/article/0460d68f5f4fd423750dd6bf67937b7e/data/88724363503101
>> 19241; store size is 186.0m
>> 2010-12-20 17:25:49,329 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region article,c050050bf22e8f276a8a6bd3c900b34d6600d6d0,12927316
>> 90903.0460d68f5f4fd423750dd6bf67937b7e. in 0sec
>> 2010-12-20 17:25:49,329 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on region article,933fc87674f7132b5d91ce6e9c60bc68981a56a7,129270464
>> 1421.2551c0c46387d1479d25fb359ef4c5c9.
>> 2010-12-20 17:25:49,336 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: article,0643e00cb1b172034b2627567d00d26913a25f97,12927089
>> 81266.5e0ac5bd27f34da31ded307e2ea932cf.2010-12-20 17:25:49,337 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Creating region article,0643e00cb1b172034b2627567d00d26913a25f97,1292708981266.5e0ac5bd
>> 27f34da31ded307e2ea932cf.
>> 2010-12-20 17:25:49,338 DEBUG org.apache.hadoop.hbase.regionserver.Store: Compaction size of data: 186.5m; Skipped 2 file(s), size: 190928927
>> 2010-12-20 17:25:49,338 INFO org.apache.hadoop.hbase.regionserver.Store: Started compaction of 2 file(s) in data of article,933fc87674f7132b5d91ce6e9c60bc68981a56a
>> 7,1292704641421.2551c0c46387d1479d25fb359ef4c5c9. into hdfs://00.hbase.xxxx:54310/hbase/article/2551c0c46387d1479d25fb359ef4c5c9/.tmp, sequenceid=63434974
>> 2010-12-20 17:25:49,362 DEBUG org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://00.hbase.xxxx:54310/hbase/article/5e0ac5bd27f34da31ded307e2ea932cf/data/421
>> 7589706037747905, isReference=false, isBulkLoadResult=false, seqid=53910828, majorCompaction=true
>> 2010-12-20 17:25:49,372 DEBUG org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://00.hbase.xxxx:54310/hbase/article/5e0ac5bd27f34da31ded307e2ea932cf/data/540
>> 8931691681201713, isReference=false, isBulkLoadResult=false, seqid=63429839, majorCompaction=false
>> 2010-12-20 17:25:49,379 DEBUG org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://00.hbase.xxxx:54310/hbase/article/5e0ac5bd27f34da31ded307e2ea932cf/data/779
>> 2081177431743589, isReference=false, isBulkLoadResult=false, seqid=62768007, majorCompaction=false
>> 2010-12-20 17:25:49,402 DEBUG org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://00.hbase.xxxx:54310/hbase/article/5e0ac5bd27f34da31ded307e2ea932cf/data/793
>> 5329939530860970, isReference=false, isBulkLoadResult=false, seqid=62300392, majorCompaction=false
>> 2010-12-20 17:25:49,427 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://00.hbase.xxxx:54310/hbase/article/5e0ac5bd27f34da31ded307e2e
>> a932cf/recovered.edits/0000000000052141022; minSequenceid=63429839
>> 2010-12-20 17:25:49,609 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region article,933fc87674f7132b5d91ce6e9c60bc68981a56a7,1292704641421.2551c0c46387d1479d25fb359ef4c5c9. in 0sec
>> 2010-12-20 17:25:49,609 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on region article,fc36770defdbeba6e2841181a7da6f7bbae7f64f,1292701971082.399bf454e0c2a73adc038810e8fe4017.
>> 2010-12-20 17:25:49,610 DEBUG org.apache.hadoop.hbase.regionserver.Store: Compaction size of data: 186.6m; Skipped 2 file(s), size: 192468499
>> 2010-12-20 17:25:49,610 INFO org.apache.hadoop.hbase.regionserver.Store: Started compaction of 2 file(s) in data of article,fc36770defdbeba6e2841181a7da6f7bbae7f64f,1292701971082.399bf454e0c2a73adc038810e8fe4017. into hdfs://00.hbase.xxxx:54310/hbase/article/399bf454e0c2a73adc038810e8fe4017/.tmp, sequenceid=63427381
>> 2010-12-20 17:26:37,543 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 56739ms for sessionid 0x12ce6e6154d00b3, closing socket connection and attempting reconnect
>> 2010-12-20 17:26:37,632 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Applied 0, skipped 5388, firstSequenceidInLog=52141022, maxSequenceidInLog=63429839
>> 2010-12-20 17:26:37,644 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: Aborting region server serverName=02.hbase.xxxx,60020,1292865913422, load=(requests=44, regions=142, usedHeap=97, maxHeap=5991): Unhandled exception
>> org.apache.hadoop.hbase.YouAreDeadException: org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing 02.hbase.xxxx,60020,1292865913422 as dead server
>> at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:217)
>> at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:271)
>> at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:744)
>> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:561)
>> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>>
>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>> at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
>> at org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
>> at org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
>> at org.apache.hadoop.hbase.regionserver.HRegionServer.doRun(HRegionServer.java:635)
>> at org.apache.hadoop.hbase.regionserver.HRegionServer.access$000(HRegionServer.java:126)
>> at org.apache.hadoop.hbase.regionserver.HRegionServer$1.run(HRegionServer.java:518)
>> at org.apache.hadoop.hbase.regionserver.HRegionServer$1.run(HRegionServer.java:516)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:337)
>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1046)
>> at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:516)
>> at java.lang.Thread.run(Thread.java:662)
>> 2010-12-20 17:26:37,645 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: request=0.0, regions=142, stores=142, storefiles=462, storefileIndexSize=43, memstoreSize=4, compactionQueueSize=36, usedHeap=105, maxHeap=5991, blockCacheSize=10308888, blockCacheFree=1246239080, blockCacheCount=0, blockCacheHitRatio=0
>> 2010-12-20 17:26:37,645 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60020
>> 2010-12-20 17:26:37,645 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 0 on 60020: exiting
>> 2010-12-20 17:26:37,645 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 1 on 60020: exiting
>> 2010-12-20 17:26:37,645 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 2 on 60020: exiting
>>
>> ////////////////DATANODE
>>
>> 2010-12-20 17:25:48,970 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 1 for block blk_7313855026629458364_85588 terminating
>> 2010-12-20 17:25:49,012 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:50010, dest: /10.101.25.249:53108, bytes: 232, op: HDFS_READ, cliID: DFSClient_hb_rs_02.hbase.xxxx,60020,1292865913422_1292865913443, offset: 5637120, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_7313855026629458364_85588, duration: 168436
>> 2010-12-20 17:25:49,013 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:50010, dest: /10.101.25.249:53109, bytes: 7456, op: HDFS_READ, cliID: DFSClient_hb_rs_02.hbase.xxxx,60020,1292865913422_1292865913443, offset: 5629952, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_7313855026629458364_85588, duration: 178950
>> 2010-12-20 17:25:49,061 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_5943185653696604572_85589 src: /10.101.25.249:53112 dest: /10.101.25.249:50010
>> 2010-12-20 17:25:49,103 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:50010, dest: /10.101.25.249:53111, bytes: 349508, op: HDFS_READ, cliID: DFSClient_hb_rs_02.hbase.xxxx,60020,1292865913422_1292865913443, offset: 0, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_7780046853952422303_85486, duration: 73022705
>> 2010-12-20 17:25:49,307 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:53112, dest: /10.101.25.249:50010, bytes: 5564634, op: HDFS_WRITE, cliID: DFSClient_hb_rs_02.hbase.xxxx,60020,1292865913422_1292865913443, offset: 0, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_5943185653696604572_85589, duration: 242139544
>> 2010-12-20 17:25:49,308 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 1 for block blk_5943185653696604572_85589 terminating
>> 2010-12-20 17:25:49,324 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:50010, dest: /10.101.25.249:53114, bytes: 222, op: HDFS_READ, cliID: DFSClient_hb_rs_02.hbase.xxxx,60020,1292865913422_1292865913443, offset: 5564416, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_5943185653696604572_85589, duration: 144173
>> 2010-12-20 17:25:49,325 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:50010, dest: /10.101.25.249:53115, bytes: 7446, op: HDFS_READ, cliID: DFSClient_hb_rs_02.hbase.xxxx,60020,1292865913422_1292865913443, offset: 5557248, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_5943185653696604572_85589, duration: 157056
>> 2010-12-20 17:25:49,355 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_-5312992283942058338_85590 src: /10.101.25.249:53118 dest: /10.101.25.249:50010
>> 2010-12-20 17:25:49,358 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:50010, dest: /10.101.25.249:53120, bytes: 199, op: HDFS_READ, cliID: DFSClient_hb_rs_02.hbase.xxxx,60020,1292865913422_1292865913443, offset: 25063424, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_-2003525593082351171_69708, duration: 138638
>> 2010-12-20 17:25:49,359 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:50010, dest: /10.101.25.249:53121, bytes: 190603, op: HDFS_READ, cliID: DFSClient_hb_rs_02.hbase.xxxx,60020,1292865913422_1292865913443, offset: 24874496, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_-2003525593082351171_69708, duration: 735353
>> 2010-12-20 17:25:49,411 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:50010, dest: /10.101.25.249:53117, bytes: 3070642, op: HDFS_READ, cliID: DFSClient_hb_rs_02.hbase.xxxx,60020,1292865913422_1292865913443, offset: 0, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_8585904126405165382_85484, duration: 68063489
>> 2010-12-20 17:25:49,597 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:53118, dest: /10.101.25.249:50010, bytes: 4631054, op: HDFS_WRITE, cliID: DFSClient_hb_rs_02.hbase.xxxx,60020,1292865913422_1292865913443, offset: 0, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_-5312992283942058338_85590, duration: 234095142
>> 2010-12-20 17:25:49,597 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 1 for block blk_-5312992283942058338_85590 terminating
>> 2010-12-20 17:25:49,604 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:50010, dest: /10.101.25.249:53129, bytes: 534, op: HDFS_READ, cliID: DFSClient_hb_rs_02.hbase.xxxx,60020,1292865913422_1292865913443, offset: 4630528, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_-5312992283942058338_85590, duration: 186485
>> 2010-12-20 17:25:49,605 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:50010, dest: /10.101.25.249:53130, bytes: 6210, op: HDFS_READ, cliID: DFSClient_hb_rs_02.hbase.xxxx,60020,1292865913422_1292865913443, offset: 4624896, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_-5312992283942058338_85590, duration: 155935
>> 2010-12-20 17:25:49,672 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_-7865815939426325649_85591 src: /10.101.25.249:53133 dest: /10.101.25.249:50010
>> 2010-12-20 17:26:22,770 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:50010, dest: /10.112.79.14:41575, bytes: 166, op: HDFS_READ, cliID: DFSClient_-1843172643, offset: 39720448, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_-2090804821592279293_75264, duration: 12128121
>> 2010-12-20 17:26:22,810 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:50010, dest: /10.112.79.14:41576, bytes: 208114, op: HDFS_READ, cliID: DFSClient_-1843172643, offset: 39514112, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_-2090804821592279293_75264, duration: 38720643
>> 2010-12-20 17:26:23,107 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:50010, dest: /10.112.79.14:41581, bytes: 312, op: HDFS_READ, cliID: DFSClient_-1843172643, offset: 16006656, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_4633255788259970659_77483, duration: 250202855
>> 2010-12-20 17:26:37,538 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-8913559857952133277_85447 file /dist2/data/hadoop-data/current/subdir34/blk_-8913559857952133277 for deletion
>> 2010-12-20 17:26:37,547 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:50010, dest: /10.112.79.14:41582, bytes: 19920, op: HDFS_READ, cliID: DFSClient_-1843172643, offset: 15987200, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_4633255788259970659_77483, duration: 4042558
>> 2010-12-20 17:26:37,547 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-8511507105977516601_84587 file /dist3/data/hadoop-data/current/subdir31/blk_-8511507105977516601 for deletion
>> 2010-12-20 17:26:37,549 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block blk_-8913559857952133277_85447 at file /dist2/data/hadoop-data/current/subdir34/blk_-8913559857952133277
>> 2010-12-20 17:26:37,555 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-8508054838724986047_84580 file /dist2/data/hadoop-data/current/subdir41/blk_-8508054838724986047 for deletion
>> 2010-12-20 17:26:37,556 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-7845924152894793652_85398 file /dist3/data/hadoop-data/current/subdir43/blk_-7845924152894793652 for deletion
>> 2010-12-20 17:26:37,556 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-7088372163619556394_84570 file /dist2/data/hadoop-data/current/subdir19/blk_-7088372163619556394 for deletion
>> 2010-12-20 17:26:37,556 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-6970445900444688558_85387 file /dist1/data/hadoop-data/current/subdir57/blk_-6970445900444688558 for deletion
>> 2010-12-20 17:26:37,557 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-6453226657933735816_84505 file /dist1/data/hadoop-data/current/subdir43/blk_-6453226657933735816 for deletion
>> 2010-12-20 17:26:37,557 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-6417800567255755267_84658 file /dist2/data/hadoop-data/current/subdir41/blk_-6417800567255755267 for deletion
>> 2010-12-20 17:26:37,557 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-6349908890584705232_84612 file /dist1/data/hadoop-data/current/subdir57/blk_-6349908890584705232 for deletion
>> 2010-12-20 17:26:37,557 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-6260136960291589474_84706 file /dist2/data/hadoop-data/current/subdir41/blk_-6260136960291589474 for deletion
>> 2010-12-20 17:26:37,557 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-6236925753292021036_84691 file /dist2/data/hadoop-data/current/subdir41/blk_-6236925753292021036 for deletion
>> 2010-12-20 17:26:37,557 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-6108807505647576258_84708 file /dist3/data/hadoop-data/current/subdir59/blk_-6108807505647576258 for deletion
>> 2010-12-20 17:26:37,557 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-5322405269255073859_84530 file /dist2/data/hadoop-data/current/subdir19/blk_-5322405269255073859 for deletion
>> 2010-12-20 17:26:37,558 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-4932349484140214063_84528 file /dist3/data/hadoop-data/current/subdir31/blk_-4932349484140214063 for deletion
>> 2010-12-20 17:26:37,558 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-4903335785153601370_84648 file /dist3/data/hadoop-data/current/subdir59/blk_-4903335785153601370 for deletion
>> 2010-12-20 17:26:37,558 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-4036825098230519223_84531 file /dist3/data/hadoop-data/current/subdir31/blk_-4036825098230519223 for deletion
>> 2010-12-20 17:26:37,558 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-3437992750138243050_85434 file /dist2/data/hadoop-data/current/subdir34/blk_-3437992750138243050 for deletion
>> 2010-12-20 17:26:37,558 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block blk_-6970445900444688558_85387 at file /dist1/data/hadoop-data/current/subdir57/blk_-6970445900444688558
>> 2010-12-20 17:26:37,558 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-3364192718686291807_85433 file /dist1/data/hadoop-data/current/subdir48/blk_-3364192718686291807 for deletion
>> 2010-12-20 17:26:37,558 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-3357432203841337074_84713 file /dist1/data/hadoop-data/current/subdir57/blk_-3357432203841337074 for deletion
>> 2010-12-20 17:26:37,558 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-3238846993152973042_84497 file /dist1/data/hadoop-data/current/subdir43/blk_-3238846993152973042 for deletion
>> 2010-12-20 17:26:37,558 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-2996249640811055720_83272 file /dist1/data/hadoop-data/current/subdir31/blk_-2996249640811055720 for deletion
>> 2010-12-20 17:26:37,558 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-2790489595075270318_85390 file /dist3/data/hadoop-data/current/subdir43/blk_-2790489595075270318 for deletion
>> 2010-12-20 17:26:37,558 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-2781143678253104315_84697 file /dist3/data/hadoop-data/current/subdir59/blk_-2781143678253104315 for deletion
>> 2010-12-20 17:26:37,558 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-2640743692778440261_84536 file /dist2/data/hadoop-data/current/subdir19/blk_-2640743692778440261 for deletion
>> 2010-12-20 17:26:37,558 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-2220284194502441495_85381 file /dist3/data/hadoop-data/current/subdir59/blk_-2220284194502441495 for deletion
>> 2010-12-20 17:26:37,559 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-2126728115231716918_84647 file /dist2/data/hadoop-data/current/subdir41/blk_-2126728115231716918 for deletion
>> 2010-12-20 17:26:37,559 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-2101072572804865450_84585 file /dist1/data/hadoop-data/current/subdir43/blk_-2101072572804865450 for deletion
>> 2010-12-20 17:26:37,559 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-1725324590496466521_84695 file /dist1/data/hadoop-data/current/subdir57/blk_-1725324590496466521 for deletion
>> 2010-12-20 17:26:37,559 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-1569098086020238107_83275 file /dist2/data/hadoop-data/current/subdir57/blk_-1569098086020238107 for deletion
>> 2010-12-20 17:26:37,559 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-638559696544756508_85446 file /dist1/data/hadoop-data/current/subdir48/blk_-638559696544756508 for deletion
>> 2010-12-20 17:26:37,559 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-306662003767169937_84555 file /dist1/data/hadoop-data/current/subdir43/blk_-306662003767169937 for deletion
>> 2010-12-20 17:26:37,559 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-278326848888782043_84594 file /dist1/data/hadoop-data/current/subdir57/blk_-278326848888782043 for deletion
>> 2010-12-20 17:26:37,559 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block blk_-6453226657933735816_84505 at file /dist1/data/hadoop-data/current/subdir43/blk_-6453226657933735816
>> 2010-12-20 17:26:37,559 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-91580339643929179_85459 file /dist3/data/hadoop-data/current/subdir43/blk_-91580339643929179 for deletion
>> 2010-12-20 17:26:37,559 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_9419128090539524_84662 file /dist1/data/hadoop-data/current/subdir57/blk_9419128090539524 for deletion
>> 2010-12-20 17:26:37,559 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_679544100533753423_84719 file /dist1/data/hadoop-data/current/subdir57/blk_679544100533753423 for deletion
>> 2010-12-20 17:26:37,559 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_730403277102726249_84590 file /dist3/data/hadoop-data/current/subdir31/blk_730403277102726249 for deletion
>> 2010-12-20 17:26:37,572 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_797603508126576279_84551 file /dist3/data/hadoop-data/current/subdir31/blk_797603508126576279 for deletion
>> 2010-12-20 17:26:37,572 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_867345031868208479_85424 file /dist3/data/hadoop-data/current/subdir43/blk_867345031868208479 for deletion
>> 2010-12-20 17:26:37,572 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_1054277086916165125_84630 file /dist2/data/hadoop-data/current/subdir41/blk_1054277086916165125 for deletion
>> 2010-12-20 17:26:37,572 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_1276716639229193075_85420 file /dist1/data/hadoop-data/current/subdir57/blk_1276716639229193075 for deletion
>> 2010-12-20 17:26:37,572 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_1747684879678419666_84627 file /dist1/data/hadoop-data/current/subdir57/blk_1747684879678419666 for deletion
>> 2010-12-20 17:26:37,572 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_1962685174146987659_84561 file /dist3/data/hadoop-data/current/subdir31/blk_1962685174146987659 for deletion
>> 2010-12-20 17:26:37,572 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_2025768890877930088_84552 file /dist1/data/hadoop-data/current/subdir43/blk_2025768890877930088 for deletion
>> 2010-12-20 17:26:37,572 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_2156546746312159503_84617 file /dist3/data/hadoop-data/current/subdir59/blk_2156546746312159503 for deletion
>> 2010-12-20 17:26:37,572 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_2580899260621247350_85413 file /dist2/data/hadoop-data/current/subdir34/blk_2580899260621247350 for deletion
>> 2010-12-20 17:26:37,572 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_2600394775430572452_85458 file /dist2/data/hadoop-data/current/subdir34/blk_2600394775430572452 for deletion
>> 2010-12-20 17:26:37,572 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_2906704921361114164_84593 file /dist3/data/hadoop-data/current/subdir31/blk_2906704921361114164 for deletion
>> 2010-12-20 17:26:37,572 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_3088125590653329452_85466 file /dist1/data/hadoop-data/current/subdir48/blk_3088125590653329452 for deletion
>> 2010-12-20 17:26:37,572 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_3263574674943605427_84527 file /dist2/data/hadoop-data/current/subdir19/blk_3263574674943605427 for deletion
>> 2010-12-20 17:26:37,573 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_3271569822852131416_85399 file /dist1/data/hadoop-data/current/subdir57/blk_3271569822852131416 for deletion
>> 2010-12-20 17:26:37,573 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_3629755351194789263_84618 file /dist1/data/hadoop-data/current/subdir57/blk_3629755351194789263 for deletion
>> 2010-12-20 17:26:37,573 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_3872473819820471284_84646 file /dist1/data/hadoop-data/current/subdir57/blk_3872473819820471284 for deletion
>> 2010-12-20 17:26:37,573 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_3967429886843094718_84586 file /dist2/data/hadoop-data/current/subdir41/blk_3967429886843094718 for deletion
>> 2010-12-20 17:26:37,573 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_4177484748478595642_84600 file /dist1/data/hadoop-data/current/subdir57/blk_4177484748478595642 for deletion
>> 2010-12-20 17:26:37,573 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_4741750624588255690_85455 file /dist3/data/hadoop-data/current/subdir43/blk_4741750624588255690 for deletion
>> 2010-12-20 17:26:37,573 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_5189334311200664471_84687 file /dist3/data/hadoop-data/current/subdir59/blk_5189334311200664471 for deletion
>> 2010-12-20 17:26:37,573 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_5554094926453315166_84616 file /dist2/data/hadoop-data/current/subdir41/blk_5554094926453315166 for deletion
>> 2010-12-20 17:26:37,573 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_5630019573499502243_85460 file /dist1/data/hadoop-data/current/subdir48/blk_5630019573499502243 for deletion
>> 2010-12-20 17:26:37,573 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_6462688363000457793_84620 file /dist3/data/hadoop-data/current/subdir59/blk_6462688363000457793 for deletion
>> 2010-12-20 17:26:37,573 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_6470741303722852096_84592 file /dist2/data/hadoop-data/current/subdir41/blk_6470741303722852096 for deletion
>> 2010-12-20 17:26:37,573 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_6605638457633532879_84632 file /dist3/data/hadoop-data/current/subdir59/blk_6605638457633532879 for deletion
>> 2010-12-20 17:26:37,573 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_6794523298330118718_77314 file /dist3/data/hadoop-data/current/subdir11/blk_6794523298330118718 for deletion
>> 2010-12-20 17:26:37,574 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_6926053514624905958_84695 file /dist3/data/hadoop-data/current/subdir59/blk_6926053514624905958 for deletion
>> 2010-12-20 17:26:37,574 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_7116690323560112908_85418 file /dist2/data/hadoop-data/current/subdir34/blk_7116690323560112908 for deletion
>> 2010-12-20 17:26:37,574 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_7202997804403926942_84705 file /dist1/data/hadoop-data/current/subdir57/blk_7202997804403926942 for deletion
>> 2010-12-20 17:26:37,574 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_7434927287494785191_85394 file /dist2/data/hadoop-data/current/subdir34/blk_7434927287494785191 for deletion
>> 2010-12-20 17:26:37,574 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_7594986426526893113_84520 file /dist3/data/hadoop-data/current/subdir31/blk_7594986426526893113 for deletion
>> 2010-12-20 17:26:37,574 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_7661061963483856377_85463 file /dist3/data/hadoop-data/current/subdir43/blk_7661061963483856377 for deletion
>> 2010-12-20 17:26:37,574 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_7780046853952422303_85486 file /dist3/data/hadoop-data/current/subdir43/blk_7780046853952422303 for deletion
>> 2010-12-20 17:26:37,574 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_7789033250788024385_83279 file /dist1/data/hadoop-data/current/subdir31/blk_7789033250788024385 for deletion
>> 2010-12-20 17:26:37,574 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_7805477460105158814_84523 file /dist1/data/hadoop-data/current/subdir43/blk_7805477460105158814 for deletion
>> 2010-12-20 17:26:37,574 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_7852099683471709790_84591 file /dist1/data/hadoop-data/current/subdir57/blk_7852099683471709790 for deletion
>> 2010-12-20 17:26:37,574 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_8229638397827768666_85415 file /dist3/data/hadoop-data/current/subdir43/blk_8229638397827768666 for deletion
>> 2010-12-20 17:26:37,574 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_8541283056878555347_85477 file /dist1/data/hadoop-data/current/subdir48/blk_8541283056878555347 for deletion
>> 2010-12-20 17:26:37,574 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_8585904126405165382_85484 file /dist1/data/hadoop-data/current/subdir48/blk_8585904126405165382 for deletion
>> 2010-12-20 17:26:37,575 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_8606266970652887102_84483 file /dist1/data/hadoop-data/current/subdir43/blk_8606266970652887102 for deletion
>> 2010-12-20 17:26:37,575 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_9082358445568727636_85437 file /dist2/data/hadoop-data/current/subdir34/blk_9082358445568727636 for deletion
>> 2010-12-20 17:26:37,575 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block blk_-6349908890584705232_84612 at file /dist1/data/hadoop-data/current/subdir57/blk_-6349908890584705232
>>
>> //////////////////////GC
>>
>> 2010-12-20T17:25:33.242+0000: 20.929: [CMS-concurrent-mark: 0.079/0.380 secs] [Times: user=0.36 sys=0.04, real=0.38 secs]
>> 2010-12-20T17:25:33.242+0000: 20.929: [CMS-concurrent-preclean-start]
>> 2010-12-20T17:25:33.245+0000: 20.932: [CMS-concurrent-preclean: 0.002/0.003 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:33.245+0000: 20.932: [CMS-concurrent-abortable-preclean-start]
>> 2010-12-20T17:25:33.302+0000: 20.989: [GC 20.989: [ParNew: 17796K->1070K(19136K), 0.0025870 secs] 68068K->51343K(101612K) icms_dc=9 , 0.0027060 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:33.441+0000: 21.127: [GC 21.127: [ParNew: 18034K->2086K(19136K), 0.0042130 secs] 68307K->53943K(101612K) icms_dc=9 , 0.0043330 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
>> 2010-12-20T17:25:33.514+0000: 21.201: [CMS-concurrent-abortable-preclean: 0.007/0.269 secs] [Times: user=0.19 sys=0.05, real=0.27 secs]
>> 2010-12-20T17:25:33.515+0000: 21.202: [GC[YG occupancy: 11042 K (19136 K)]21.202: [Rescan (parallel) , 0.0158670 secs]21.218: [weak refs processing, 0.0000950 secs] [1 CMS-remark: 51857K(82476K)] 62900K(101612K), 0.0160770 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
>> 2010-12-20T17:25:33.531+0000: 21.218: [CMS-concurrent-sweep-start]
>> 2010-12-20T17:25:33.566+0000: 21.253: [GC 21.253: [ParNew: 19070K->1999K(19136K), 0.0027670 secs] 70927K->54085K(101612K) icms_dc=9 , 0.0028940 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:33.648+0000: 21.335: [GC 21.335: [ParNew: 19023K->2112K(19136K), 0.0046590 secs] 70961K->55368K(101612K) icms_dc=9 , 0.0047840 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
>> 2010-12-20T17:25:33.713+0000: 21.400: [CMS-concurrent-sweep: 0.013/0.182 secs] [Times: user=0.14 sys=0.03, real=0.18 secs]
>> 2010-12-20T17:25:33.714+0000: 21.401: [CMS-concurrent-reset-start]
>> 2010-12-20T17:25:33.893+0000: 21.580: [GC 21.580: [ParNew: 19123K->598K(19136K), 0.0015670 secs] 72284K->53771K(107740K) icms_dc=9 , 0.0016740 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:33.945+0000: 21.632: [CMS-concurrent-reset: 0.020/0.231 secs] [Times: user=0.19 sys=0.07, real=0.23 secs]
>> 2010-12-20T17:25:34.078+0000: 21.764: [GC 21.765: [ParNew: 17622K->1665K(19136K), 0.0021650 secs] 70795K->54839K(107740K) icms_dc=4 , 0.0022950 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:34.127+0000: 21.814: [GC [1 CMS-initial-mark: 53173K(88604K)] 63030K(107740K), 0.0031010 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:34.130+0000: 21.817: [CMS-concurrent-mark-start]
>> 2010-12-20T17:25:34.181+0000: 21.867: [GC 21.867: [ParNew: 18633K->1360K(19136K), 0.0044440 secs] 71806K->55614K(107740K) icms_dc=5 , 0.0045740 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:34.387+0000: 22.074: [GC 22.074: [ParNew: 18370K->1923K(19136K), 0.0020250 secs] 72624K->56177K(107740K) icms_dc=5 , 0.0021580 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:34.557+0000: 22.244: [GC 22.244: [ParNew: 18947K->2112K(19136K), 0.0225430 secs] 73201K->58091K(107740K) icms_dc=6 , 0.0226640 secs] [Times: user=0.02 sys=0.00, real=0.03 secs]
>> 2010-12-20T17:25:34.681+0000: 22.367: [GC 22.367: [ParNew: 19098K->811K(19136K), 0.0031310 secs] 75077K->57280K(107740K) icms_dc=6 , 0.0032510 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:34.751+0000: 22.437: [CMS-concurrent-mark: 0.087/0.620 secs] [Times: user=0.42 sys=0.04, real=0.63 secs]
>> 2010-12-20T17:25:34.751+0000: 22.438: [CMS-concurrent-preclean-start]
>> 2010-12-20T17:25:34.757+0000: 22.444: [CMS-concurrent-preclean: 0.005/0.006 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:34.757+0000: 22.444: [CMS-concurrent-abortable-preclean-start]
>> 2010-12-20T17:25:34.865+0000: 22.552: [GC 22.552: [ParNew: 17835K->1114K(19136K), 0.0016200 secs] 74304K->57582K(107740K) icms_dc=6 , 0.0017310 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:34.966+0000: 22.652: [CMS-concurrent-abortable-preclean: 0.003/0.209 secs] [Times: user=0.10 sys=0.02, real=0.21 secs]
>> 2010-12-20T17:25:34.966+0000: 22.653: [GC[YG occupancy: 9657 K (19136 K)]22.653: [Rescan (parallel) , 0.0020570 secs]22.655: [weak refs processing, 0.0000610 secs] [1 CMS-remark: 56468K(88604K)] 66126K(107740K), 0.0022320 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
>> 2010-12-20T17:25:34.969+0000: 22.656: [CMS-concurrent-sweep-start]
>> 2010-12-20T17:25:35.101+0000: 22.787: [GC 22.787: [ParNew: 18138K->2112K(19136K), 0.0038570 secs] 74584K->60794K(107740K) icms_dc=6 , 0.0039870 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:35.214+0000: 22.900: [CMS-concurrent-sweep: 0.013/0.245 secs] [Times: user=0.07 sys=0.01, real=0.24 secs]
>> 2010-12-20T17:25:35.214+0000: 22.900: [CMS-concurrent-reset-start]
>> 2010-12-20T17:25:35.354+0000: 23.041: [GC 23.041: [ParNew: 19136K->1909K(19136K), 0.0035100 secs] 75099K->58683K(112412K) icms_dc=6 , 0.0036250 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
>> 2010-12-20T17:25:35.449+0000: 23.136: [GC 23.136: [ParNew: 18931K->1384K(19136K), 0.0027830 secs] 75706K->59022K(112412K) icms_dc=6 , 0.0029070 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:35.499+0000: 23.186: [CMS-concurrent-reset: 0.018/0.285 secs] [Times: user=18446743517.92 sys=0.04, real=0.29 secs]
>> 2010-12-20T17:25:35.606+0000: 23.292: [GC 23.292: [ParNew: 18352K->1675K(19136K), 0.0019650 secs] 75990K->59313K(112412K) icms_dc=1 , 0.0020900 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
>> 2010-12-20T17:25:35.661+0000: 23.348: [GC [1 CMS-initial-mark: 57637K(93276K)] 67851K(112412K), 0.0032750 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:35.664+0000: 23.351: [CMS-concurrent-mark-start]
>> 2010-12-20T17:25:35.779+0000: 23.466: [GC 23.466: [ParNew: 18699K->2112K(19136K), 0.0049320 secs] 76337K->60788K(112412K) icms_dc=4 , 0.0050890 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
>> 2010-12-20T17:25:35.963+0000: 23.650: [GC 23.650: [ParNew: 19073K->1947K(19136K), 0.0080830 secs] 77750K->61429K(112412K) icms_dc=4 , 0.0082030 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
>> 2010-12-20T17:25:36.237+0000: 23.924: [GC 23.924: [ParNew: 18964K->1547K(19136K), 0.0041330 secs] 78447K->61930K(112412K) icms_dc=4 , 0.0042610 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
>> 2010-12-20T17:25:36.361+0000: 24.048: [GC 24.048: [ParNew: 18514K->987K(19136K), 0.0038260 secs] 78897K->61370K(112412K) icms_dc=4 , 0.0039330 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
>> 2010-12-20T17:25:36.454+0000: 24.141: [CMS-concurrent-mark: 0.089/0.790 secs] [Times: user=36893488016.88 sys=0.06, real=0.79 secs]
>> 2010-12-20T17:25:36.454+0000: 24.141: [CMS-concurrent-preclean-start]
>> 2010-12-20T17:25:36.460+0000: 24.147: [CMS-concurrent-preclean: 0.005/0.006 secs] [Times: user=0.00 sys=0.02, real=0.01 secs]
>> 2010-12-20T17:25:36.460+0000: 24.147: [CMS-concurrent-abortable-preclean-start]
>> 2010-12-20T17:25:36.520+0000: 24.207: [GC 24.207: [ParNew: 17969K->1341K(19136K), 0.0023940 secs] 78352K->61724K(112412K) icms_dc=4 , 0.0025130 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:36.728+0000: 24.414: [GC 24.414: [ParNew: 18356K->1010K(19136K), 0.0025740 secs] 78739K->61870K(112412K) icms_dc=4 , 0.0026990 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
>> 2010-12-20T17:25:36.800+0000: 24.487: [CMS-concurrent-abortable-preclean: 0.009/0.340 secs] [Times: user=0.26 sys=0.02, real=0.34 secs]
>> 2010-12-20T17:25:36.869+0000: 24.555: [GC[YG occupancy: 9861 K (19136 K)]24.555: [Rescan (parallel) , 0.0012070 secs]24.557: [weak refs processing, 0.0000890 secs] [1 CMS-remark: 60859K(93276K)] 70720K(112412K), 0.0014050 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:36.870+0000: 24.557: [CMS-concurrent-sweep-start]
>> 2010-12-20T17:25:36.948+0000: 24.634: [GC 24.634: [ParNew: 18027K->1183K(19136K), 0.0035250 secs] 78793K->61949K(112412K) icms_dc=4 , 0.0036410 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:37.088+0000: 24.775: [CMS-concurrent-sweep: 0.015/0.218 secs] [Times: user=0.05 sys=0.00, real=0.21 secs]
>> 2010-12-20T17:25:37.088+0000: 24.775: [CMS-concurrent-reset-start]
>> 2010-12-20T17:25:37.105+0000: 24.791: [CMS-concurrent-reset: 0.016/0.016 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
>> 2010-12-20T17:25:37.155+0000: 24.842: [GC 24.842: [ParNew: 18153K->961K(19136K), 0.0233150 secs] 76441K->59351K(116284K) icms_dc=0 , 0.0234410 secs] [Times: user=0.02 sys=0.00, real=0.03 secs]
>> 2010-12-20T17:25:37.293+0000: 24.980: [GC 24.980: [ParNew: 17985K->1566K(19136K), 0.0078520 secs] 76375K->59957K(116284K) icms_dc=0 , 0.0079770 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
>> 2010-12-20T17:25:37.470+0000: 25.157: [GC 25.157: [ParNew: 18533K->2095K(19136K), 0.0028390 secs] 76923K->60833K(116284K) icms_dc=0 , 0.0029650 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:37.628+0000: 25.315: [GC 25.315: [ParNew: 19119K->2112K(19136K), 0.0032590 secs] 77857K->61573K(116284K) icms_dc=0 , 0.0033810 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
>> 2010-12-20T17:25:37.692+0000: 25.379: [GC 25.379: [ParNew: 19092K->1664K(19136K), 0.0027200 secs] 78554K->61695K(116284K) icms_dc=0 , 0.0028400 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:37.814+0000: 25.501: [GC 25.501: [ParNew: 18688K->1461K(19136K), 0.0017560 secs] 78719K->61492K(116284K) icms_dc=0 , 0.0018670 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:37.946+0000: 25.632: [GC 25.632: [ParNew: 18449K->1309K(19136K), 0.0052360 secs] 78481K->61761K(116284K) icms_dc=0 , 0.0053520 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
>> 2010-12-20T17:25:38.114+0000: 25.800: [GC 25.800: [ParNew: 18333K->2112K(19136K), 0.0046840 secs] 78785K->64510K(116284K) icms_dc=0 , 0.0048140 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:38.229+0000: 25.915: [GC 25.915: [ParNew: 19132K->1927K(19136K), 0.0301210 secs] 81531K->65315K(116284K) icms_dc=0 , 0.0302420 secs] [Times: user=0.03 sys=0.00, real=0.03 secs]
>> 2010-12-20T17:25:38.404+0000: 26.091: [GC 26.091: [ParNew: 18887K->1639K(19136K), 0.0029570 secs] 82275K->65027K(116284K) icms_dc=0 , 0.0030810 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:38.551+0000: 26.237: [GC 26.237: [ParNew: 18655K->2092K(19136K), 0.0040070 secs] 82042K->66522K(116284K) icms_dc=0 , 0.0041290 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
>> 2010-12-20T17:25:38.810+0000: 26.497: [GC 26.497: [ParNew: 19101K->1244K(19136K), 0.0036340 secs] 83531K->66458K(116284K) icms_dc=0 , 0.0037620 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:38.920+0000: 26.607: [GC 26.607: [ParNew: 18253K->684K(19136K), 0.0014730 secs] 83468K->65898K(116284K) icms_dc=0 , 0.0015700 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:39.065+0000: 26.751: [GC 26.751: [ParNew: 17669K->782K(19136K), 0.0015630 secs] 82884K->65996K(116284K) icms_dc=0 , 0.0016660 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
>> 2010-12-20T17:25:39.213+0000: 26.900: [GC 26.900: [ParNew: 17756K->783K(19136K), 0.0015390 secs] 82970K->65998K(116284K) icms_dc=0 , 0.0016400 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:39.374+0000: 27.061: [GC 27.061: [ParNew: 17787K->1548K(19136K), 0.0023500 secs] 83001K->66975K(116284K) icms_dc=0 , 0.0024690 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
>> 2010-12-20T17:25:39.624+0000: 27.311: [GC 27.311: [ParNew: 18571K->798K(19136K), 0.0015980 secs] 83998K->66324K(116284K) icms_dc=0 , 0.0016950 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:39.736+0000: 27.423: [GC 27.423: [ParNew: 17771K->936K(19136K), 0.0014100 secs] 83297K->66463K(116284K) icms_dc=0 , 0.0015130 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:39.906+0000: 27.592: [GC 27.592: [ParNew: 17920K->1886K(19136K), 0.0020160 secs] 83447K->67413K(116284K) icms_dc=0 , 0.0021370 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
>> 2010-12-20T17:25:40.042+0000: 27.729: [GC 27.729: [ParNew: 18905K->1886K(19136K), 0.0040910 secs] 84431K->68678K(116284K) icms_dc=0 , 0.0042270 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:40.213+0000: 27.900: [GC 27.900: [ParNew: 18855K->1500K(19136K), 0.0031110 secs] 85647K->68745K(116284K) icms_dc=0 , 0.0032440 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:40.323+0000: 28.010: [GC 28.010: [ParNew: 18524K->2112K(19136K), 0.0060030 secs] 85769K->72197K(116284K) icms_dc=4 , 0.0061260 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:40.484+0000: 28.170: [GC [1 CMS-initial-mark: 70085K(97148K)] 80410K(116284K), 0.0038900 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
>> 2010-12-20T17:25:40.488+0000: 28.175: [CMS-concurrent-mark-start]
>> 2010-12-20T17:25:40.567+0000: 28.254: [GC 28.254: [ParNew: 19136K->1886K(19136K), 0.0043620 secs] 89221K->72619K(116284K) icms_dc=10 , 0.0044950 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:40.752+0000: 28.439: [GC 28.439: [ParNew: 18883K->1596K(19136K), 0.0045560 secs] 89617K->73198K(116284K) icms_dc=10 , 0.0046680 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
>> 2010-12-20T17:25:41.018+0000: 28.704: [GC 28.705: [ParNew: 18590K->650K(19136K), 0.0046350 secs] 90191K->72706K(116284K) icms_dc=10 , 0.0047500 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:41.129+0000: 28.816: [GC 28.816: [ParNew: 17674K->399K(19136K), 0.0014520 secs] 89730K->72455K(116284K) icms_dc=10 , 0.0015540 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:41.290+0000: 28.977: [GC 28.977: [ParNew: 17396K->581K(19136K), 0.0021900 secs] 89452K->72637K(116284K) icms_dc=10 , 0.0022930 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
>> 2010-12-20T17:25:41.478+0000: 29.164: [GC 29.165: [ParNew: 17569K->929K(19136K), 0.0018590 secs] 89625K->72986K(116284K) icms_dc=10 , 0.0019820 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:41.653+0000: 29.340: [GC 29.340: [ParNew: 17894K->1013K(19136K), 0.0333480 secs] 89950K->73172K(116284K) icms_dc=10 , 0.0334710 secs] [Times: user=0.07 sys=0.00, real=0.04 secs]
>> 2010-12-20T17:25:41.763+0000: 29.450: [CMS-concurrent-mark: 0.101/1.275 secs] [Times: user=18446743473.28 sys=0.08, real=1.28 secs]
>> 2010-12-20T17:25:41.763+0000: 29.450: [CMS-concurrent-preclean-start]
>> 2010-12-20T17:25:41.887+0000: 29.574: [GC 29.574: [ParNew: 18024K->1580K(19136K), 0.0025720 secs] 90183K->73783K(116284K) icms_dc=10 , 0.0026960 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:41.948+0000: 29.634: [CMS-concurrent-preclean: 0.006/0.184 secs] [Times: user=0.11 sys=0.01, real=0.18 secs]
>> 2010-12-20T17:25:41.948+0000: 29.634: [CMS-concurrent-abortable-preclean-start]
>> 2010-12-20T17:25:42.010+0000: 29.697: [GC 29.697: [ParNew: 18571K->1529K(19136K), 0.0261550 secs] 90773K->74878K(116284K) icms_dc=10 , 0.0262850 secs] [Times: user=0.01 sys=0.01, real=0.03 secs]
>> 2010-12-20T17:25:42.037+0000: 29.723: [CMS-concurrent-abortable-preclean: 0.002/0.089 secs] [Times: user=0.05 sys=0.01, real=0.09 secs]
>> 2010-12-20T17:25:42.037+0000: 29.724: [GC[YG occupancy: 1657 K (19136 K)]29.724: [Rescan (parallel) , 0.0021770 secs]29.726: [weak refs processing, 0.0002560 secs] [1 CMS-remark: 73349K(97148K)] 75006K(116284K), 0.0025180 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:42.040+0000: 29.727: [CMS-concurrent-sweep-start]
>> 2010-12-20T17:25:42.143+0000: 29.830: [GC 29.830: [ParNew: 18553K->2112K(19136K), 0.0039990 secs] 91513K->76482K(116284K) icms_dc=10 , 0.0041230 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:42.159+0000: 29.846: [CMS-concurrent-sweep: 0.021/0.119 secs] [Times: user=0.11 sys=0.02, real=0.12 secs]
>> 2010-12-20T17:25:42.159+0000: 29.846: [CMS-concurrent-reset-start]
>> 2010-12-20T17:25:42.180+0000: 29.867: [CMS-concurrent-reset: 0.021/0.021 secs] [Times: user=0.02 sys=0.01, real=0.02 secs]
>> 2010-12-20T17:25:42.220+0000: 29.907: [GC [1 CMS-initial-mark: 72204K(120344K)] 82008K(139480K), 0.0031690 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
>> 2010-12-20T17:25:42.223+0000: 29.910: [CMS-concurrent-mark-start]
>> 2010-12-20T17:25:42.363+0000: 30.050: [GC 30.050: [ParNew: 19089K->1911K(19136K), 0.0287550 secs] 91294K->74871K(139480K) icms_dc=5 , 0.0288700 secs] [Times: user=0.02 sys=0.00, real=0.03 secs]
>> 2010-12-20T17:25:42.520+0000: 30.207: [GC 30.207: [ParNew: 18890K->2095K(19136K), 0.0026020 secs] 91850K->76245K(139480K) icms_dc=5 , 0.0027290 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:42.626+0000: 30.313: [GC 30.313: [ParNew: 19116K->1072K(19136K), 0.0021620 secs] 93267K->75735K(139480K) icms_dc=5 , 0.0022790 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:42.794+0000: 30.480: [GC 30.480: [ParNew: 18095K->1985K(19136K), 0.0224570 secs] 92758K->76648K(139480K) icms_dc=5 , 0.0225750 secs] [Times: user=0.02 sys=0.00, real=0.03 secs]
>> 2010-12-20T17:25:43.028+0000: 30.715: [GC 30.715: [ParNew: 18975K->1230K(19136K), 0.0029720 secs] 93638K->76922K(139480K) icms_dc=5 , 0.0031070 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:43.127+0000: 30.814: [CMS-concurrent-mark: 0.101/0.904 secs] [Times: user=45.86 sys=0.07, real=0.90 secs]
>> 2010-12-20T17:25:43.127+0000: 30.814: [CMS-concurrent-preclean-start]
>> 2010-12-20T17:25:43.140+0000: 30.826: [CMS-concurrent-preclean: 0.010/0.012 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
>> 2010-12-20T17:25:43.140+0000: 30.826: [CMS-concurrent-abortable-preclean-start]
>> 2010-12-20T17:25:43.263+0000: 30.950: [GC 30.950: [ParNew: 18242K->1343K(19136K), 0.0013570 secs] 93934K->77035K(139480K) icms_dc=5 , 0.0014830 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:43.439+0000: 31.126: [GC 31.126: [ParNew: 18367K->1276K(19136K), 0.0013350 secs] 94059K->76968K(139480K) icms_dc=5 , 0.0014590 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:43.539+0000: 31.225: [CMS-concurrent-abortable-preclean: 0.008/0.399 secs] [Times: user=0.16 sys=0.02, real=0.40 secs]
>> 2010-12-20T17:25:43.539+0000: 31.226: [GC[YG occupancy: 11143 K (19136 K)]31.226: [Rescan (parallel) , 0.0012840 secs]31.227: [weak refs processing, 0.0000800 secs] [1 CMS-remark: 75692K(120344K)] 86835K(139480K), 0.0014660 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:43.566+0000: 31.252: [CMS-concurrent-sweep-start]
>> 2010-12-20T17:25:43.588+0000: 31.275: [CMS-concurrent-sweep: 0.022/0.022 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
>> 2010-12-20T17:25:43.588+0000: 31.275: [CMS-concurrent-reset-start]
>> 2010-12-20T17:25:43.606+0000: 31.292: [CMS-concurrent-reset: 0.017/0.017 secs] [Times: user=0.01 sys=0.00, real=0.02 secs]
>> 2010-12-20T17:25:43.617+0000: 31.303: [GC 31.303: [ParNew: 18237K->1557K(19136K), 0.0016650 secs] 91506K->74826K(141252K) icms_dc=0 , 0.0017980 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:43.766+0000: 31.452: [GC 31.452: [ParNew: 18525K->986K(19136K), 0.0019760 secs] 91794K->74744K(141252K) icms_dc=0 , 0.0021000 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:43.984+0000: 31.671: [GC 31.671: [ParNew: 18010K->1004K(19136K), 0.0011630 secs] 91768K->74762K(141252K) icms_dc=0 , 0.0012740 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:44.155+0000: 31.842: [GC 31.842: [ParNew: 17986K->1535K(19136K), 0.0020850 secs] 91744K->75294K(141252K) icms_dc=0 , 0.0022080 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:44.279+0000: 31.965: [GC 31.965: [ParNew: 18525K->1844K(19136K), 0.0036450 secs] 92284K->75963K(141252K) icms_dc=0 , 0.0037850 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
>> 2010-12-20T17:25:44.446+0000: 32.132: [GC 32.133: [ParNew: 18868K->2112K(19136K), 0.0027860 secs] 92987K->76905K(141252K) icms_dc=0 , 0.0029120 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
>> 2010-12-20T17:25:44.560+0000: 32.247: [GC 32.247: [ParNew: 19136K->2062K(19136K), 0.0035670 secs] 93929K->77864K(141252K) icms_dc=0 , 0.0036890 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
>> 2010-12-20T17:25:44.786+0000: 32.473: [GC 32.473: [ParNew: 19081K->2099K(19136K), 0.0042870 secs] 94883K->79044K(141252K) icms_dc=0 , 0.0044120 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:45.051+0000: 32.737: [GC 32.738: [ParNew: 19082K->2112K(19136K), 0.0057110 secs] 96026K->79932K(141252K) icms_dc=0 , 0.0058410 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:45.159+0000: 32.845: [GC 32.845: [ParNew: 19132K->1902K(19136K), 0.0048970 secs] 96953K->80775K(141252K) icms_dc=0 , 0.0050310 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:45.315+0000: 33.001: [GC 33.001: [ParNew: 18888K->1172K(19136K), 0.0016980 secs] 97761K->80045K(141252K) icms_dc=0 , 0.0018120 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
>> 2010-12-20T17:25:45.416+0000: 33.103: [GC 33.103: [ParNew: 18143K->1113K(19136K), 0.0016580 secs] 97016K->79987K(141252K) icms_dc=0 , 0.0017670 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:45.571+0000: 33.258: [GC 33.258: [ParNew: 18117K->1175K(19136K), 0.0018150 secs] 96990K->80049K(141252K) icms_dc=0 , 0.0019290 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:45.716+0000: 33.403: [GC 33.403: [ParNew: 18193K->2092K(19136K), 0.0243240 secs] 97067K->83760K(141252K) icms_dc=0 , 0.0244520 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
>> 2010-12-20T17:25:46.275+0000: 33.961: [GC 33.961: [ParNew: 19116K->1282K(19136K), 0.0251550 secs] 100784K->83201K(141252K) icms_dc=0 , 0.0252890 secs] [Times: user=0.06 sys=0.00, real=0.03 secs]
>> 2010-12-20T17:25:46.421+0000: 34.108: [GC 34.108: [ParNew: 18286K->1295K(19136K), 0.0186440 secs] 100205K->83214K(141252K) icms_dc=0 , 0.0187610 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
>> 2010-12-20T17:25:46.676+0000: 34.363: [GC 34.363: [ParNew: 18319K->2112K(19136K), 0.0062280 secs] 100238K->84044K(141252K) icms_dc=0 , 0.0063550 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:46.825+0000: 34.512: [GC 34.512: [ParNew: 19136K->2112K(19136K), 0.0235520 secs] 101068K->87526K(141252K) icms_dc=0 , 0.0236890 secs] [Times: user=0.02 sys=0.00, real=0.03 secs]
>> 2010-12-20T17:25:47.010+0000: 34.696: [GC 34.696: [ParNew: 19136K->1126K(19136K), 0.0036520 secs] 104550K->86998K(141252K) icms_dc=0 , 0.0037680 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:47.179+0000: 34.866: [GC 34.866: [ParNew: 18094K->2112K(19136K), 0.0036740 secs] 103967K->89106K(141252K) icms_dc=0 , 0.0037950 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:47.354+0000: 35.041: [GC 35.041: [ParNew: 19136K->1976K(19136K), 0.0042910 secs] 106130K->90305K(141252K) icms_dc=0 , 0.0044170 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:47.523+0000: 35.210: [GC 35.210: [ParNew: 18973K->2112K(19136K), 0.0020850 secs] 107303K->90449K(141252K) icms_dc=0 , 0.0022090 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:47.757+0000: 35.443: [GC 35.443: [ParNew: 19086K->1740K(19136K), 0.0047220 secs] 107424K->91042K(141252K) icms_dc=0 , 0.0048520 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:47.949+0000: 35.635: [GC 35.635: [ParNew: 18758K->1349K(19136K), 0.0100480 secs] 108060K->90651K(141252K) icms_dc=0 , 0.0101550 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
>> 2010-12-20T17:25:48.160+0000: 35.846: [GC 35.847: [ParNew: 18373K->1062K(19136K), 0.0035050 secs] 107675K->90707K(141252K) icms_dc=0 , 0.0036170 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:48.363+0000: 36.050: [GC 36.050: [ParNew: 18057K->746K(19136K), 0.0150130 secs] 107702K->90391K(141252K) icms_dc=0 , 0.0151200 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
>> 2010-12-20T17:25:48.547+0000: 36.234: [GC 36.234: [ParNew: 17752K->2057K(19136K), 0.0048080 secs] 107397K->92215K(141252K) icms_dc=0 , 0.0049410 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:48.755+0000: 36.442: [GC 36.442: [ParNew: 19042K->1221K(19136K), 0.0231150 secs] 109199K->91615K(141252K) icms_dc=0 , 0.0232640 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
>> 2010-12-20T17:25:48.867+0000: 36.554: [GC 36.554: [ParNew: 18227K->2102K(19136K), 0.0218400 secs] 108621K->95180K(141252K) icms_dc=0 , 0.0219730 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
>> 2010-12-20T17:25:49.086+0000: 36.773: [GC 36.773: [ParNew: 19090K->1186K(19136K), 0.0040110 secs] 112167K->94679K(141252K) icms_dc=0 , 0.0041370 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:49.186+0000: 36.873: [GC 36.873: [ParNew: 18171K->1369K(19136K), 0.0018640 secs] 111664K->94862K(141252K) icms_dc=0 , 0.0019870 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:49.311+0000: 36.998: [GC 36.998: [ParNew: 18393K->154K(19136K), 0.0049150 secs] 111886K->94007K(141252K) icms_dc=0 , 0.0050180 secs] [Times: user=0.00 sys=0.01, real=0.01 secs]
>> 2010-12-20T17:25:49.433+0000: 37.120: [GC 37.120: [ParNew: 17165K->2079K(19136K), 0.0081480 secs] 111018K->98366K(141252K) icms_dc=4 , 0.0082710 secs] [Times: user=0.00 sys=0.01, real=0.01 secs]
>> 2010-12-20T17:25:49.544+0000: 37.230: [GC [1 CMS-initial-mark: 96287K(122116K)] 106652K(141252K), 0.0041720 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:25:49.548+0000: 37.235: [CMS-concurrent-mark-start]
>> 2010-12-20T17:25:49.706+0000: 37.393: [GC 37.393: [ParNew: 19059K->818K(19136K), 47.8334700 secs] 115346K->97662K(141252K) icms_dc=10 , 47.8335900 secs] [Times: user=90.47 sys=0.01, real=47.86 secs]
>> 2010-12-20T17:26:37.884+0000: 85.570: [GC 85.571: [ParNew: 17836K->541K(19136K), 0.0405420 secs] 114680K->97386K(141252K) icms_dc=10 , 0.0406480 secs] [Times: user=0.00 sys=0.00, real=0.04 secs]
>> 2010-12-20T17:26:47.779+0000: 95.466: [CMS-concurrent-mark: 0.103/58.231 secs] [Times: user=36893487576.24 sys=0.98, real=58.22 secs]
>> 2010-12-20T17:26:47.779+0000: 95.466: [CMS-concurrent-preclean-start]
>> 2010-12-20T17:26:47.790+0000: 95.476: [CMS-concurrent-preclean: 0.008/0.010 secs] [Times: user=0.00 sys=0.01, real=0.01 secs]
>> 2010-12-20T17:26:47.790+0000: 95.476: [CMS-concurrent-abortable-preclean-start]
>> 2010-12-20T17:26:47.790+0000: 95.476: [CMS-concurrent-abortable-preclean: 0.000/0.000 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:26:47.790+0000: 95.477: [GC[YG occupancy: 3204 K (19136 K)]95.477: [Rescan (parallel) , 0.0018430 secs]95.479: [weak refs processing, 0.0006910 secs] [1 CMS-remark: 96844K(122116K)] 100048K(141252K), 0.0026330 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
>> 2010-12-20T17:26:47.793+0000: 95.479: [CMS-concurrent-sweep-start]
>> 2010-12-20T17:26:47.826+0000: 95.513: [CMS-concurrent-sweep: 0.033/0.033 secs] [Times: user=0.03 sys=0.00, real=0.03 secs]
>> 2010-12-20T17:26:47.826+0000: 95.513: [CMS-concurrent-reset-start]
>> 2010-12-20T17:26:47.842+0000: 95.528: [CMS-concurrent-reset: 0.016/0.016 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
>> Heap
>> par new generation total 19136K, used 3204K [0x00007f611e340000, 0x00007f611f800000, 0x00007f6123670000)
>> eden space 17024K, 15% used [0x00007f611e340000, 0x00007f611e5d9b38, 0x00007f611f3e0000)
>> from space 2112K, 25% used [0x00007f611f3e0000, 0x00007f611f467768, 0x00007f611f5f0000)
>> to space 2112K, 0% used [0x00007f611f5f0000, 0x00007f611f5f0000, 0x00007f611f800000)
>> concurrent mark-sweep generation total 122116K, used 73206K [0x00007f6123670000, 0x00007f612adb1000, 0x00007f6295340000)
>> concurrent-mark-sweep perm gen total 29708K, used 17810K [0x00007f6295340000, 0x00007f6297043000, 0x00007f629a740000)
>>
>>
>>
>> On Dec 9, 2010, at 4:09 PM, Lance Riedel wrote:
>>
>>> I came across just probably the culprit - I just discovered that the machines that were dying had OS swap turned OFF.
>>>
>>> So, I added swap to those machines, but also reduced the memory configs for tasks. (was at 1 gig, now at 512m). Seems to be stable right now, but the nightly longer term jobs run at night, so we will see how they do.
>>>
>>> This is an ec2 c1.xlarge, so the memory is at 8 gigs. I had 4 mappers per node, which was fine when all they had on them were Tasktrackers and datanodes. But with the hbase regionservers, the machines could definitely get over-allocated.
>>>
>>> Lance
>>>
>>> On Dec 9, 2010, at 3:57 PM, Ted Dunning wrote:
>>>
>>>> This could indicate swapping during GC.
>>>>
>>>> On Thu, Dec 9, 2010 at 12:13 PM, Lance Riedel <la...@gmail.com> wrote:
>>>>
>>>>> Seems reasonable, but having trouble making sense of the GC logs I had
>>>>> turned on. Basically since there was a full GC a minute before this happens
>>>>> on that server that lasts less than a second.
>>>>>
>>>>>
>>>>> Example:
>>>>> So, here is what the last of the GC logs say for that Regionserver
>>>>> (04.hadoop on 10.100.154.10) (note the last full GC only 1 minutes before @
>>>>> 5:03:29.850+0000):
>>>>>
>>>>> 2010-12-09T05:02:19.472+0000: 12170.437: [GC [PSYoungGen:
>>>>> 278229K->5024K(287360K)] 694269K->421693K(721856K), 0.0274840 secs] [Times:
>>>>> user=0.04 sys=0.00, real=0.03 secs]
>>>>> 2010-12-09T05:02:55.479+0000: 12206.444: [GC [PSYoungGen:
>>>>> 281450K->7968K(273344K)] 698120K->429658K(707840K), 0.0503000 secs] [Times:
>>>>> user=0.10 sys=0.00, real=0.05 secs]
>>>>> 2010-12-09T05:03:29.800+0000: 12240.766: [GC [PSYoungGen:
>>>>> 273295K->7712K(262464K)] 694986K->437306K(696960K), 0.0490770 secs] [Times:
>>>>> user=0.12 sys=0.01, real=0.05 secs]
>>>>> 2010-12-09T05:03:29.850+0000: 12240.815: [Full GC [PSYoungGen:
>>>>> 7712K->0K(262464K)] [PSOldGen: 429594K->353152K(429824K)]
>>>>> 437306K->353152K(692288K) [PSPermGen: 18016K->18016K(21248K)], 0.8960760
>>>>> secs] [Times: user=0.89 sys=0.00, real=0.90 secs]
>>>>> 2010-12-09T05:04:03.345+0000: 12274.310: [GC [PSYoungGen:
>>>>> 254720K->6944K(251584K)] 607872K->360528K(681408K), 0.0331060 secs] [Times:
>>>>> user=0.03 sys=0.00, real=0.04 secs]
>>>>> 2010-12-09T05:04:33.875+0000: 12304.840: [GC [PSYoungGen:
>>>>> 251552K->7200K(242240K)] 605136K->367800K(672064K), 0.0336420 secs] [Times:
>>>>> user=0.09 sys=0.01, real=0.03 secs]
>>>>> Heap
>>>>> PSYoungGen total 242240K, used 223390K [0x00007fcc640b0000,
>>>>> 0x00007fcc77a90000, 0x00007fcc78e00000)
>>>>> eden space 235008K, 91% used
>>>>> [0x00007fcc640b0000,0x00007fcc713cfa30,0x00007fcc72630000)
>>>>> from space 7232K, 99% used
>>>>> [0x00007fcc769b0000,0x00007fcc770b8090,0x00007fcc770c0000)
>>>>> to space 10048K, 0% used
>>>>> [0x00007fcc770c0000,0x00007fcc770c0000,0x00007fcc77a90000)
>>>>> PSOldGen total 429824K, used 360600K [0x00007fcc3a600000,
>>>>> 0x00007fcc549c0000, 0x00007fcc640b0000)
>>>>> object space 429824K, 83% used
>>>>> [0x00007fcc3a600000,0x00007fcc50626060,0x00007fcc549c0000)
>>>>> PSPermGen total 21248K, used 18131K [0x00007fcc35200000,
>>>>> 0x00007fcc366c0000, 0x00007fcc3a600000)
>>>>> object space 21248K, 85% used
>>>>> [0x00007fcc35200000,0x00007fcc363b4d60,0x00007fcc366c0000)
>>>>>
>>>>> Now Dead / End of log
>>>>>
>>>>>
>>>>> That said, I will continue down the path you pointed me on .
>>>>>
>>>>> Thanks,
>>>>> Lance
>>>>>
>>>>> On Dec 9, 2010, at 11:40 AM, Jean-Daniel Cryans wrote:
>>>>>
>>>>>> Lance,
>>>>>>
>>>>>> Both those lines indicate the problem:
>>>>>>
>>>>>> IPC Server handler 13 on 60020 took 182416ms
>>>>>> Client session timed out, have not heard from server in 182936ms
>>>>>>
>>>>>> It's very clear that your region servers are suffering from
>>>>>> pause-of-the-world garbage collection issues. Basically this one GC'ed
>>>>>> for 3 minutes, which is over the 1 minute timeout to consider a region
>>>>>> server dead. The rest is just the side effect of that.
>>>>>>
>>>>>> This subject comes often on the mailing list, for example:
>>>>>> http://search-hadoop.com/m/t97q027tr7f2
>>>>>>
>>>>>> Make sure you don't swap, give plenty of RAM to HBase, use LZO
>>>>>> compression, don't underprovision your machines, etc
>>>>>>
>>>>>> J-D
>>>>>>
>>>>>> On Thu, Dec 9, 2010 at 11:27 AM, Lance Riedel <la...@gmail.com>
>>>>> wrote:
>>>>>>>
>>>>>>> We have a 6 node cluster, 5 with region serves. 2 of the region servers
>>>>> have been stable for days, but 3 of them keep crashing. Here are the logs
>>>>> around around when the crash occurs. (btw, we are shoving approximately the
>>>>> twitter firehose into hbase via flume) I'm an hbase newbie, but I have been
>>>>> reading. Not sure what else is needed to help debug. When the problem
>>>>> occurs, it is concurrent with issues that appear all the way down in the
>>>>> data node. Also, interesting to note, all 3 servers seem to fail
>>>>> differently somewhat:
>>>>>>>
>>>>>>> Servers (6):
>>>>>>> 00.hadoop is the master
>>>>>>> 01.hadoop, 03.hadoop (there is no 02.hadoop - flaky machine that got
>>>>> taken out) - are stable Region Servers
>>>>>>> 04-06.hadoop crash at different times, very different logs, Region
>>>>> Servers
>>>>>>>
>>>>>>>
>>>>>>> Master Attributes:
>>>>>>> HBase Version 0.89.20100924+28, r
>>>>>>> Hadoop Version 0.20.2+737, r98c55c28258aa6f42250569bd7fa431ac657bdbd
>>>>>>>
>>>>>>> HBase Root Directory hdfs://00.hadoop.****:54310/hbase
>>>>>>>
>>>>>>> Load average
>>>>>>> 103.0
>>>>>>> Average number of regions per regionserver. Naive computation.
>>>>>>> Regions On FS
>>>>>>> 577
>>>>>>>
>>>>>>>
>>>>>>> (This is after recently restarting 04-06)
>>>>>>>
>>>>>>>
>>>>>>> Address Start Code Load
>>>>>>> 01.hadoop:60030 1291268910638 requests=223, regions=118,
>>>>> usedHeap=519, maxHeap=987
>>>>>>> 03.hadoop:60030 1291269219610 requests=23, regions=111, usedHeap=862,
>>>>> maxHeap=987
>>>>>>> 04.hadoop:60030 1291910665912 requests=169, regions=82, usedHeap=194,
>>>>> maxHeap=888
>>>>>>> 05.hadoop:60030 1291909584060 requests=232, regions=110,
>>>>> usedHeap=477, maxHeap=888
>>>>>>> 06.hadoop60030 1291909723787 requests=99, regions=94, usedHeap=394,
>>>>> maxHeap=888
>>>>>>> Total:
>>>>>>> servers: 5
>>>>>>> requests=746, regions=515
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Following are various logs around the time of the failures for 04-06
>>>>>>>
>>>>>>>
>>>>> ************************************************************************************
>>>>>>> 04.hadoop
>>>>>>>
>>>>> ************************************************************************************
>>>>>>> REGION SERVER ->
>>>>>>> 2010-12-09 05:01:58,640 INFO
>>>>> org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region
>>>>> article,a83858a08f2270d319f75a7b43c756c2453988e7,1291811929868.58f6d9fc80
>>>>>>> c78f3ca490b0280b4f1226. in 0sec
>>>>>>> 2010-12-09 05:04:30,898 DEBUG
>>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.46 MB,
>>>>> free=176.33 MB, max=177.79 MB, blocks=0, accesses=200583, hits=0, hitRatio=
>>>>>>> 0.00%%, evictions=0, evicted=0, evictedPerRun=NaN
>>>>>>> 2010-12-09 05:04:59,612 INFO
>>>>> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs
>>>>> -- HDFS-200
>>>>>>> 2010-12-09 05:04:59,618 INFO
>>>>> org.apache.hadoop.hbase.regionserver.wal.HLog: Roll
>>>>> /hbase/.logs/04.hadoop.****,60020,1291858770125/10.100.154.103%3A60020.1291870585253,
>>>>> entries=5318
>>>>>>> 6, filesize=63776311. New hlog
>>>>> /hbase/.logs/04.hadoop.****,60020,1291858770125/10.100.154.103
>>>>> %3A60020.1291871099582
>>>>>>> 2010-12-09 05:08:02,033 DEBUG
>>>>> org.apache.hadoop.hbase.regionserver.wal.HLog: Found 1 hlogs to remove out
>>>>> of total 6; oldest outstanding sequenceid is 32891801 from region article
>>>>>>>
>>>>> ,4b2039b791e894dd479b90661ca97087f61645d3,1291813201039.7dff47c63bb5648bb1ee6670c60553a5.
>>>>>>> 2010-12-09 05:08:02,034 WARN
>>>>> org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler 13 on
>>>>> 60020 took 182416ms appending an edit to hlog; editcount=0
>>>>>>> 2010-12-09 05:08:02,066 INFO
>>>>> org.apache.hadoop.hbase.regionserver.wal.HLog: moving old hlog file
>>>>> /hbase/.logs/04.hadoop.****,60020,1291858770125/10.100.154.103
>>>>> %3A60020.12918679476
>>>>>>> 65 whose highest sequenceid is 32880227 to /hbase/.oldlogs/
>>>>> 10.100.154.103%3A60020.1291867947665
>>>>>>> 2010-12-09 05:08:02,066 INFO org.apache.zookeeper.ClientCnxn: Client
>>>>> session timed out, have not heard from server in 182936ms for sessionid
>>>>> 0x12ca58c08ff0048, closing socket conn
>>>>>>> ection and attempting reconnect
>>>>>>> 2010-12-09 05:08:02,089 FATAL
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Aborting region server
>>>>> serverName=04.hadoop.****,60020,1291858770125, load=(requests=24, regions=
>>>>>>> 64, usedHeap=548, maxHeap=888): Unhandled exception
>>>>>>> org.apache.hadoop.hbase.YouAreDeadException:
>>>>> org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected;
>>>>> currently processing 04.hadoop.****,60020,1291858770125 as dead s
>>>>>>> erver
>>>>>>> at
>>>>> org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:217)
>>>>>>> at
>>>>> org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:271)
>>>>>>> at
>>>>> org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:744)
>>>>>>> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>>>>>> at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>> at
>>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:561)
>>>>>>> at
>>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>>>>>>>
>>>>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>>>> Method)
>>>>>>> at
>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>>>>>>> at
>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>>>>>>> at
>>>>> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
>>>>>>> at
>>>>> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
>>>>>>> at
>>>>> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
>>>>>>> at
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.doRun(HRegionServer.java:635)
>>>>>>> at
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.access$000(HRegionServer.java:126)
>>>>>>> at
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer$1.run(HRegionServer.java:518)
>>>>>>> at
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer$1.run(HRegionServer.java:516)
>>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>>> at javax.security.auth.Subject.doAs(Subject.java:337)
>>>>>>> at
>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1046)
>>>>>>> at
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:516)
>>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>>> 2010-12-09 05:08:02,090 INFO
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
>>>>> request=0.0, regions=64, stores=64, storefiles=136, storefileIndexSize=27,
>>>>> memsto
>>>>>>> reSize=353, compactionQueueSize=0, usedHeap=549, maxHeap=888,
>>>>> blockCacheSize=1530552, blockCacheFree=184893160, blockCacheCount=0,
>>>>> blockCacheHitRatio=0
>>>>>>> 2010-12-09 05:08:02,090 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
>>>>> server on 60020
>>>>>>> 2010-12-09 05:08:02,090 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>>>> Server handler 0 on 60020: exiting2010-12-09 05:08:02,090 INFO
>>>>> org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server listener on 60020
>>>>>>>
>>>>>>>
>>>>>>> MASTER HBASE ->
>>>>>>>
>>>>>>> 2010-12-09 05:05:06,072 INFO org.apache.hadoop.hbase.master.BaseScanner:
>>>>> All 1 .META. region(s) scanned
>>>>>>> 2010-12-09 05:05:40,004 INFO
>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>> 04.hadoop.xxxx,60020,1291858770125 znode expired
>>>>>>> 2010-12-09 05:05:40,005 DEBUG
>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>> Added=04.hadoop.xxxx,60020,1291858770125 to dead servers, added shutdown
>>>>> processing operation
>>>>>>> 2010-12-09 05:05:40,005 DEBUG
>>>>> org.apache.hadoop.hbase.master.RegionServerOperationQueue: Processing todo:
>>>>> ProcessServerShutdown of 04.hadoop.xxxx,60020,12918587701252010-12-09
>>>>> 05:05:40,005 INFO org.apache.hadoop.hbase.master.RegionServerOperation:
>>>>> Process shutdown of server 04.hadoop.xxxx,60020,1291858770125: logSplit:
>>>>> false, rootRescanned: f
>>>>>>> alse, numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
>>>>>>> 2010-12-09 05:05:40,008 INFO
>>>>> org.apache.hadoop.hbase.regionserver.wal.HLog: Splitting 7 hlog(s) in
>>>>> hdfs://00.hadoop.xxxx:54310/hbase/.logs/04.hadoop.xxxx,60020,12918587701252010-12-09
>>>>> 05:05:40,008 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: Splitting
>>>>> hlog 1 of 7:
>>>>> hdfs://00.hadoop.xxxx:54310/hbase/.logs/04.hadoop.xxxx,60020,1291858770125/10.1
>>>>>>> 00.154.103%3A60020.1291867947665, length=637577092010-12-09 05:05:40,008
>>>>> INFO org.apache.hadoop.hbase.util.FSUtils: Recovering
>>>>> filehdfs://00.hadoop.xxxx:54310/hbase/.logs/04.hadoop.xxxx,60020,1291858770125/
>>>>> 10.100.154.103%3A60020
>>>>>>> .1291867947665
>>>>>>>
>>>>>>>
>>>>>>> NAMENODE ->
>>>>>>>
>>>>>>> 2010-12-09 05:08:02,471 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
>>>>> NameSystem.addStoredBlock: blockMap updated: 10.100.159.13:50010 is added
>>>>> to blk_1531008743226086399_251615 size 63757709
>>>>>>> 2010-12-09 05:08:02,473 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>>> handler 9 on 54310, call
>>>>> complete(/hbase/.logs/04.hadoop.xxxx,60020,1291858770125/10.100.154.103%3A60020.1291867947665,
>>>>> DFSClient_hb_m_10.194.194.79:60000_1291788452343) from 10.194.194.79:44117:
>>>>> error: org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No
>>>>> lease on /hbase/.logs/04.hadoop.xxxx,60020,1291858770125/10.100.154.103%3A60020.1291867947665
>>>>> File does not exist. [Lease. Holder:
>>>>> DFSClient_hb_m_10.194.194.79:60000_1291788452343, pendingcreates: 1]
>>>>>>> org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease
>>>>> on /hbase/.logs/04.hadoop.xxxx,60020,1291858770125/10.100.154.103%3A60020.1291867947665
>>>>> File does not exist. [Lease. Holder:
>>>>> DFSClient_hb_m_10.194.194.79:60000_1291788452343, pendingcreates: 1]
>>>>>>> at
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1488)
>>>>>>> at
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1479)
>>>>>>> at
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:1534)
>>>>>>> at
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:1522)
>>>>>>> at
>>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.complete(NameNode.java:610)
>>>>>>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>>>>>>> at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:528)
>>>>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1319)
>>>>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1315)
>>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>>>>>> at
>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
>>>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1313)
>>>>>>> 2010-12-09 05:08:04,206 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
>>>>> NameSystem.addStoredBlock: blockMap updated: 10.99.97.106:50010 is added
>>>>> to blk_-734280257049179934_251614 size 7330
>>>>>>>
>>>>>>> DATANODE ->
>>>>>>>
>>>>>>> 2010-12-09 05:08:02,212 WARN
>>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in
>>>>> BlockReceiver.lastNodeRun: java.io.IOException: Broken pipe
>>>>>>> at sun.nio.ch.FileDispatcher.write0(Native Method)
>>>>>>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
>>>>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100) at
>>>>> sun.nio.ch.IOUtil.write(IOUtil.java:71)
>>>>>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
>>>>>>> at
>>>>> org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
>>>>> at
>>>>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
>>>>> at
>>>>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
>>>>>>> at
>>>>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
>>>>> at java.io.DataOutputStream.writeLong(DataOutputStream.java:207)
>>>>>>> at
>>>>> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.write(DataTransferProtocol.java:133)
>>>>> at
>>>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.lastDataNodeRun(BlockReceiver.java:840)
>>>>>>> at
>>>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:870)
>>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>>> 2010-12-09 05:08:02,213 WARN
>>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: checkDiskError: exception:
>>>>>>> 2010-12-09 05:08:02,213 WARN
>>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: checkDiskError: exception:
>>>>> java.io.IOException: Broken pipe
>>>>>>> at sun.nio.ch.FileDispatcher.write0(Native Method)
>>>>>>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
>>>>>>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100)
>>>>>>> at sun.nio.ch.IOUtil.write(IOUtil.java:71)
>>>>>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
>>>>>>> at
>>>>> org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
>>>>>>> at
>>>>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
>>>>>>> at
>>>>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
>>>>>>> at
>>>>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
>>>>>>> at java.io.DataOutputStream.writeLong(DataOutputStream.java:207)
>>>>>>> at
>>>>> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.write(DataTransferProtocol.java:133)
>>>>>>> at
>>>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.lastDataNodeRun(BlockReceiver.java:840)
>>>>>>> at
>>>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:870)
>>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>>> 2010-12-09 05:08:02,221 INFO
>>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder
>>>>> blk_-8817504198034990390_251613 0 Exception java.io.IOException: Broken pipe
>>>>>>> at sun.nio.ch.FileDispatcher.write0(Native Method)
>>>>>>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
>>>>>>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100)
>>>>>>> at sun.nio.ch.IOUtil.write(IOUtil.java:71)
>>>>>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
>>>>>>> at
>>>>> org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
>>>>>>> at
>>>>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
>>>>>>> at
>>>>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
>>>>>>> at
>>>>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
>>>>>>> at java.io.DataOutputStream.writeLong(DataOutputStream.java:207)
>>>>>>> at
>>>>> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.write(DataTransferProtocol.java:133)
>>>>>>> at
>>>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.lastDataNodeRun(BlockReceiver.java:840)
>>>>>>> at
>>>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:870)
>>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>> 2010-12-09 05:08:02,221 INFO
>>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block
>>>>> blk_-8817504198034990390_251613 terminating
>>>>>>> 2010-12-09 05:08:02,224 INFO
>>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock
>>>>> blk_-8817504198034990390_251613 received exception java.io.IOException:
>>>>> Connection reset by peer
>>>>>>> 2010-12-09 05:08:02,224 INFO
>>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block
>>>>> blk_4505306716668305567_250752 file
>>>>> /dist1/data/hadoop-data/current/subdir19/blk_4505306716668305567 for
>>>>> deletion
>>>>>>> 2010-12-09 05:08:02,226 ERROR
>>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
>>>>> 10.100.154.103:50010,
>>>>> storageID=DS-866555524-10.100.154.103-50010-1291262739506, infoPort=50075,
>>>>> ipcPort=50020):DataXceiver
>>>>>>> java.io.IOException: Connection reset by peer
>>>>>>> at sun.nio.ch.FileDispatcher.read0(Native Method)
>>>>>>> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
>>>>>>> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:237)
>>>>>>> at sun.nio.ch.IOUtil.read(IOUtil.java:210)
>>>>>>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
>>>>>>> at
>>>>> org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:55)
>>>>>>> at
>>>>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
>>>>>>> at
>>>>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>>>>>> at
>>>>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>>>>>> at
>>>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
>>>>>>> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
>>>>>>> at java.io.DataInputStream.read(DataInputStream.java:132)
>>>>>>> at
>>>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:267)
>>>>>>> at
>>>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:357)
>>>>>>> at
>>>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:378)
>>>>>>> at
>>>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:534)
>>>>>>> at
>>>>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:417)
>>>>>>> at
>>>>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:122)
>>>>>>> 2010-12-09 05:08:02,226 INFO
>>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block
>>>>> blk_1422641301942366074_250732 at file
>>>>> /dist1/data/hadoop-data/current/subdir19/blk_1422641301942366074
>>>>>>> 2010-12-09 05:08:02,227 INFO
>>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block
>>>>> blk_4505306716668305567_250752 at file
>>>>> /dist1/data/hadoop-data/current/subdir19/blk_4505306716668305567
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>> ************************************************************************************
>>>>>>> 05.hadoop
>>>>>>>
>>>>> ************************************************************************************
>>>>>>> REGION SERVER ->
>>>>>>> 2010-12-09 04:29:25,047 DEBUG
>>>>> org.apache.hadoop.hbase.regionserver.Store: Compaction size of data: 250.5m;
>>>>> Skipped 1 file(s), size: 206896367
>>>>>>> 2010-12-09 04:29:25,047 INFO org.apache.hadoop.hbase.regionserver.Store:
>>>>> Started compaction of 2 file(s) in data of
>>>>> article,060c4b75067378c045eead9edec2d4fb7d8f3e5e,1291812563166.95e92e97823e11e57850720a63433a62.
>>>>> into
>>>>> hdfs://00.hadoop.****:54310/hbase/article/95e92e97823e11e57850720a63433a62/.tmp,
>>>>> sequenceid=33050221
>>>>>>> 2010-12-09 04:29:26,328 INFO org.apache.hadoop.hbase.regionserver.Store:
>>>>> Completed compaction of 2 file(s) in data of
>>>>> article,060c4b75067378c045eead9edec2d4fb7d8f3e5e,1291812563166.95e92e97823e11e57850720a63433a62.;
>>>>> new storefile is
>>>>> hdfs://00.hadoop.****:54310/hbase/article/95e92e97823e11e57850720a63433a62/data/2787768957794924068;
>>>>> store size is 250.5m
>>>>>>> 2010-12-09 04:29:26,328 INFO
>>>>> org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region
>>>>> article,060c4b75067378c045eead9edec2d4fb7d8f3e5e,1291812563166.95e92e97823e11e57850720a63433a62.
>>>>> in 1sec
>>>>>>> 2010-12-09 04:29:26,328 INFO
>>>>> org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on region
>>>>> article,ed0cdc7ae6db73432e2ca47c973d545bee11ddd8,1291813400028.85a8d31d099f409dd36f3bcc4be1eba6.
>>>>>>> 2010-12-09 04:29:26,330 INFO
>>>>> org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region
>>>>> article,ed0cdc7ae6db73432e2ca47c973d545bee11ddd8,1291813400028.85a8d31d099f409dd36f3bcc4be1eba6.
>>>>> in 0sec
>>>>>>> 2010-12-09 04:32:33,081 INFO
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGIONSERVER_STOP
>>>>>>> 2010-12-09 04:32:33,081 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
>>>>> server on 60020
>>>>>>> 2010-12-09 04:32:33,089 WARN org.apache.hadoop.hdfs.DFSClient:
>>>>> DFSOutputStream ResponseProcessor exception for block
>>>>> blk_2012253533858803341_250248java.io.EOFException
>>>>>>> at java.io.DataInputStream.readFully(DataInputStream.java:180)
>>>>>>> at java.io.DataInputStream.readLong(DataInputStream.java:399)
>>>>>>> at
>>>>> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:120)
>>>>>>> at
>>>>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2621)
>>>>>>>
>>>>>>> 2010-12-09 04:32:33,089 INFO org.apache.zookeeper.ClientCnxn: Unable to
>>>>> read additional data from server sessionid 0x12ca58c08ff0049, likely server
>>>>> has closed socket, closing sock
>>>>>>> et connection and attempting reconnect
>>>>>>> 2010-12-09 04:32:33,089 WARN
>>>>> org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler 5 on 60020
>>>>> took 75446ms appending an edit to hlog; editcount=50505
>>>>>>> 2010-12-09 04:32:33,090 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>>>>> Server handler 18 on 60020: exiting
>>>>>>> 2010-12-09 04:32:33,139 INFO org.apache.hadoop.hdfs.DFSClient: Error
>>>>> Recovery for block blk_2012253533858803341_250248 waiting for responder to
>>>>> exit.
>>>>>>> 2010-12-09 04:32:33,139 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>>>>> Server handler 8 on 60020: exiting
>>>>>>> 2010-12-09 04:32:33,139 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>>>> Server handler 1 on 60020: exiting
>>>>>>> 2010-12-09 04:32:33,139 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>>>>> Server handler 10 on 60020: exiting
>>>>>>> 2010-12-09 04:32:33,140 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>>>> Server handler 0 on 60020: exiting2010-12-09 04:32:33,140 INFO
>>>>> org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 5 on 60020:
>>>>> exiting
>>>>>>> 2010-12-09 04:32:33,139 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>>>> Server handler 8 on 60020: exiting2010-12-09 04:32:33,140 INFO
>>>>> org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 4 on 60020:
>>>>> exiting
>>>>>>> 2010-12-09 04:32:33,140 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>>>> Server handler 3 on 60020: exiting2010-12-09 04:32:33,142 INFO
>>>>> org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 7 on 60020:
>>>>> exiting
>>>>>>> 2010-12-09 04:32:33,142 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>>>> Server handler 9 on 60020: exiting2010-12-09 04:32:33,143 INFO
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
>>>>>>> 2010-12-09 04:32:33,143 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>>>> Server handler 2 on 60020: exiting
>>>>>>> 2010-12-09 04:32:33,143 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
>>>>> IPC Server Responder
>>>>>>> 2010-12-09 04:32:33,143 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>>>> Server handler 6 on 60020: exiting
>>>>>>> 2010-12-09 04:32:33,144 INFO org.mortbay.log: Stopped
>>>>> SelectChannelConnector@0.0.0.0:60030
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> NAMENODE ->
>>>>>>> 2010-12-09 04:32:08,408 WARN org.apache.hadoop.hdfs.StateChange: DIR*
>>>>> NameSystem.startFile: failed to create file
>>>>> /hbase/.logs/05.hadoop.xxxx,60020,1291862294517/10.100.159.13%3A60020.1291868481555
>>>>> for DFSClient_hb_m_10.194.194.79:60000_1291788452343 on client
>>>>> 10.194.194.79, because this file is already being created by NN_Recovery on
>>>>> 10.100.159.13
>>>>>>> 2010-12-09 04:32:08,408 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>>> handler 6 on 54310, call
>>>>> append(/hbase/.logs/05.hadoop.xxxx,60020,1291862294517/10.100.159.13%3A60020.1291868481555,
>>>>> DFSClient_hb_m_10.194.194.79:60000_1291788452343) from 10.194.194.79:59118:
>>>>> error: org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed
>>>>> to create file /hba
>>>>>>> se/.logs/05.hadoop.xxxx,60020,1291862294517/10.100.159.13%3A60020.1291868481555
>>>>> for DFSClient_hb_m_10.194.194.79:60000_1291788452343 on client
>>>>> 10.194.194.79, because this file is already being created by NN_Recovery on
>>>>> 10.100.159.13
>>>>>>> org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to
>>>>> create file /hbase/.logs/05.hadoop.xxxx,60020,1291862294517/10.100.159.13%3A60020.1291868481555
>>>>> for DFSClient_hb_m_10.194.194.79:60000_1291788452343 on client
>>>>> 10.194.194.79, because this file is already being created by NN_Recovery on
>>>>> 10.100.159.13
>>>>>>> at
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1194)
>>>>> at
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:1282)
>>>>>>> at
>>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.append(NameNode.java:541)
>>>>> at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
>>>>>>> at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:528) at
>>>>> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1319)
>>>>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1315)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>>> at javax.security.auth.Subject.doAs(Subject.java:396) at
>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
>>>>>>> at
>>>>> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1313)2010-12-09
>>>>> 04:32:09,411 WARN org.apache.hadoop.hdfs.StateChange: DIR*
>>>>> NameSystem.startFile: failed to create file
>>>>> /hbase/.logs/05.hadoop.xxxx,60020,1291862294517/10.100.159.13%3A6
>>>>>>> 0020.1291868481555 for DFSClient_hb_m_10.194.194.79:60000_1291788452343
>>>>> on client 10.194.194.79, because this file is already being created by
>>>>> NN_Recovery on 10.100.159.132010-12-09 04:32:09,411 INFO
>>>>> org.apache.hadoop.ipc.Server: IPC Server handler 9 on 54310, call
>>>>> append(/hbase/.logs/05.hadoop.xxxx,60020,1291862294517/10.100.159.13
>>>>> %3A60020.1291868
>>>>>>> 481555, DFSClient_hb_m_10.194.194.79:60000_1291788452343) from
>>>>> 10.194.194.79:59118: error:
>>>>> org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to
>>>>> create file /hbase/.logs/05.hadoop.xxxx,60020,1291862294517/10.100.159.13%3A60020.1291868481555
>>>>> for DFSClient_hb_m_10.194.194.79:60000_1291788452343 on client
>>>>> 10.194.194.79, because this file is
>>>>>>> already being created by NN_Recovery on
>>>>> 10.100.159.13org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException:
>>>>> failed to create file /hbase/.logs/05.hadoop.xxxx,60020,1291862294517/
>>>>> 10.100.159.13%3A60020.1291868481555 for DFSClie
>>>>>>> nt_hb_m_10.194.194.79:60000_1291788452343 on client 10.194.194.79,
>>>>> because this file is already being created by NN_Recovery on 10.100.159.13
>>>>> at
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1194)
>>>>>>> at
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:1282)
>>>>> at
>>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.append(NameNode.java:541)
>>>>>>> at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
>>>>> at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597) at
>>>>> org.apache.hadoop.ipc.RPC$Server.call(RPC.java:528)
>>>>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1319)
>>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1315)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> DATANODE
>>>>>>>
>>>>>>> 2010-12-09 04:32:09,534 INFO
>>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder
>>>>> blk_2012253533858803341_250248 1 : Thread is interrupted.
>>>>>>> 2010-12-09 04:32:09,534 ERROR
>>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
>>>>> 10.100.159.13:50010,
>>>>> storageID=DS-145997007-10.100.159.13-50010-1291262743182, infoPort=50075,
>>>>> ipcPort=50020):DataXceiver
>>>>>>> java.io.IOException: Interrupted receiveBlock
>>>>>>> at
>>>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:579)
>>>>>>> at
>>>>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:417)
>>>>>>> at
>>>>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:122)
>>>>>>> 2010-12-09 04:32:09,534 INFO
>>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 1 for block
>>>>> blk_2012253533858803341_250248 terminating
>>>>>>>
>>>>>>>
>>>>>>>
>>>>> ************************************************************************************
>>>>>>> 06.hadoop
>>>>>>>
>>>>> ************************************************************************************
>>>>>>> REGION SERVER ->
>>>>>>>
>>>>>>> 2010-12-09 06:17:47,215 DEBUG
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>>>> NotServingRegionException;
>>>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>>>> 2010-12-09 06:17:47,430 DEBUG
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>>>> NotServingRegionException;
>>>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>>>> 2010-12-09 06:17:47,549 DEBUG
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>>>> NotServingRegionException;
>>>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>>>> 2010-12-09 06:17:47,557 DEBUG
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>>>> NotServingRegionException;
>>>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>>>> 2010-12-09 06:17:47,563 DEBUG
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>>>> NotServingRegionException;
>>>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>>>> 2010-12-09 06:17:48,619 DEBUG
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>>>> NotServingRegionException;
>>>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>>>> 2010-12-09 06:17:48,619 DEBUG
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>>>> NotServingRegionException;
>>>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>>>> 2010-12-09 06:17:48,619 DEBUG
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>>>> NotServingRegionException;
>>>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>>>> 2010-12-09 06:17:48,619 DEBUG
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>>>> NotServingRegionException;
>>>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>>>> 2010-12-09 06:17:48,619 DEBUG
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>>>> NotServingRegionException;
>>>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>>>> 2010-12-09 06:17:49,703 DEBUG
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>>>> NotServingRegionException;
>>>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>>>> 2010-12-09 06:17:49,703 DEBUG
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>>>> NotServingRegionException;
>>>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>>>> 2010-12-09 06:17:49,703 DEBUG
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>>>> NotServingRegionException;
>>>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>>>> 2010-12-09 06:17:49,705 DEBUG
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>>>> NotServingRegionException;
>>>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>>>> 2010-12-09 06:17:49,729 DEBUG
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>>>> NotServingRegionException;
>>>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>>>>
>>>>>>> A bunch of these ^
>>>>>>>
>>>>>>> Then a little later::
>>>>>>> ....
>>>>>>>
>>>>>>> 2010-12-09 06:23:23,561 DEBUG
>>>>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction
>>>>> requested for region
>>>>> article,993143fca3613014990cc6975cacdd62f4147e03,12918133247
>>>>>>> 53.19413f33fc660536d9d4f315a1283440. because:
>>>>> regionserver60020.cacheFlusher
>>>>>>> 2010-12-09 06:23:27,058 INFO
>>>>> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs
>>>>> -- HDFS-200
>>>>>>> 2010-12-09 06:23:27,080 INFO
>>>>> org.apache.hadoop.hbase.regionserver.wal.HLog: Roll
>>>>> /hbase/.logs/06.hadoop.****,60020,1291864620851/10.99.97.106%3A60020.1291875047947,
>>>>> entries=53315,
>>>>>>> filesize=63781925. New hlog
>>>>> /hbase/.logs/06.hadoop.****,60020,1291864620851/10.99.97.106
>>>>> %3A60020.1291875807006
>>>>>>> 2010-12-09 06:23:27,080 DEBUG
>>>>> org.apache.hadoop.hbase.regionserver.wal.HLog: Found 1 hlogs to remove out
>>>>> of total 8; oldest outstanding sequenceid is 33526881 from region article
>>>>>>>
>>>>> ,1221b0013c61fd7446cb822f698d0bcafa487342,1291872610788.bb2aa6ae8c694a096a00e7bb91fc46c3.
>>>>>>> 2010-12-09 06:23:27,080 INFO
>>>>> org.apache.hadoop.hbase.regionserver.wal.HLog: moving old hlog file
>>>>> /hbase/.logs/06.hadoop.****,60020,1291864620851/10.99.97.106
>>>>> %3A60020.1291872064424
>>>>>>> whose highest sequenceid is 33506639 to /hbase/.oldlogs/10.99.97.106
>>>>> %3A60020.1291872064424
>>>>>>> 2010-12-09 06:25:39,162 WARN
>>>>> org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler 12 on
>>>>> 60020 took 119034ms appending an edit to hlog; editcount=3683
>>>>>>> 2010-12-09 06:25:39,162 WARN org.apache.hadoop.hdfs.DFSClient:
>>>>> DataStreamer Exception: java.io.IOException: Broken pipe
>>>>>>> at sun.nio.ch.FileDispatcher.write0(Native Method)
>>>>>>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
>>>>>>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100)
>>>>>>> at sun.nio.ch.IOUtil.write(IOUtil.java:71)
>>>>>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
>>>>>>> at
>>>>> org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
>>>>>>> at
>>>>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
>>>>>>> at
>>>>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
>>>>>>> at
>>>>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
>>>>>>> at
>>>>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
>>>>>>> at java.io.DataOutputStream.write(DataOutputStream.java:90)
>>>>>>> at
>>>>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2509)
>>>>>>> 2010-12-09 06:25:39,163 WARN org.apache.hadoop.hdfs.DFSClient:
>>>>> DFSOutputStream ResponseProcessor exception for block
>>>>> blk_1141147913164105086_258331java.io.EOFException
>>>>>>> at java.io.DataInputStream.readFully(DataInputStream.java:180)
>>>>>>> at java.io.DataInputStream.readLong(DataInputStream.java:399)
>>>>>>> at
>>>>> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:120)
>>>>>>> at
>>>>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2621)
>>>>>>>
>>>>>>> 2010-12-09 06:25:39,163 INFO org.apache.hadoop.hdfs.DFSClient: Error
>>>>> Recovery for block blk_1141147913164105086_258331 waiting for responder to
>>>>> exit.
>>>>>>> 2010-12-09 06:25:39,164 WARN org.apache.hadoop.hdfs.DFSClient:
>>>>> DFSOutputStream ResponseProcessor exception for block
>>>>> blk_-7076574401277381592_258324java.io.EOFException
>>>>>>> at java.io.DataInputStream.readFully(DataInputStream.java:180)
>>>>>>> at java.io.DataInputStream.readLong(DataInputStream.java:399)
>>>>>>> at
>>>>> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:120)
>>>>>>> at
>>>>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2621)
>>>>>>>
>>>>>>> 2010-12-09 06:25:39,164 WARN org.apache.hadoop.hdfs.DFSClient: Error
>>>>> Recovery for block blk_-7076574401277381592_258324 bad datanode[0]
>>>>> 10.99.97.106:50010
>>>>>>> 2010-12-09 06:25:39,164 INFO org.apache.zookeeper.ClientCnxn: Unable to
>>>>> read additional data from server sessionid 0x12ca58c08ff004b, likely server
>>>>> has closed socket, closing socket connection and attempting reconnect
>>>>>>> 2010-12-09 06:25:39,164 WARN org.apache.hadoop.hdfs.DFSClient: Error
>>>>> Recovery for block blk_-7076574401277381592_258324 in pipeline
>>>>> 10.99.97.106:50010, 10.100.154.103:50010: bad datanode 10.99.97.106:50010
>>>>>>> 2010-12-09 06:25:39,165 WARN org.apache.hadoop.hdfs.DFSClient: Error
>>>>> Recovery for block blk_1141147913164105086_258331 bad datanode[0]
>>>>> 10.99.97.106:50010
>>>>>>> 2010-12-09 06:25:39,167 INFO org.apache.zookeeper.ClientCnxn: Unable to
>>>>> read additional data from server sessionid 0x12ca58c08ff004c, likely server
>>>>> has closed socket, closing socket connection and attempting reconnect
>>>>>>> 2010-12-09 06:25:39,169 INFO org.apache.hadoop.ipc.Client: Retrying
>>>>> connect to server: /10.100.154.103:50020. Already tried 0 time(s).
>>>>>>> 2010-12-09 06:25:39,169 WARN org.apache.hadoop.hdfs.DFSClient: Error
>>>>> Recovery for block blk_1141147913164105086_258331 in pipeline
>>>>> 10.99.97.106:50010, 10.100.159.13:50010: bad datanode 10.99.97.106:50010
>>>>>>> 2010-12-09 06:25:39,209 INFO
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGIONSERVER_STOP
>>>>>>> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
>>>>> server on 60020
>>>>>>> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>>>>> Server handler 3 on 60020: exiting
>>>>>>> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>>>>> Server handler 9 on 60020: exiting
>>>>>>> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>>>>> Server handler 24 on 60020: exiting
>>>>>>> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>>>> Server handler 1 on 60020: exiting
>>>>>>> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>>>>> Server handler 15 on 60020: exiting
>>>>>>> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>>>> Server handler 0 on 60020: exiting
>>>>>>> 2010-12-09 06:25:39,214 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>>>> Server handler 3 on 60020: exiting
>>>>>>> 2010-12-09 06:25:39,214 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
>>>>> IPC Server listener on 60020
>>>>>>> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>>>> Server handler 2 on 60020: exiting
>>>>>>> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>>>> Server handler 7 on 60020: exiting
>>>>>>> 2010-12-09 06:25:39,213 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>>>>> Server handler 21 on 60020: exiting
>>>>>>> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>>>> Server handler 4 on 60020: exiting
>>>>>>> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
>>>>> IPC Server Responder
>>>>>>> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>>>> Server handler 5 on 60020: exiting
>>>>>>> 2010-12-09 06:25:39,216 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>>>> Server handler 8 on 60020: exiting
>>>>>>> 2010-12-09 06:25:39,216 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>>>> Server handler 9 on 60020: exiting
>>>>>>> 2010-12-09 06:25:39,216 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>>>> Server handler 6 on 60020: exiting
>>>>>>> 2010-12-09 06:25:39,216 INFO
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
>>>>>>> 2010-12-09 06:25:39,223 WARN org.apache.hadoop.hdfs.DFSClient: Failed
>>>>> recovery attempt #0 from primary datanode 10.100.159.13:50010
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>
>>
>>
Re: Crashing Region Servers
Posted by Stack <st...@duboce.net>.
On Mon, Dec 20, 2010 at 11:13 AM, Lance Riedel <la...@gmail.com> wrote:
> Still having issues, so taking a step back here for some scale questions:
> We are doing:
> - inserts at the rate of up to 1k/sec with each entry just under 1k (roughly)
> - roughly 60-100 megs/min - 5-10 gigs/day/ 100 million records per day
> - rolling 30 days of data (so will kill off a days worth of data per day)
How you do the rolling?
> ~ 300gigs of raw data per month (maybe some more, but not orders more)
>
> What size of machines / number do you think we need?
>
> Here is what we have been using with lots of issues for just 1 weeks worth of data:
> -4 compute.xlarge = each with 4 cpus, 16 gigs ram
> - Each regionserver is getting 6 gigs allocated
>
Multiply all hardware requirements by N (where N is 2-4?) if running
on EC2. It looks like you are trying to do test loading so you'll be
best situated regards how much HW is needed.
> Problem we are having:
> Region server crashes
>
> Summary of problem:
> We get a region server crashing after about 12 hours of runtime. The only load really is the inserts as described above. All the logs (master, region, datanode) are below during the hangup.
>
FYI, its tough reading logs pasted into email. Stick them on pastebin
or up in an issue.
> Where it gets weird:
> The only way to startup the region server is to turn off concurrent sweep GC, otherwise it just dies on startup (see 2nd set of logs at bottom)
>
Looking at 2nd set of logs I see this:
> 2010-12-20 17:26:37,543 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 56739ms for sessionid 0x12ce6e6154d00b3, closing socket connection and attempting reconnect
That the zk client running in regionserver saying that it hasn't
talked to the zk ensemble in near 60 seconds. Given that it pings
every two or three seconds, you probably had a big GC pause in there
-- were you swapping at the time? -- and since the zk timeout is 60
seconds, the server lost its lease and exited.
Can you figure whats going on here on the machine when it goes down?
Its just starting up doing a few compactions. Doesn't look
particularly unusual.
For the first crash, it looks like you are using default JVM and then
as you note, there is the 14minute outage... then it says:
> 2010-12-20 12:12:55,175 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: Aborting region server serverName=01.hbase.jive,60020,1292805088631, load=(requests=486, regions=317, usedHeap=2185, maxHeap=5333): Unhandled exception
So, are your machines swapping?
St.Ack
>
> Below are 2 sets of logs, (Dies after 12 hours, and Dies on startup) - only difference is GC params
>
> **********************************************************
> DIES AFTER 12 HOURS LOGS:
> Non Concurrent Sweep Death Logs
> Runs for up to 12 hours, then here are the various logs when it dies
> export HBASE_OPTS="$HBASE_OPTS -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:$HBASE_HOME/logs/gc-hbase.log"
> **********************************************************
>
>
> ///////////REGION SERVER
> 2010-12-20 11:58:32,252 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Roll /hbase/.logs/01.hbase.jive,60020,1292805088631/10.101.25.203%3A6002
> 0.1292845076146, entries=53328, filesize=63945980. New hlog /hbase/.logs/01.hbase.jive,60020,1292805088631/10.101.25.203%3A60020.1292846312039
> 2010-12-20 11:58:32,252 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: Found 2 hlogs to remove out of total 34; oldest outstanding sequenceid
> is 57439916 from region article,6addb2ace7f12175eb8ec92d16faa26f7592c64e,1292731695023.ec7bc70e76c6f4fc3b0779c412680375.
> 2010-12-20 11:58:32,252 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: moving old hlog file /hbase/.logs/01.hbase.jive,60020,1292805088631/10.1
> 01.25.203%3A60020.1292826985598 whose highest sequenceid is 57337628 to /hbase/.oldlogs/10.101.25.203%3A60020.1292826985598
> 2010-12-20 11:58:32,253 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: moving old hlog file /hbase/.logs/01.hbase.jive,60020,1292805088631/10.1
> 01.25.203%3A60020.1292827095006 whose highest sequenceid is 57390617 to /hbase/.oldlogs/10.101.25.203%3A60020.1292827095006
> 2010-12-20 11:59:24,061 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler 3 on 60020 took 5172ms appending an edit to hlog; edi
> tcount=29066
> ****NOTE 14 minute gap here!!!! *********
> 2010-12-20 12:12:55,016 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=32.86 MB, free=1.01 GB, max=1.04 GB, blocks=1836, accesses=1250468, hits=775218, hitRatio=61.99%%, evictions=0, evicted=0, evictedPerRun=NaN
> 2010-12-20 12:12:55,016 WARN org.apache.hadoop.hdfs.DFSClient: DFSOutputStream ResponseProcessor exception for block blk_-6892804755545789592_78455java.net.SocketTimeoutException: 66000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.101.25.203:57216 remote=/10.101.25.203:50010]
> 2010-12-20 12:12:55,017 INFO org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_-6892804755545789592_78455 waiting for responder to exit.
> 2010-12-20 12:12:55,044 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 784310ms for sessionid 0x12ce6e6154d00a3, closing socket connection and attempting reconnect
> 2010-12-20 12:12:55,062 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_-6892804755545789592_78455 bad datanode[0] 10.101.25.203:50010
> 2010-12-20 12:12:55,062 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_-6892804755545789592_78455 in pipeline 10.101.25.203:50010, 10.124.189.106:50010: bad datanode 10.101.25.203:50010
> 2010-12-20 12:12:55,089 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: HDFS pipeline error detected. Found 1 replicas but expecting 2 replicas. Requesting close of hlog.
> 2010-12-20 12:12:55,148 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs -- HDFS-200
> 2010-12-20 12:12:55,155 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event, state: Disconnected, type: None, path: null
> 2010-12-20 12:12:55,155 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event, state: Disconnected, type: None, path: null
> 2010-12-20 12:12:55,175 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: Aborting region server serverName=01.hbase.jive,60020,1292805088631, load=(requests=486, regions=317, usedHeap=2185, maxHeap=5333): Unhandled exception
> 2010-12-20 12:12:55,176 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: request=0.0, regions=317, stores=318, storefiles=679, storefileIndexSize=88, memstoreSize=2096, compactionQueueSize=0, usedHeap=2192, maxHeap=5333, blockCacheSize=34460320, blockCacheFree=1084029536, blockCacheCount=1836, blockCacheHitRatio=61
> 2010-12-20 12:12:55,176 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60020
>
> ///////////////////MASTER
> 2010-12-20 12:12:55,022 DEBUG org.apache.hadoop.hbase.master.ServerManager: Server REPORT rejected; currently processing 01.hbase.jive,60020,1292805088631 as dead server
> 2010-12-20 12:12:55,761 INFO org.apache.hadoop.hbase.util.FSUtils: Finished lease recover attempt for hdfs://00.hbase.jive:54310/hbase/.logs/01.hbase.jive,60020,1292805088631/10.101.25.203%3A60020.1292827812330
> 2010-12-20 12:12:56,824 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: Pushed=52893 entries from hdfs://00.hbase.jive:54310/hbase/.logs/01.hbase.jive,60020,1292805088631/10.101.25.203%3A60020.1292827812330
> 2010-12-20 12:12:56,825 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: Splitting hlog 3 of 33: hdfs://00.hbase.jive:54310/hbase/.logs/01.hbase.jive,60020,1292805088631/10.101.25.203%3A60020.1292828346328, length=63757023
> 2010-12-20 12:12:56,825 INFO org.apache.hadoop.hbase.util.FSUtils: Recovering filehdfs://00.hbase.jive:54310/hbase/.logs/01.hbase.jive,60020,1292805088631/10.101.25.203%3A60020.1292828346328
> 2010-12-20 12:12:57,279 INFO org.apache.hadoop.hbase.util.FSUtils: Finished lease recover attempt for hdfs://00.hbase.jive:54310/hbase/.logs/01.hbase.jive,60020,1292805088631/10.101.25.203%3A60020.1292828346328
> 2010-12-20 12:12:58,735 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: Pushed=52918 entries from hdfs://00.hbase.jive:54310/hbase/.logs/01.hbase.jive,60020,1292805088631/10.101.25.203%3A60020.1292828346328
>
>
> ////////////////DATANODE
>
> 2010-12-20 12:12:55,067 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in receiveBlock for block blk_-6892804755545789592_78455 java.io.EOFException: while trying to read 4377 bytes
> 2010-12-20 12:12:55,068 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder blk_-6892804755545789592_78455 1 : Thread is interrupted.
> 2010-12-20 12:12:55,068 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 1 for block blk_-6892804755545789592_78455 terminating
> 2010-12-20 12:12:55,068 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock blk_-6892804755545789592_78455 received exception java.io.EOFException: while trying to read 4377 bytes
> 2010-12-20 12:12:55,069 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(10.101.25.203:50010, storageID=DS-1813182679-10.101.25.203-50010-1292355638641, infoPort=50075, ipcPort=50020):DataXceiver
> 2010-12-20 12:12:55,414 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_-702706422692129802_78497 src: /10.101.25.249:55244 dest: /10.101.25.203:50010
> 2010-12-20 12:12:55,462 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_7987291438907198920_78499 src: /10.101.25.203:33387 dest: /10.101.25.203:50010
> 2010-12-20 12:12:55,464 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_-2741441115648159702_78500 src: /10.101.25.203:33388 dest: /10.101.25.203:50010
> 2010-12-20 12:12:55,486 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected: wildcard ongoingCreates exists for block blk_-702706422692129802_78496
> 2010-12-20 12:12:55,486 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Reopen Block for append blk_-702706422692129802_78496
> 2010-12-20 12:12:55,487 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in receiveBlock for block blk_-702706422692129802_78496 java.io.EOFException: while trying to read 65557 bytes
> 2010-12-20 12:12:55,487 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block blk_-702706422692129802_78496 Interrupted.
> 2010-12-20 12:12:55,487 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block blk_-702706422692129802_78496 terminating
> 2010-12-20 12:12:55,487 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock blk_-702706422692129802_78496 received exception java.io.EOFException: while trying to read 65557 bytes
> 2010-12-20 12:12:55,487 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(10.101.25.203:50010, storageID=DS-1813182679-10.101.25.203-50010-1292355638641, infoPort=50075, ipcPort=50020):DataXceiver
> 2010-12-20 12:12:55,489 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: oldblock=blk_287777955370148653_76327(length=63761157), newblock=blk_287777955370148653_78501(length=63761157), datanode=10.101.25.203:50010
> 2010-12-20 12:12:55,501 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in receiveBlock for block blk_-702706422692129802_78497 java.io.EOFException: while trying to read 65536 bytes
> 2010-12-20 12:12:55,501 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Block blk_-702706422692129802_78497 unfinalized and removed.
> 2010-12-20 12:12:55,501 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock blk_-702706422692129802_78497 received exception java.io.EOFException: while trying to read 65536 bytes
> 2010-12-20 12:12:55,501 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(10.101.25.203:50010, storageID=DS-1813182679-10.101.25.203-50010-1292355638641, infoPort=50075, ipcPort=50020):DataXceiver
> 2010-12-20 12:12:55,610 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.203:33388, dest: /10.101.25.203:50010, bytes: 5736187, op: HDFS_WRITE, cliID: DFSClient_-1650596123, offset: 0, srvID: DS-1813182679-10.101.25.203-50010-1292355638641, blockid: blk_-2741441115648159702_78500, duration: 119469848
>
> ////////////////////GC
> 2010-12-20T11:48:25.248+0000: 40617.905: [GC [PSYoungGen: 312017K->31408K(354880K)] 2396181K->2115572K(2563648K), 0.0727710 secs] [Times: user=0.18 sys=0.00, real=0.07 secs]
> 2010-12-20T11:57:47.651+0000: 41180.308: [GC [PSYoungGen: 312048K->33351K(307712K)] 2396212K->2117515K(2516480K), 0.0643350 secs] [Times: user=0.16 sys=0.01, real=0.06 secs]
> 2010-12-20T11:57:54.525+0000: 41187.182: [GC [PSYoungGen: 307655K->37385K(341504K)] 2391819K->2121549K(2550272K), 0.0710110 secs] [Times: user=0.17 sys=0.00, real=0.07 secs]
> 2010-12-20T11:58:00.157+0000: 41192.814: [GC [PSYoungGen: 305545K->26719K(289024K)] 2389709K->2127955K(2497792K), 0.0779900 secs] [Times: user=0.18 sys=0.01, real=0.08 secs]
> 2010-12-20T11:58:04.530+0000: 41197.188: [GC [PSYoungGen: 288991K->11195K(339968K)] 2390227K->2127423K(2548736K), 0.0531710 secs] [Times: user=0.12 sys=0.00, real=0.06 secs]
> 2010-12-20T11:58:15.908+0000: 41208.565: [GC [PSYoungGen: 275835K->10768K(269696K)] 2392063K->2131139K(2478464K), 0.0256330 secs] [Times: user=0.06 sys=0.00, real=0.02 secs]
> 2010-12-20T11:58:20.325+0000: 41212.982: [GC [PSYoungGen: 269648K->14982K(311744K)] 2390019K->2142213K(2520512K), 0.0394690 secs] [Times: user=0.07 sys=0.00, real=0.04 secs]
> 2010-12-20T11:58:29.751+0000: 41222.409: [GC [PSYoungGen: 268258K->7168K(255104K)] 2395489K->2141035K(2463872K), 0.0367260 secs] [Times: user=0.09 sys=0.00, real=0.03 secs]
> 2010-12-20T11:58:34.476+0000: 41227.133: [GC [PSYoungGen: 255067K->7424K(298624K)] 2388934K->2148539K(2507392K), 0.0379800 secs] [Times: user=0.12 sys=0.00, real=0.03 secs]
> 2010-12-20T11:58:38.866+0000: 41231.523: [GC [PSYoungGen: 250094K->7104K(244800K)] 2391210K->2155495K(2453568K), 0.0414990 secs] [Times: user=0.09 sys=0.00, real=0.04 secs]
> 2010-12-20T11:58:43.223+0000: 41235.880: [GC [PSYoungGen: 244694K->7072K(301120K)] 2393086K->2162599K(2509888K), 0.0404940 secs] [Times: user=0.08 sy
> s=0.00, real=0.04 secs]
> 2010-12-20T11:58:47.589+0000: 41240.246: [GC [PSYoungGen: 247054K->7264K(298752K)] 2402582K->2169607K(2507520K), 0.0322390 secs] [Times: user=0.08 sy
> s=0.00, real=0.04 secs]
> 2010-12-20T11:58:52.521+0000: 41245.178: [GC [PSYoungGen: 247208K->7200K(326016K)] 2409552K->2176615K(2534784K), 0.0366060 secs] [Times: user=0.07 sy
> s=0.00, real=0.03 secs]
> 2010-12-20T11:58:57.931+0000: 41250.588: [GC [PSYoungGen: 281759K->8128K(328448K)] 2451174K->2184519K(2537216K), 0.0359190 secs] [Times: user=0.09 sy
> s=0.00, real=0.03 secs]
> 2010-12-20T11:59:09.681+0000: 41262.338: [GC [PSYoungGen: 282649K->8000K(276480K)] 2459041K->2192471K(2485248K), 0.0432830 secs] [Times: user=0.11 sys=0.01, real=0.05 secs]
> 2010-12-20T11:59:14.136+0000: 41266.793: [GC [PSYoungGen: 276405K->7872K(324416K)] 2460876K->2200295K(2533184K), 0.0565900 secs] [Times: user=0.11 sys=0.01, real=0.06 secs]
> 2010-12-20T11:59:18.889+0000: 41271.546: [GC [PSYoungGen: 283525K->8288K(356544K)] 2475949K->2208559K(2565312K), 0.0390340 secs] [Times: user=0.10 sys=0.01, real=0.04 secs]
> 2010-12-20T11:59:18.928+0000: 41271.585: [Full GC [PSYoungGen: 8288K->0K(356544K)] [PSOldGen: 2200271K->1991701K(2358656K)] 2208559K->1991701K(2715200K) [PSPermGen: 18540K->18540K(21248K)], 5.1322070 secs] [Times: user=5.07 sys=0.00, real=5.14 secs]
> 2010-12-20T11:59:29.253+0000: 41281.910: [GC [PSYoungGen: 310221K->9152K(312256K)] 2301922K->2000853K(2670912K), 0.0268860 secs] [Times: user=0.06 sys=0.00, real=0.04 secs]
> 2010-12-20T11:59:34.610+0000: 41287.267: [GC [PSYoungGen: 312172K->8928K(325888K)] 2303873K->2009821K(2684544K), 0.0403310 secs] [Times: user=0.09 sys=0.00, real=0.05 secs]
> 2010-12-20T11:59:41.074+0000: 41293.731: [GC [PSYoungGen: 304982K->8544K(297920K)] 2305875K->2018269K(2656576K), 0.0468460 secs] [Times: user=0.07 sys=0.00, real=0.05 secs]
> 2010-12-20T11:59:47.708+0000: 41300.365: [GC [PSYoungGen: 297888K->8448K(309504K)] 2307613K->2026869K(2668160K), 0.0409660 secs] [Times: user=0.09 sys=0.00, real=0.04 secs]
> Heap
> PSYoungGen total 309504K, used 243942K [0x00007f5556d50000, 0x00007f556c610000, 0x00007f55d3d50000)
> eden space 282880K, 83% used [0x00007f5556d50000,0x00007f55653498f0,0x00007f5568190000)
> from space 26624K, 31% used [0x00007f556ac10000,0x00007f556b4500c0,0x00007f556c610000)
> to space 35072K, 0% used [0x00007f5568190000,0x00007f5568190000,0x00007f556a3d0000)
> PSOldGen total 2358656K, used 2018421K [0x00007f545cd50000, 0x00007f54eccb0000, 0x00007f5556d50000)
> object space 2358656K, 85% used [0x00007f545cd50000,0x00007f54d806d480,0x00007f54eccb0000)
> PSPermGen total 21248K, used 18595K [0x00007f5457950000, 0x00007f5458e10000, 0x00007f545cd50000)
> object space 21248K, 87% used [0x00007f5457950000,0x00007f5458b78c50,0x00007f5458e10000)
>
>
> **********************************************************
> DIES ON STARTUP LOGS:
> Non Concurrent Sweep
> export HBASE_OPTS="$HBASE_OPTS -verbose:gc -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:$HBASE_HOME/logs/gc-hbase.log"
> **********************************************************
>
> ///////////REGION SERVER
>
> 2010-12-20 17:25:49,020 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region article,26b5ee07c80cf31d88719b1a510726591f2b01da,12927082
> 91003.aae953527ae2ce697fda39699e29f70e. in 0sec
> 2010-12-20 17:25:49,020 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on region article,c050050bf22e8f276a8a6bd3c900b34d6600d6d0,129273169
> 0903.0460d68f5f4fd423750dd6bf67937b7e.2010-12-20 17:25:49,022 DEBUG org.apache.hadoop.hbase.regionserver.Store: Compaction size of data: 186.0m; Skipped 2 file(s), size: 1894568042010-12-20 17:25:49,023 INFO org.apache.hadoop.hbase.regionserver.Store: Started compaction of 2 file(s) in data of article,c050050bf22e8f276a8a6bd3c900b34d6600d6d
> 0,1292731690903.0460d68f5f4fd423750dd6bf67937b7e. into hdfs://00.hbase.jive:54310/hbase/article/0460d68f5f4fd423750dd6bf67937b7e/.tmp, sequenceid=63434988
> 2010-12-20 17:25:49,324 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Applied 0, skipped 10953, firstSequenceidInLog=52141208, maxSequenceidInLog=63429846
> 2010-12-20 17:25:49,327 INFO org.apache.hadoop.hbase.regionserver.HRegion: Onlined article,150bfc2bd7a0effc9f3dfa22312878bf916d68df,1292597687450.f213d3923dca3a7e3
> d020cf7dbf51918.; next sequenceid=63429847
> 2010-12-20 17:25:49,328 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 2 file(s) in data of article,c050050bf22e8f276a8a6bd3c900b34d6600d
> 6d0,1292731690903.0460d68f5f4fd423750dd6bf67937b7e.; new storefile is hdfs://00.hbase.jive:54310/hbase/article/0460d68f5f4fd423750dd6bf67937b7e/data/88724363503101
> 19241; store size is 186.0m
> 2010-12-20 17:25:49,329 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region article,c050050bf22e8f276a8a6bd3c900b34d6600d6d0,12927316
> 90903.0460d68f5f4fd423750dd6bf67937b7e. in 0sec
> 2010-12-20 17:25:49,329 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on region article,933fc87674f7132b5d91ce6e9c60bc68981a56a7,129270464
> 1421.2551c0c46387d1479d25fb359ef4c5c9.
> 2010-12-20 17:25:49,336 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: article,0643e00cb1b172034b2627567d00d26913a25f97,12927089
> 81266.5e0ac5bd27f34da31ded307e2ea932cf.2010-12-20 17:25:49,337 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Creating region article,0643e00cb1b172034b2627567d00d26913a25f97,1292708981266.5e0ac5bd
> 27f34da31ded307e2ea932cf.
> 2010-12-20 17:25:49,338 DEBUG org.apache.hadoop.hbase.regionserver.Store: Compaction size of data: 186.5m; Skipped 2 file(s), size: 190928927
> 2010-12-20 17:25:49,338 INFO org.apache.hadoop.hbase.regionserver.Store: Started compaction of 2 file(s) in data of article,933fc87674f7132b5d91ce6e9c60bc68981a56a
> 7,1292704641421.2551c0c46387d1479d25fb359ef4c5c9. into hdfs://00.hbase.jive:54310/hbase/article/2551c0c46387d1479d25fb359ef4c5c9/.tmp, sequenceid=63434974
> 2010-12-20 17:25:49,362 DEBUG org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://00.hbase.jive:54310/hbase/article/5e0ac5bd27f34da31ded307e2ea932cf/data/421
> 7589706037747905, isReference=false, isBulkLoadResult=false, seqid=53910828, majorCompaction=true
> 2010-12-20 17:25:49,372 DEBUG org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://00.hbase.jive:54310/hbase/article/5e0ac5bd27f34da31ded307e2ea932cf/data/540
> 8931691681201713, isReference=false, isBulkLoadResult=false, seqid=63429839, majorCompaction=false
> 2010-12-20 17:25:49,379 DEBUG org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://00.hbase.jive:54310/hbase/article/5e0ac5bd27f34da31ded307e2ea932cf/data/779
> 2081177431743589, isReference=false, isBulkLoadResult=false, seqid=62768007, majorCompaction=false
> 2010-12-20 17:25:49,402 DEBUG org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://00.hbase.jive:54310/hbase/article/5e0ac5bd27f34da31ded307e2ea932cf/data/793
> 5329939530860970, isReference=false, isBulkLoadResult=false, seqid=62300392, majorCompaction=false
> 2010-12-20 17:25:49,427 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://00.hbase.jive:54310/hbase/article/5e0ac5bd27f34da31ded307e2e
> a932cf/recovered.edits/0000000000052141022; minSequenceid=63429839
> 2010-12-20 17:25:49,609 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region article,933fc87674f7132b5d91ce6e9c60bc68981a56a7,1292704641421.2551c0c46387d1479d25fb359ef4c5c9. in 0sec
> 2010-12-20 17:25:49,609 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on region article,fc36770defdbeba6e2841181a7da6f7bbae7f64f,1292701971082.399bf454e0c2a73adc038810e8fe4017.
> 2010-12-20 17:25:49,610 DEBUG org.apache.hadoop.hbase.regionserver.Store: Compaction size of data: 186.6m; Skipped 2 file(s), size: 192468499
> 2010-12-20 17:25:49,610 INFO org.apache.hadoop.hbase.regionserver.Store: Started compaction of 2 file(s) in data of article,fc36770defdbeba6e2841181a7da6f7bbae7f64f,1292701971082.399bf454e0c2a73adc038810e8fe4017. into hdfs://00.hbase.jive:54310/hbase/article/399bf454e0c2a73adc038810e8fe4017/.tmp, sequenceid=63427381
> 2010-12-20 17:26:37,543 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 56739ms for sessionid 0x12ce6e6154d00b3, closing socket connection and attempting reconnect
> 2010-12-20 17:26:37,632 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Applied 0, skipped 5388, firstSequenceidInLog=52141022, maxSequenceidInLog=63429839
> 2010-12-20 17:26:37,644 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: Aborting region server serverName=02.hbase.jive,60020,1292865913422, load=(requests=44, regions=142, usedHeap=97, maxHeap=5991): Unhandled exception
> org.apache.hadoop.hbase.YouAreDeadException: org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing 02.hbase.jive,60020,1292865913422 as dead server
> at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:217)
> at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:271)
> at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:744)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:561)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
> at org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
> at org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.doRun(HRegionServer.java:635)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.access$000(HRegionServer.java:126)
> at org.apache.hadoop.hbase.regionserver.HRegionServer$1.run(HRegionServer.java:518)
> at org.apache.hadoop.hbase.regionserver.HRegionServer$1.run(HRegionServer.java:516)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:337)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1046)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:516)
> at java.lang.Thread.run(Thread.java:662)
> 2010-12-20 17:26:37,645 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: request=0.0, regions=142, stores=142, storefiles=462, storefileIndexSize=43, memstoreSize=4, compactionQueueSize=36, usedHeap=105, maxHeap=5991, blockCacheSize=10308888, blockCacheFree=1246239080, blockCacheCount=0, blockCacheHitRatio=0
> 2010-12-20 17:26:37,645 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60020
> 2010-12-20 17:26:37,645 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 0 on 60020: exiting
> 2010-12-20 17:26:37,645 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 1 on 60020: exiting
> 2010-12-20 17:26:37,645 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 2 on 60020: exiting
>
> ////////////////DATANODE
>
> 2010-12-20 17:25:48,970 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 1 for block blk_7313855026629458364_85588 terminating
> 2010-12-20 17:25:49,012 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:50010, dest: /10.101.25.249:53108, bytes: 232, op: HDFS_READ, cliID: DFSClient_hb_rs_02.hbase.jive,60020,1292865913422_1292865913443, offset: 5637120, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_7313855026629458364_85588, duration: 168436
> 2010-12-20 17:25:49,013 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:50010, dest: /10.101.25.249:53109, bytes: 7456, op: HDFS_READ, cliID: DFSClient_hb_rs_02.hbase.jive,60020,1292865913422_1292865913443, offset: 5629952, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_7313855026629458364_85588, duration: 178950
> 2010-12-20 17:25:49,061 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_5943185653696604572_85589 src: /10.101.25.249:53112 dest: /10.101.25.249:50010
> 2010-12-20 17:25:49,103 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:50010, dest: /10.101.25.249:53111, bytes: 349508, op: HDFS_READ, cliID: DFSClient_hb_rs_02.hbase.jive,60020,1292865913422_1292865913443, offset: 0, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_7780046853952422303_85486, duration: 73022705
> 2010-12-20 17:25:49,307 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:53112, dest: /10.101.25.249:50010, bytes: 5564634, op: HDFS_WRITE, cliID: DFSClient_hb_rs_02.hbase.jive,60020,1292865913422_1292865913443, offset: 0, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_5943185653696604572_85589, duration: 242139544
> 2010-12-20 17:25:49,308 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 1 for block blk_5943185653696604572_85589 terminating
> 2010-12-20 17:25:49,324 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:50010, dest: /10.101.25.249:53114, bytes: 222, op: HDFS_READ, cliID: DFSClient_hb_rs_02.hbase.jive,60020,1292865913422_1292865913443, offset: 5564416, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_5943185653696604572_85589, duration: 144173
> 2010-12-20 17:25:49,325 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:50010, dest: /10.101.25.249:53115, bytes: 7446, op: HDFS_READ, cliID: DFSClient_hb_rs_02.hbase.jive,60020,1292865913422_1292865913443, offset: 5557248, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_5943185653696604572_85589, duration: 157056
> 2010-12-20 17:25:49,355 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_-5312992283942058338_85590 src: /10.101.25.249:53118 dest: /10.101.25.249:50010
> 2010-12-20 17:25:49,358 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:50010, dest: /10.101.25.249:53120, bytes: 199, op: HDFS_READ, cliID: DFSClient_hb_rs_02.hbase.jive,60020,1292865913422_1292865913443, offset: 25063424, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_-2003525593082351171_69708, duration: 138638
> 2010-12-20 17:25:49,359 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:50010, dest: /10.101.25.249:53121, bytes: 190603, op: HDFS_READ, cliID: DFSClient_hb_rs_02.hbase.jive,60020,1292865913422_1292865913443, offset: 24874496, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_-2003525593082351171_69708, duration: 735353
> 2010-12-20 17:25:49,411 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:50010, dest: /10.101.25.249:53117, bytes: 3070642, op: HDFS_READ, cliID: DFSClient_hb_rs_02.hbase.jive,60020,1292865913422_1292865913443, offset: 0, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_8585904126405165382_85484, duration: 68063489
> 2010-12-20 17:25:49,597 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:53118, dest: /10.101.25.249:50010, bytes: 4631054, op: HDFS_WRITE, cliID: DFSClient_hb_rs_02.hbase.jive,60020,1292865913422_1292865913443, offset: 0, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_-5312992283942058338_85590, duration: 234095142
> 2010-12-20 17:25:49,597 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 1 for block blk_-5312992283942058338_85590 terminating
> 2010-12-20 17:25:49,604 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:50010, dest: /10.101.25.249:53129, bytes: 534, op: HDFS_READ, cliID: DFSClient_hb_rs_02.hbase.jive,60020,1292865913422_1292865913443, offset: 4630528, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_-5312992283942058338_85590, duration: 186485
> 2010-12-20 17:25:49,605 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:50010, dest: /10.101.25.249:53130, bytes: 6210, op: HDFS_READ, cliID: DFSClient_hb_rs_02.hbase.jive,60020,1292865913422_1292865913443, offset: 4624896, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_-5312992283942058338_85590, duration: 155935
> 2010-12-20 17:25:49,672 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_-7865815939426325649_85591 src: /10.101.25.249:53133 dest: /10.101.25.249:50010
> 2010-12-20 17:26:22,770 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:50010, dest: /10.112.79.14:41575, bytes: 166, op: HDFS_READ, cliID: DFSClient_-1843172643, offset: 39720448, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_-2090804821592279293_75264, duration: 12128121
> 2010-12-20 17:26:22,810 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:50010, dest: /10.112.79.14:41576, bytes: 208114, op: HDFS_READ, cliID: DFSClient_-1843172643, offset: 39514112, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_-2090804821592279293_75264, duration: 38720643
> 2010-12-20 17:26:23,107 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:50010, dest: /10.112.79.14:41581, bytes: 312, op: HDFS_READ, cliID: DFSClient_-1843172643, offset: 16006656, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_4633255788259970659_77483, duration: 250202855
> 2010-12-20 17:26:37,538 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-8913559857952133277_85447 file /dist2/data/hadoop-data/current/subdir34/blk_-8913559857952133277 for deletion
> 2010-12-20 17:26:37,547 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:50010, dest: /10.112.79.14:41582, bytes: 19920, op: HDFS_READ, cliID: DFSClient_-1843172643, offset: 15987200, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_4633255788259970659_77483, duration: 4042558
> 2010-12-20 17:26:37,547 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-8511507105977516601_84587 file /dist3/data/hadoop-data/current/subdir31/blk_-8511507105977516601 for deletion
> 2010-12-20 17:26:37,549 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block blk_-8913559857952133277_85447 at file /dist2/data/hadoop-data/current/subdir34/blk_-8913559857952133277
> 2010-12-20 17:26:37,555 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-8508054838724986047_84580 file /dist2/data/hadoop-data/current/subdir41/blk_-8508054838724986047 for deletion
> 2010-12-20 17:26:37,556 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-7845924152894793652_85398 file /dist3/data/hadoop-data/current/subdir43/blk_-7845924152894793652 for deletion
> 2010-12-20 17:26:37,556 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-7088372163619556394_84570 file /dist2/data/hadoop-data/current/subdir19/blk_-7088372163619556394 for deletion
> 2010-12-20 17:26:37,556 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-6970445900444688558_85387 file /dist1/data/hadoop-data/current/subdir57/blk_-6970445900444688558 for deletion
> 2010-12-20 17:26:37,557 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-6453226657933735816_84505 file /dist1/data/hadoop-data/current/subdir43/blk_-6453226657933735816 for deletion
> 2010-12-20 17:26:37,557 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-6417800567255755267_84658 file /dist2/data/hadoop-data/current/subdir41/blk_-6417800567255755267 for deletion
> 2010-12-20 17:26:37,557 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-6349908890584705232_84612 file /dist1/data/hadoop-data/current/subdir57/blk_-6349908890584705232 for deletion
> 2010-12-20 17:26:37,557 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-6260136960291589474_84706 file /dist2/data/hadoop-data/current/subdir41/blk_-6260136960291589474 for deletion
> 2010-12-20 17:26:37,557 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-6236925753292021036_84691 file /dist2/data/hadoop-data/current/subdir41/blk_-6236925753292021036 for deletion
> 2010-12-20 17:26:37,557 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-6108807505647576258_84708 file /dist3/data/hadoop-data/current/subdir59/blk_-6108807505647576258 for deletion
> 2010-12-20 17:26:37,557 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-5322405269255073859_84530 file /dist2/data/hadoop-data/current/subdir19/blk_-5322405269255073859 for deletion
> 2010-12-20 17:26:37,558 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-4932349484140214063_84528 file /dist3/data/hadoop-data/current/subdir31/blk_-4932349484140214063 for deletion
> 2010-12-20 17:26:37,558 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-4903335785153601370_84648 file /dist3/data/hadoop-data/current/subdir59/blk_-4903335785153601370 for deletion
> 2010-12-20 17:26:37,558 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-4036825098230519223_84531 file /dist3/data/hadoop-data/current/subdir31/blk_-4036825098230519223 for deletion
> 2010-12-20 17:26:37,558 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-3437992750138243050_85434 file /dist2/data/hadoop-data/current/subdir34/blk_-3437992750138243050 for deletion
> 2010-12-20 17:26:37,558 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block blk_-6970445900444688558_85387 at file /dist1/data/hadoop-data/current/subdir57/blk_-6970445900444688558
> 2010-12-20 17:26:37,558 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-3364192718686291807_85433 file /dist1/data/hadoop-data/current/subdir48/blk_-3364192718686291807 for deletion
> 2010-12-20 17:26:37,558 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-3357432203841337074_84713 file /dist1/data/hadoop-data/current/subdir57/blk_-3357432203841337074 for deletion
> 2010-12-20 17:26:37,558 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-3238846993152973042_84497 file /dist1/data/hadoop-data/current/subdir43/blk_-3238846993152973042 for deletion
> 2010-12-20 17:26:37,558 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-2996249640811055720_83272 file /dist1/data/hadoop-data/current/subdir31/blk_-2996249640811055720 for deletion
> 2010-12-20 17:26:37,558 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-2790489595075270318_85390 file /dist3/data/hadoop-data/current/subdir43/blk_-2790489595075270318 for deletion
> 2010-12-20 17:26:37,558 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-2781143678253104315_84697 file /dist3/data/hadoop-data/current/subdir59/blk_-2781143678253104315 for deletion
> 2010-12-20 17:26:37,558 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-2640743692778440261_84536 file /dist2/data/hadoop-data/current/subdir19/blk_-2640743692778440261 for deletion
> 2010-12-20 17:26:37,558 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-2220284194502441495_85381 file /dist3/data/hadoop-data/current/subdir59/blk_-2220284194502441495 for deletion
> 2010-12-20 17:26:37,559 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-2126728115231716918_84647 file /dist2/data/hadoop-data/current/subdir41/blk_-2126728115231716918 for deletion
> 2010-12-20 17:26:37,559 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-2101072572804865450_84585 file /dist1/data/hadoop-data/current/subdir43/blk_-2101072572804865450 for deletion
> 2010-12-20 17:26:37,559 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-1725324590496466521_84695 file /dist1/data/hadoop-data/current/subdir57/blk_-1725324590496466521 for deletion
> 2010-12-20 17:26:37,559 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-1569098086020238107_83275 file /dist2/data/hadoop-data/current/subdir57/blk_-1569098086020238107 for deletion
> 2010-12-20 17:26:37,559 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-638559696544756508_85446 file /dist1/data/hadoop-data/current/subdir48/blk_-638559696544756508 for deletion
> 2010-12-20 17:26:37,559 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-306662003767169937_84555 file /dist1/data/hadoop-data/current/subdir43/blk_-306662003767169937 for deletion
> 2010-12-20 17:26:37,559 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-278326848888782043_84594 file /dist1/data/hadoop-data/current/subdir57/blk_-278326848888782043 for deletion
> 2010-12-20 17:26:37,559 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block blk_-6453226657933735816_84505 at file /dist1/data/hadoop-data/current/subdir43/blk_-6453226657933735816
> 2010-12-20 17:26:37,559 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-91580339643929179_85459 file /dist3/data/hadoop-data/current/subdir43/blk_-91580339643929179 for deletion
> 2010-12-20 17:26:37,559 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_9419128090539524_84662 file /dist1/data/hadoop-data/current/subdir57/blk_9419128090539524 for deletion
> 2010-12-20 17:26:37,559 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_679544100533753423_84719 file /dist1/data/hadoop-data/current/subdir57/blk_679544100533753423 for deletion
> 2010-12-20 17:26:37,559 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_730403277102726249_84590 file /dist3/data/hadoop-data/current/subdir31/blk_730403277102726249 for deletion
> 2010-12-20 17:26:37,572 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_797603508126576279_84551 file /dist3/data/hadoop-data/current/subdir31/blk_797603508126576279 for deletion
> 2010-12-20 17:26:37,572 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_867345031868208479_85424 file /dist3/data/hadoop-data/current/subdir43/blk_867345031868208479 for deletion
> 2010-12-20 17:26:37,572 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_1054277086916165125_84630 file /dist2/data/hadoop-data/current/subdir41/blk_1054277086916165125 for deletion
> 2010-12-20 17:26:37,572 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_1276716639229193075_85420 file /dist1/data/hadoop-data/current/subdir57/blk_1276716639229193075 for deletion
> 2010-12-20 17:26:37,572 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_1747684879678419666_84627 file /dist1/data/hadoop-data/current/subdir57/blk_1747684879678419666 for deletion
> 2010-12-20 17:26:37,572 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_1962685174146987659_84561 file /dist3/data/hadoop-data/current/subdir31/blk_1962685174146987659 for deletion
> 2010-12-20 17:26:37,572 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_2025768890877930088_84552 file /dist1/data/hadoop-data/current/subdir43/blk_2025768890877930088 for deletion
> 2010-12-20 17:26:37,572 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_2156546746312159503_84617 file /dist3/data/hadoop-data/current/subdir59/blk_2156546746312159503 for deletion
> 2010-12-20 17:26:37,572 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_2580899260621247350_85413 file /dist2/data/hadoop-data/current/subdir34/blk_2580899260621247350 for deletion
> 2010-12-20 17:26:37,572 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_2600394775430572452_85458 file /dist2/data/hadoop-data/current/subdir34/blk_2600394775430572452 for deletion
> 2010-12-20 17:26:37,572 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_2906704921361114164_84593 file /dist3/data/hadoop-data/current/subdir31/blk_2906704921361114164 for deletion
> 2010-12-20 17:26:37,572 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_3088125590653329452_85466 file /dist1/data/hadoop-data/current/subdir48/blk_3088125590653329452 for deletion
> 2010-12-20 17:26:37,572 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_3263574674943605427_84527 file /dist2/data/hadoop-data/current/subdir19/blk_3263574674943605427 for deletion
> 2010-12-20 17:26:37,573 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_3271569822852131416_85399 file /dist1/data/hadoop-data/current/subdir57/blk_3271569822852131416 for deletion
> 2010-12-20 17:26:37,573 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_3629755351194789263_84618 file /dist1/data/hadoop-data/current/subdir57/blk_3629755351194789263 for deletion
> 2010-12-20 17:26:37,573 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_3872473819820471284_84646 file /dist1/data/hadoop-data/current/subdir57/blk_3872473819820471284 for deletion
> 2010-12-20 17:26:37,573 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_3967429886843094718_84586 file /dist2/data/hadoop-data/current/subdir41/blk_3967429886843094718 for deletion
> 2010-12-20 17:26:37,573 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_4177484748478595642_84600 file /dist1/data/hadoop-data/current/subdir57/blk_4177484748478595642 for deletion
> 2010-12-20 17:26:37,573 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_4741750624588255690_85455 file /dist3/data/hadoop-data/current/subdir43/blk_4741750624588255690 for deletion
> 2010-12-20 17:26:37,573 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_5189334311200664471_84687 file /dist3/data/hadoop-data/current/subdir59/blk_5189334311200664471 for deletion
> 2010-12-20 17:26:37,573 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_5554094926453315166_84616 file /dist2/data/hadoop-data/current/subdir41/blk_5554094926453315166 for deletion
> 2010-12-20 17:26:37,573 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_5630019573499502243_85460 file /dist1/data/hadoop-data/current/subdir48/blk_5630019573499502243 for deletion
> 2010-12-20 17:26:37,573 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_6462688363000457793_84620 file /dist3/data/hadoop-data/current/subdir59/blk_6462688363000457793 for deletion
> 2010-12-20 17:26:37,573 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_6470741303722852096_84592 file /dist2/data/hadoop-data/current/subdir41/blk_6470741303722852096 for deletion
> 2010-12-20 17:26:37,573 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_6605638457633532879_84632 file /dist3/data/hadoop-data/current/subdir59/blk_6605638457633532879 for deletion
> 2010-12-20 17:26:37,573 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_6794523298330118718_77314 file /dist3/data/hadoop-data/current/subdir11/blk_6794523298330118718 for deletion
> 2010-12-20 17:26:37,574 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_6926053514624905958_84695 file /dist3/data/hadoop-data/current/subdir59/blk_6926053514624905958 for deletion
> 2010-12-20 17:26:37,574 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_7116690323560112908_85418 file /dist2/data/hadoop-data/current/subdir34/blk_7116690323560112908 for deletion
> 2010-12-20 17:26:37,574 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_7202997804403926942_84705 file /dist1/data/hadoop-data/current/subdir57/blk_7202997804403926942 for deletion
> 2010-12-20 17:26:37,574 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_7434927287494785191_85394 file /dist2/data/hadoop-data/current/subdir34/blk_7434927287494785191 for deletion
> 2010-12-20 17:26:37,574 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_7594986426526893113_84520 file /dist3/data/hadoop-data/current/subdir31/blk_7594986426526893113 for deletion
> 2010-12-20 17:26:37,574 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_7661061963483856377_85463 file /dist3/data/hadoop-data/current/subdir43/blk_7661061963483856377 for deletion
> 2010-12-20 17:26:37,574 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_7780046853952422303_85486 file /dist3/data/hadoop-data/current/subdir43/blk_7780046853952422303 for deletion
> 2010-12-20 17:26:37,574 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_7789033250788024385_83279 file /dist1/data/hadoop-data/current/subdir31/blk_7789033250788024385 for deletion
> 2010-12-20 17:26:37,574 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_7805477460105158814_84523 file /dist1/data/hadoop-data/current/subdir43/blk_7805477460105158814 for deletion
> 2010-12-20 17:26:37,574 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_7852099683471709790_84591 file /dist1/data/hadoop-data/current/subdir57/blk_7852099683471709790 for deletion
> 2010-12-20 17:26:37,574 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_8229638397827768666_85415 file /dist3/data/hadoop-data/current/subdir43/blk_8229638397827768666 for deletion
> 2010-12-20 17:26:37,574 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_8541283056878555347_85477 file /dist1/data/hadoop-data/current/subdir48/blk_8541283056878555347 for deletion
> 2010-12-20 17:26:37,574 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_8585904126405165382_85484 file /dist1/data/hadoop-data/current/subdir48/blk_8585904126405165382 for deletion
> 2010-12-20 17:26:37,575 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_8606266970652887102_84483 file /dist1/data/hadoop-data/current/subdir43/blk_8606266970652887102 for deletion
> 2010-12-20 17:26:37,575 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_9082358445568727636_85437 file /dist2/data/hadoop-data/current/subdir34/blk_9082358445568727636 for deletion
> 2010-12-20 17:26:37,575 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block blk_-6349908890584705232_84612 at file /dist1/data/hadoop-data/current/subdir57/blk_-6349908890584705232
>
> //////////////////////GC
>
> 2010-12-20T17:25:33.242+0000: 20.929: [CMS-concurrent-mark: 0.079/0.380 secs] [Times: user=0.36 sys=0.04, real=0.38 secs]
> 2010-12-20T17:25:33.242+0000: 20.929: [CMS-concurrent-preclean-start]
> 2010-12-20T17:25:33.245+0000: 20.932: [CMS-concurrent-preclean: 0.002/0.003 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:33.245+0000: 20.932: [CMS-concurrent-abortable-preclean-start]
> 2010-12-20T17:25:33.302+0000: 20.989: [GC 20.989: [ParNew: 17796K->1070K(19136K), 0.0025870 secs] 68068K->51343K(101612K) icms_dc=9 , 0.0027060 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:33.441+0000: 21.127: [GC 21.127: [ParNew: 18034K->2086K(19136K), 0.0042130 secs] 68307K->53943K(101612K) icms_dc=9 , 0.0043330 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
> 2010-12-20T17:25:33.514+0000: 21.201: [CMS-concurrent-abortable-preclean: 0.007/0.269 secs] [Times: user=0.19 sys=0.05, real=0.27 secs]
> 2010-12-20T17:25:33.515+0000: 21.202: [GC[YG occupancy: 11042 K (19136 K)]21.202: [Rescan (parallel) , 0.0158670 secs]21.218: [weak refs processing, 0.0000950 secs] [1 CMS-remark: 51857K(82476K)] 62900K(101612K), 0.0160770 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
> 2010-12-20T17:25:33.531+0000: 21.218: [CMS-concurrent-sweep-start]
> 2010-12-20T17:25:33.566+0000: 21.253: [GC 21.253: [ParNew: 19070K->1999K(19136K), 0.0027670 secs] 70927K->54085K(101612K) icms_dc=9 , 0.0028940 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:33.648+0000: 21.335: [GC 21.335: [ParNew: 19023K->2112K(19136K), 0.0046590 secs] 70961K->55368K(101612K) icms_dc=9 , 0.0047840 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
> 2010-12-20T17:25:33.713+0000: 21.400: [CMS-concurrent-sweep: 0.013/0.182 secs] [Times: user=0.14 sys=0.03, real=0.18 secs]
> 2010-12-20T17:25:33.714+0000: 21.401: [CMS-concurrent-reset-start]
> 2010-12-20T17:25:33.893+0000: 21.580: [GC 21.580: [ParNew: 19123K->598K(19136K), 0.0015670 secs] 72284K->53771K(107740K) icms_dc=9 , 0.0016740 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:33.945+0000: 21.632: [CMS-concurrent-reset: 0.020/0.231 secs] [Times: user=0.19 sys=0.07, real=0.23 secs]
> 2010-12-20T17:25:34.078+0000: 21.764: [GC 21.765: [ParNew: 17622K->1665K(19136K), 0.0021650 secs] 70795K->54839K(107740K) icms_dc=4 , 0.0022950 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:34.127+0000: 21.814: [GC [1 CMS-initial-mark: 53173K(88604K)] 63030K(107740K), 0.0031010 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:34.130+0000: 21.817: [CMS-concurrent-mark-start]
> 2010-12-20T17:25:34.181+0000: 21.867: [GC 21.867: [ParNew: 18633K->1360K(19136K), 0.0044440 secs] 71806K->55614K(107740K) icms_dc=5 , 0.0045740 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:34.387+0000: 22.074: [GC 22.074: [ParNew: 18370K->1923K(19136K), 0.0020250 secs] 72624K->56177K(107740K) icms_dc=5 , 0.0021580 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:34.557+0000: 22.244: [GC 22.244: [ParNew: 18947K->2112K(19136K), 0.0225430 secs] 73201K->58091K(107740K) icms_dc=6 , 0.0226640 secs] [Times: user=0.02 sys=0.00, real=0.03 secs]
> 2010-12-20T17:25:34.681+0000: 22.367: [GC 22.367: [ParNew: 19098K->811K(19136K), 0.0031310 secs] 75077K->57280K(107740K) icms_dc=6 , 0.0032510 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:34.751+0000: 22.437: [CMS-concurrent-mark: 0.087/0.620 secs] [Times: user=0.42 sys=0.04, real=0.63 secs]
> 2010-12-20T17:25:34.751+0000: 22.438: [CMS-concurrent-preclean-start]
> 2010-12-20T17:25:34.757+0000: 22.444: [CMS-concurrent-preclean: 0.005/0.006 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:34.757+0000: 22.444: [CMS-concurrent-abortable-preclean-start]
> 2010-12-20T17:25:34.865+0000: 22.552: [GC 22.552: [ParNew: 17835K->1114K(19136K), 0.0016200 secs] 74304K->57582K(107740K) icms_dc=6 , 0.0017310 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:34.966+0000: 22.652: [CMS-concurrent-abortable-preclean: 0.003/0.209 secs] [Times: user=0.10 sys=0.02, real=0.21 secs]
> 2010-12-20T17:25:34.966+0000: 22.653: [GC[YG occupancy: 9657 K (19136 K)]22.653: [Rescan (parallel) , 0.0020570 secs]22.655: [weak refs processing, 0.0000610 secs] [1 CMS-remark: 56468K(88604K)] 66126K(107740K), 0.0022320 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
> 2010-12-20T17:25:34.969+0000: 22.656: [CMS-concurrent-sweep-start]
> 2010-12-20T17:25:35.101+0000: 22.787: [GC 22.787: [ParNew: 18138K->2112K(19136K), 0.0038570 secs] 74584K->60794K(107740K) icms_dc=6 , 0.0039870 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:35.214+0000: 22.900: [CMS-concurrent-sweep: 0.013/0.245 secs] [Times: user=0.07 sys=0.01, real=0.24 secs]
> 2010-12-20T17:25:35.214+0000: 22.900: [CMS-concurrent-reset-start]
> 2010-12-20T17:25:35.354+0000: 23.041: [GC 23.041: [ParNew: 19136K->1909K(19136K), 0.0035100 secs] 75099K->58683K(112412K) icms_dc=6 , 0.0036250 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
> 2010-12-20T17:25:35.449+0000: 23.136: [GC 23.136: [ParNew: 18931K->1384K(19136K), 0.0027830 secs] 75706K->59022K(112412K) icms_dc=6 , 0.0029070 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:35.499+0000: 23.186: [CMS-concurrent-reset: 0.018/0.285 secs] [Times: user=18446743517.92 sys=0.04, real=0.29 secs]
> 2010-12-20T17:25:35.606+0000: 23.292: [GC 23.292: [ParNew: 18352K->1675K(19136K), 0.0019650 secs] 75990K->59313K(112412K) icms_dc=1 , 0.0020900 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
> 2010-12-20T17:25:35.661+0000: 23.348: [GC [1 CMS-initial-mark: 57637K(93276K)] 67851K(112412K), 0.0032750 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:35.664+0000: 23.351: [CMS-concurrent-mark-start]
> 2010-12-20T17:25:35.779+0000: 23.466: [GC 23.466: [ParNew: 18699K->2112K(19136K), 0.0049320 secs] 76337K->60788K(112412K) icms_dc=4 , 0.0050890 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
> 2010-12-20T17:25:35.963+0000: 23.650: [GC 23.650: [ParNew: 19073K->1947K(19136K), 0.0080830 secs] 77750K->61429K(112412K) icms_dc=4 , 0.0082030 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
> 2010-12-20T17:25:36.237+0000: 23.924: [GC 23.924: [ParNew: 18964K->1547K(19136K), 0.0041330 secs] 78447K->61930K(112412K) icms_dc=4 , 0.0042610 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
> 2010-12-20T17:25:36.361+0000: 24.048: [GC 24.048: [ParNew: 18514K->987K(19136K), 0.0038260 secs] 78897K->61370K(112412K) icms_dc=4 , 0.0039330 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
> 2010-12-20T17:25:36.454+0000: 24.141: [CMS-concurrent-mark: 0.089/0.790 secs] [Times: user=36893488016.88 sys=0.06, real=0.79 secs]
> 2010-12-20T17:25:36.454+0000: 24.141: [CMS-concurrent-preclean-start]
> 2010-12-20T17:25:36.460+0000: 24.147: [CMS-concurrent-preclean: 0.005/0.006 secs] [Times: user=0.00 sys=0.02, real=0.01 secs]
> 2010-12-20T17:25:36.460+0000: 24.147: [CMS-concurrent-abortable-preclean-start]
> 2010-12-20T17:25:36.520+0000: 24.207: [GC 24.207: [ParNew: 17969K->1341K(19136K), 0.0023940 secs] 78352K->61724K(112412K) icms_dc=4 , 0.0025130 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:36.728+0000: 24.414: [GC 24.414: [ParNew: 18356K->1010K(19136K), 0.0025740 secs] 78739K->61870K(112412K) icms_dc=4 , 0.0026990 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
> 2010-12-20T17:25:36.800+0000: 24.487: [CMS-concurrent-abortable-preclean: 0.009/0.340 secs] [Times: user=0.26 sys=0.02, real=0.34 secs]
> 2010-12-20T17:25:36.869+0000: 24.555: [GC[YG occupancy: 9861 K (19136 K)]24.555: [Rescan (parallel) , 0.0012070 secs]24.557: [weak refs processing, 0.0000890 secs] [1 CMS-remark: 60859K(93276K)] 70720K(112412K), 0.0014050 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:36.870+0000: 24.557: [CMS-concurrent-sweep-start]
> 2010-12-20T17:25:36.948+0000: 24.634: [GC 24.634: [ParNew: 18027K->1183K(19136K), 0.0035250 secs] 78793K->61949K(112412K) icms_dc=4 , 0.0036410 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:37.088+0000: 24.775: [CMS-concurrent-sweep: 0.015/0.218 secs] [Times: user=0.05 sys=0.00, real=0.21 secs]
> 2010-12-20T17:25:37.088+0000: 24.775: [CMS-concurrent-reset-start]
> 2010-12-20T17:25:37.105+0000: 24.791: [CMS-concurrent-reset: 0.016/0.016 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
> 2010-12-20T17:25:37.155+0000: 24.842: [GC 24.842: [ParNew: 18153K->961K(19136K), 0.0233150 secs] 76441K->59351K(116284K) icms_dc=0 , 0.0234410 secs] [Times: user=0.02 sys=0.00, real=0.03 secs]
> 2010-12-20T17:25:37.293+0000: 24.980: [GC 24.980: [ParNew: 17985K->1566K(19136K), 0.0078520 secs] 76375K->59957K(116284K) icms_dc=0 , 0.0079770 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
> 2010-12-20T17:25:37.470+0000: 25.157: [GC 25.157: [ParNew: 18533K->2095K(19136K), 0.0028390 secs] 76923K->60833K(116284K) icms_dc=0 , 0.0029650 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:37.628+0000: 25.315: [GC 25.315: [ParNew: 19119K->2112K(19136K), 0.0032590 secs] 77857K->61573K(116284K) icms_dc=0 , 0.0033810 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
> 2010-12-20T17:25:37.692+0000: 25.379: [GC 25.379: [ParNew: 19092K->1664K(19136K), 0.0027200 secs] 78554K->61695K(116284K) icms_dc=0 , 0.0028400 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:37.814+0000: 25.501: [GC 25.501: [ParNew: 18688K->1461K(19136K), 0.0017560 secs] 78719K->61492K(116284K) icms_dc=0 , 0.0018670 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:37.946+0000: 25.632: [GC 25.632: [ParNew: 18449K->1309K(19136K), 0.0052360 secs] 78481K->61761K(116284K) icms_dc=0 , 0.0053520 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
> 2010-12-20T17:25:38.114+0000: 25.800: [GC 25.800: [ParNew: 18333K->2112K(19136K), 0.0046840 secs] 78785K->64510K(116284K) icms_dc=0 , 0.0048140 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:38.229+0000: 25.915: [GC 25.915: [ParNew: 19132K->1927K(19136K), 0.0301210 secs] 81531K->65315K(116284K) icms_dc=0 , 0.0302420 secs] [Times: user=0.03 sys=0.00, real=0.03 secs]
> 2010-12-20T17:25:38.404+0000: 26.091: [GC 26.091: [ParNew: 18887K->1639K(19136K), 0.0029570 secs] 82275K->65027K(116284K) icms_dc=0 , 0.0030810 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:38.551+0000: 26.237: [GC 26.237: [ParNew: 18655K->2092K(19136K), 0.0040070 secs] 82042K->66522K(116284K) icms_dc=0 , 0.0041290 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
> 2010-12-20T17:25:38.810+0000: 26.497: [GC 26.497: [ParNew: 19101K->1244K(19136K), 0.0036340 secs] 83531K->66458K(116284K) icms_dc=0 , 0.0037620 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:38.920+0000: 26.607: [GC 26.607: [ParNew: 18253K->684K(19136K), 0.0014730 secs] 83468K->65898K(116284K) icms_dc=0 , 0.0015700 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:39.065+0000: 26.751: [GC 26.751: [ParNew: 17669K->782K(19136K), 0.0015630 secs] 82884K->65996K(116284K) icms_dc=0 , 0.0016660 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
> 2010-12-20T17:25:39.213+0000: 26.900: [GC 26.900: [ParNew: 17756K->783K(19136K), 0.0015390 secs] 82970K->65998K(116284K) icms_dc=0 , 0.0016400 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:39.374+0000: 27.061: [GC 27.061: [ParNew: 17787K->1548K(19136K), 0.0023500 secs] 83001K->66975K(116284K) icms_dc=0 , 0.0024690 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
> 2010-12-20T17:25:39.624+0000: 27.311: [GC 27.311: [ParNew: 18571K->798K(19136K), 0.0015980 secs] 83998K->66324K(116284K) icms_dc=0 , 0.0016950 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:39.736+0000: 27.423: [GC 27.423: [ParNew: 17771K->936K(19136K), 0.0014100 secs] 83297K->66463K(116284K) icms_dc=0 , 0.0015130 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:39.906+0000: 27.592: [GC 27.592: [ParNew: 17920K->1886K(19136K), 0.0020160 secs] 83447K->67413K(116284K) icms_dc=0 , 0.0021370 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
> 2010-12-20T17:25:40.042+0000: 27.729: [GC 27.729: [ParNew: 18905K->1886K(19136K), 0.0040910 secs] 84431K->68678K(116284K) icms_dc=0 , 0.0042270 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:40.213+0000: 27.900: [GC 27.900: [ParNew: 18855K->1500K(19136K), 0.0031110 secs] 85647K->68745K(116284K) icms_dc=0 , 0.0032440 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:40.323+0000: 28.010: [GC 28.010: [ParNew: 18524K->2112K(19136K), 0.0060030 secs] 85769K->72197K(116284K) icms_dc=4 , 0.0061260 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:40.484+0000: 28.170: [GC [1 CMS-initial-mark: 70085K(97148K)] 80410K(116284K), 0.0038900 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
> 2010-12-20T17:25:40.488+0000: 28.175: [CMS-concurrent-mark-start]
> 2010-12-20T17:25:40.567+0000: 28.254: [GC 28.254: [ParNew: 19136K->1886K(19136K), 0.0043620 secs] 89221K->72619K(116284K) icms_dc=10 , 0.0044950 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:40.752+0000: 28.439: [GC 28.439: [ParNew: 18883K->1596K(19136K), 0.0045560 secs] 89617K->73198K(116284K) icms_dc=10 , 0.0046680 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
> 2010-12-20T17:25:41.018+0000: 28.704: [GC 28.705: [ParNew: 18590K->650K(19136K), 0.0046350 secs] 90191K->72706K(116284K) icms_dc=10 , 0.0047500 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:41.129+0000: 28.816: [GC 28.816: [ParNew: 17674K->399K(19136K), 0.0014520 secs] 89730K->72455K(116284K) icms_dc=10 , 0.0015540 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:41.290+0000: 28.977: [GC 28.977: [ParNew: 17396K->581K(19136K), 0.0021900 secs] 89452K->72637K(116284K) icms_dc=10 , 0.0022930 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
> 2010-12-20T17:25:41.478+0000: 29.164: [GC 29.165: [ParNew: 17569K->929K(19136K), 0.0018590 secs] 89625K->72986K(116284K) icms_dc=10 , 0.0019820 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:41.653+0000: 29.340: [GC 29.340: [ParNew: 17894K->1013K(19136K), 0.0333480 secs] 89950K->73172K(116284K) icms_dc=10 , 0.0334710 secs] [Times: user=0.07 sys=0.00, real=0.04 secs]
> 2010-12-20T17:25:41.763+0000: 29.450: [CMS-concurrent-mark: 0.101/1.275 secs] [Times: user=18446743473.28 sys=0.08, real=1.28 secs]
> 2010-12-20T17:25:41.763+0000: 29.450: [CMS-concurrent-preclean-start]
> 2010-12-20T17:25:41.887+0000: 29.574: [GC 29.574: [ParNew: 18024K->1580K(19136K), 0.0025720 secs] 90183K->73783K(116284K) icms_dc=10 , 0.0026960 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:41.948+0000: 29.634: [CMS-concurrent-preclean: 0.006/0.184 secs] [Times: user=0.11 sys=0.01, real=0.18 secs]
> 2010-12-20T17:25:41.948+0000: 29.634: [CMS-concurrent-abortable-preclean-start]
> 2010-12-20T17:25:42.010+0000: 29.697: [GC 29.697: [ParNew: 18571K->1529K(19136K), 0.0261550 secs] 90773K->74878K(116284K) icms_dc=10 , 0.0262850 secs] [Times: user=0.01 sys=0.01, real=0.03 secs]
> 2010-12-20T17:25:42.037+0000: 29.723: [CMS-concurrent-abortable-preclean: 0.002/0.089 secs] [Times: user=0.05 sys=0.01, real=0.09 secs]
> 2010-12-20T17:25:42.037+0000: 29.724: [GC[YG occupancy: 1657 K (19136 K)]29.724: [Rescan (parallel) , 0.0021770 secs]29.726: [weak refs processing, 0.0002560 secs] [1 CMS-remark: 73349K(97148K)] 75006K(116284K), 0.0025180 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:42.040+0000: 29.727: [CMS-concurrent-sweep-start]
> 2010-12-20T17:25:42.143+0000: 29.830: [GC 29.830: [ParNew: 18553K->2112K(19136K), 0.0039990 secs] 91513K->76482K(116284K) icms_dc=10 , 0.0041230 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:42.159+0000: 29.846: [CMS-concurrent-sweep: 0.021/0.119 secs] [Times: user=0.11 sys=0.02, real=0.12 secs]
> 2010-12-20T17:25:42.159+0000: 29.846: [CMS-concurrent-reset-start]
> 2010-12-20T17:25:42.180+0000: 29.867: [CMS-concurrent-reset: 0.021/0.021 secs] [Times: user=0.02 sys=0.01, real=0.02 secs]
> 2010-12-20T17:25:42.220+0000: 29.907: [GC [1 CMS-initial-mark: 72204K(120344K)] 82008K(139480K), 0.0031690 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
> 2010-12-20T17:25:42.223+0000: 29.910: [CMS-concurrent-mark-start]
> 2010-12-20T17:25:42.363+0000: 30.050: [GC 30.050: [ParNew: 19089K->1911K(19136K), 0.0287550 secs] 91294K->74871K(139480K) icms_dc=5 , 0.0288700 secs] [Times: user=0.02 sys=0.00, real=0.03 secs]
> 2010-12-20T17:25:42.520+0000: 30.207: [GC 30.207: [ParNew: 18890K->2095K(19136K), 0.0026020 secs] 91850K->76245K(139480K) icms_dc=5 , 0.0027290 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:42.626+0000: 30.313: [GC 30.313: [ParNew: 19116K->1072K(19136K), 0.0021620 secs] 93267K->75735K(139480K) icms_dc=5 , 0.0022790 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:42.794+0000: 30.480: [GC 30.480: [ParNew: 18095K->1985K(19136K), 0.0224570 secs] 92758K->76648K(139480K) icms_dc=5 , 0.0225750 secs] [Times: user=0.02 sys=0.00, real=0.03 secs]
> 2010-12-20T17:25:43.028+0000: 30.715: [GC 30.715: [ParNew: 18975K->1230K(19136K), 0.0029720 secs] 93638K->76922K(139480K) icms_dc=5 , 0.0031070 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:43.127+0000: 30.814: [CMS-concurrent-mark: 0.101/0.904 secs] [Times: user=45.86 sys=0.07, real=0.90 secs]
> 2010-12-20T17:25:43.127+0000: 30.814: [CMS-concurrent-preclean-start]
> 2010-12-20T17:25:43.140+0000: 30.826: [CMS-concurrent-preclean: 0.010/0.012 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
> 2010-12-20T17:25:43.140+0000: 30.826: [CMS-concurrent-abortable-preclean-start]
> 2010-12-20T17:25:43.263+0000: 30.950: [GC 30.950: [ParNew: 18242K->1343K(19136K), 0.0013570 secs] 93934K->77035K(139480K) icms_dc=5 , 0.0014830 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:43.439+0000: 31.126: [GC 31.126: [ParNew: 18367K->1276K(19136K), 0.0013350 secs] 94059K->76968K(139480K) icms_dc=5 , 0.0014590 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:43.539+0000: 31.225: [CMS-concurrent-abortable-preclean: 0.008/0.399 secs] [Times: user=0.16 sys=0.02, real=0.40 secs]
> 2010-12-20T17:25:43.539+0000: 31.226: [GC[YG occupancy: 11143 K (19136 K)]31.226: [Rescan (parallel) , 0.0012840 secs]31.227: [weak refs processing, 0.0000800 secs] [1 CMS-remark: 75692K(120344K)] 86835K(139480K), 0.0014660 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:43.566+0000: 31.252: [CMS-concurrent-sweep-start]
> 2010-12-20T17:25:43.588+0000: 31.275: [CMS-concurrent-sweep: 0.022/0.022 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
> 2010-12-20T17:25:43.588+0000: 31.275: [CMS-concurrent-reset-start]
> 2010-12-20T17:25:43.606+0000: 31.292: [CMS-concurrent-reset: 0.017/0.017 secs] [Times: user=0.01 sys=0.00, real=0.02 secs]
> 2010-12-20T17:25:43.617+0000: 31.303: [GC 31.303: [ParNew: 18237K->1557K(19136K), 0.0016650 secs] 91506K->74826K(141252K) icms_dc=0 , 0.0017980 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:43.766+0000: 31.452: [GC 31.452: [ParNew: 18525K->986K(19136K), 0.0019760 secs] 91794K->74744K(141252K) icms_dc=0 , 0.0021000 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:43.984+0000: 31.671: [GC 31.671: [ParNew: 18010K->1004K(19136K), 0.0011630 secs] 91768K->74762K(141252K) icms_dc=0 , 0.0012740 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:44.155+0000: 31.842: [GC 31.842: [ParNew: 17986K->1535K(19136K), 0.0020850 secs] 91744K->75294K(141252K) icms_dc=0 , 0.0022080 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:44.279+0000: 31.965: [GC 31.965: [ParNew: 18525K->1844K(19136K), 0.0036450 secs] 92284K->75963K(141252K) icms_dc=0 , 0.0037850 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
> 2010-12-20T17:25:44.446+0000: 32.132: [GC 32.133: [ParNew: 18868K->2112K(19136K), 0.0027860 secs] 92987K->76905K(141252K) icms_dc=0 , 0.0029120 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
> 2010-12-20T17:25:44.560+0000: 32.247: [GC 32.247: [ParNew: 19136K->2062K(19136K), 0.0035670 secs] 93929K->77864K(141252K) icms_dc=0 , 0.0036890 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
> 2010-12-20T17:25:44.786+0000: 32.473: [GC 32.473: [ParNew: 19081K->2099K(19136K), 0.0042870 secs] 94883K->79044K(141252K) icms_dc=0 , 0.0044120 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:45.051+0000: 32.737: [GC 32.738: [ParNew: 19082K->2112K(19136K), 0.0057110 secs] 96026K->79932K(141252K) icms_dc=0 , 0.0058410 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:45.159+0000: 32.845: [GC 32.845: [ParNew: 19132K->1902K(19136K), 0.0048970 secs] 96953K->80775K(141252K) icms_dc=0 , 0.0050310 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:45.315+0000: 33.001: [GC 33.001: [ParNew: 18888K->1172K(19136K), 0.0016980 secs] 97761K->80045K(141252K) icms_dc=0 , 0.0018120 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
> 2010-12-20T17:25:45.416+0000: 33.103: [GC 33.103: [ParNew: 18143K->1113K(19136K), 0.0016580 secs] 97016K->79987K(141252K) icms_dc=0 , 0.0017670 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:45.571+0000: 33.258: [GC 33.258: [ParNew: 18117K->1175K(19136K), 0.0018150 secs] 96990K->80049K(141252K) icms_dc=0 , 0.0019290 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:45.716+0000: 33.403: [GC 33.403: [ParNew: 18193K->2092K(19136K), 0.0243240 secs] 97067K->83760K(141252K) icms_dc=0 , 0.0244520 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
> 2010-12-20T17:25:46.275+0000: 33.961: [GC 33.961: [ParNew: 19116K->1282K(19136K), 0.0251550 secs] 100784K->83201K(141252K) icms_dc=0 , 0.0252890 secs] [Times: user=0.06 sys=0.00, real=0.03 secs]
> 2010-12-20T17:25:46.421+0000: 34.108: [GC 34.108: [ParNew: 18286K->1295K(19136K), 0.0186440 secs] 100205K->83214K(141252K) icms_dc=0 , 0.0187610 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
> 2010-12-20T17:25:46.676+0000: 34.363: [GC 34.363: [ParNew: 18319K->2112K(19136K), 0.0062280 secs] 100238K->84044K(141252K) icms_dc=0 , 0.0063550 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:46.825+0000: 34.512: [GC 34.512: [ParNew: 19136K->2112K(19136K), 0.0235520 secs] 101068K->87526K(141252K) icms_dc=0 , 0.0236890 secs] [Times: user=0.02 sys=0.00, real=0.03 secs]
> 2010-12-20T17:25:47.010+0000: 34.696: [GC 34.696: [ParNew: 19136K->1126K(19136K), 0.0036520 secs] 104550K->86998K(141252K) icms_dc=0 , 0.0037680 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:47.179+0000: 34.866: [GC 34.866: [ParNew: 18094K->2112K(19136K), 0.0036740 secs] 103967K->89106K(141252K) icms_dc=0 , 0.0037950 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:47.354+0000: 35.041: [GC 35.041: [ParNew: 19136K->1976K(19136K), 0.0042910 secs] 106130K->90305K(141252K) icms_dc=0 , 0.0044170 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:47.523+0000: 35.210: [GC 35.210: [ParNew: 18973K->2112K(19136K), 0.0020850 secs] 107303K->90449K(141252K) icms_dc=0 , 0.0022090 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:47.757+0000: 35.443: [GC 35.443: [ParNew: 19086K->1740K(19136K), 0.0047220 secs] 107424K->91042K(141252K) icms_dc=0 , 0.0048520 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:47.949+0000: 35.635: [GC 35.635: [ParNew: 18758K->1349K(19136K), 0.0100480 secs] 108060K->90651K(141252K) icms_dc=0 , 0.0101550 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
> 2010-12-20T17:25:48.160+0000: 35.846: [GC 35.847: [ParNew: 18373K->1062K(19136K), 0.0035050 secs] 107675K->90707K(141252K) icms_dc=0 , 0.0036170 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:48.363+0000: 36.050: [GC 36.050: [ParNew: 18057K->746K(19136K), 0.0150130 secs] 107702K->90391K(141252K) icms_dc=0 , 0.0151200 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
> 2010-12-20T17:25:48.547+0000: 36.234: [GC 36.234: [ParNew: 17752K->2057K(19136K), 0.0048080 secs] 107397K->92215K(141252K) icms_dc=0 , 0.0049410 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:48.755+0000: 36.442: [GC 36.442: [ParNew: 19042K->1221K(19136K), 0.0231150 secs] 109199K->91615K(141252K) icms_dc=0 , 0.0232640 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
> 2010-12-20T17:25:48.867+0000: 36.554: [GC 36.554: [ParNew: 18227K->2102K(19136K), 0.0218400 secs] 108621K->95180K(141252K) icms_dc=0 , 0.0219730 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
> 2010-12-20T17:25:49.086+0000: 36.773: [GC 36.773: [ParNew: 19090K->1186K(19136K), 0.0040110 secs] 112167K->94679K(141252K) icms_dc=0 , 0.0041370 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:49.186+0000: 36.873: [GC 36.873: [ParNew: 18171K->1369K(19136K), 0.0018640 secs] 111664K->94862K(141252K) icms_dc=0 , 0.0019870 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:49.311+0000: 36.998: [GC 36.998: [ParNew: 18393K->154K(19136K), 0.0049150 secs] 111886K->94007K(141252K) icms_dc=0 , 0.0050180 secs] [Times: user=0.00 sys=0.01, real=0.01 secs]
> 2010-12-20T17:25:49.433+0000: 37.120: [GC 37.120: [ParNew: 17165K->2079K(19136K), 0.0081480 secs] 111018K->98366K(141252K) icms_dc=4 , 0.0082710 secs] [Times: user=0.00 sys=0.01, real=0.01 secs]
> 2010-12-20T17:25:49.544+0000: 37.230: [GC [1 CMS-initial-mark: 96287K(122116K)] 106652K(141252K), 0.0041720 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:25:49.548+0000: 37.235: [CMS-concurrent-mark-start]
> 2010-12-20T17:25:49.706+0000: 37.393: [GC 37.393: [ParNew: 19059K->818K(19136K), 47.8334700 secs] 115346K->97662K(141252K) icms_dc=10 , 47.8335900 secs] [Times: user=90.47 sys=0.01, real=47.86 secs]
> 2010-12-20T17:26:37.884+0000: 85.570: [GC 85.571: [ParNew: 17836K->541K(19136K), 0.0405420 secs] 114680K->97386K(141252K) icms_dc=10 , 0.0406480 secs] [Times: user=0.00 sys=0.00, real=0.04 secs]
> 2010-12-20T17:26:47.779+0000: 95.466: [CMS-concurrent-mark: 0.103/58.231 secs] [Times: user=36893487576.24 sys=0.98, real=58.22 secs]
> 2010-12-20T17:26:47.779+0000: 95.466: [CMS-concurrent-preclean-start]
> 2010-12-20T17:26:47.790+0000: 95.476: [CMS-concurrent-preclean: 0.008/0.010 secs] [Times: user=0.00 sys=0.01, real=0.01 secs]
> 2010-12-20T17:26:47.790+0000: 95.476: [CMS-concurrent-abortable-preclean-start]
> 2010-12-20T17:26:47.790+0000: 95.476: [CMS-concurrent-abortable-preclean: 0.000/0.000 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-12-20T17:26:47.790+0000: 95.477: [GC[YG occupancy: 3204 K (19136 K)]95.477: [Rescan (parallel) , 0.0018430 secs]95.479: [weak refs processing, 0.0006910 secs] [1 CMS-remark: 96844K(122116K)] 100048K(141252K), 0.0026330 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
> 2010-12-20T17:26:47.793+0000: 95.479: [CMS-concurrent-sweep-start]
> 2010-12-20T17:26:47.826+0000: 95.513: [CMS-concurrent-sweep: 0.033/0.033 secs] [Times: user=0.03 sys=0.00, real=0.03 secs]
> 2010-12-20T17:26:47.826+0000: 95.513: [CMS-concurrent-reset-start]
> 2010-12-20T17:26:47.842+0000: 95.528: [CMS-concurrent-reset: 0.016/0.016 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
> Heap
> par new generation total 19136K, used 3204K [0x00007f611e340000, 0x00007f611f800000, 0x00007f6123670000)
> eden space 17024K, 15% used [0x00007f611e340000, 0x00007f611e5d9b38, 0x00007f611f3e0000)
> from space 2112K, 25% used [0x00007f611f3e0000, 0x00007f611f467768, 0x00007f611f5f0000)
> to space 2112K, 0% used [0x00007f611f5f0000, 0x00007f611f5f0000, 0x00007f611f800000)
> concurrent mark-sweep generation total 122116K, used 73206K [0x00007f6123670000, 0x00007f612adb1000, 0x00007f6295340000)
> concurrent-mark-sweep perm gen total 29708K, used 17810K [0x00007f6295340000, 0x00007f6297043000, 0x00007f629a740000)
>
>
>
> On Dec 9, 2010, at 4:09 PM, Lance Riedel wrote:
>
>> I came across just probably the culprit - I just discovered that the machines that were dying had OS swap turned OFF.
>>
>> So, I added swap to those machines, but also reduced the memory configs for tasks. (was at 1 gig, now at 512m). Seems to be stable right now, but the nightly longer term jobs run at night, so we will see how they do.
>>
>> This is an ec2 c1.xlarge, so the memory is at 8 gigs. I had 4 mappers per node, which was fine when all they had on them were Tasktrackers and datanodes. But with the hbase regionservers, the machines could definitely get over-allocated.
>>
>> Lance
>>
>> On Dec 9, 2010, at 3:57 PM, Ted Dunning wrote:
>>
>>> This could indicate swapping during GC.
>>>
>>> On Thu, Dec 9, 2010 at 12:13 PM, Lance Riedel <la...@gmail.com> wrote:
>>>
>>>> Seems reasonable, but having trouble making sense of the GC logs I had
>>>> turned on. Basically since there was a full GC a minute before this happens
>>>> on that server that lasts less than a second.
>>>>
>>>>
>>>> Example:
>>>> So, here is what the last of the GC logs say for that Regionserver
>>>> (04.hadoop on 10.100.154.10) (note the last full GC only 1 minutes before @
>>>> 5:03:29.850+0000):
>>>>
>>>> 2010-12-09T05:02:19.472+0000: 12170.437: [GC [PSYoungGen:
>>>> 278229K->5024K(287360K)] 694269K->421693K(721856K), 0.0274840 secs] [Times:
>>>> user=0.04 sys=0.00, real=0.03 secs]
>>>> 2010-12-09T05:02:55.479+0000: 12206.444: [GC [PSYoungGen:
>>>> 281450K->7968K(273344K)] 698120K->429658K(707840K), 0.0503000 secs] [Times:
>>>> user=0.10 sys=0.00, real=0.05 secs]
>>>> 2010-12-09T05:03:29.800+0000: 12240.766: [GC [PSYoungGen:
>>>> 273295K->7712K(262464K)] 694986K->437306K(696960K), 0.0490770 secs] [Times:
>>>> user=0.12 sys=0.01, real=0.05 secs]
>>>> 2010-12-09T05:03:29.850+0000: 12240.815: [Full GC [PSYoungGen:
>>>> 7712K->0K(262464K)] [PSOldGen: 429594K->353152K(429824K)]
>>>> 437306K->353152K(692288K) [PSPermGen: 18016K->18016K(21248K)], 0.8960760
>>>> secs] [Times: user=0.89 sys=0.00, real=0.90 secs]
>>>> 2010-12-09T05:04:03.345+0000: 12274.310: [GC [PSYoungGen:
>>>> 254720K->6944K(251584K)] 607872K->360528K(681408K), 0.0331060 secs] [Times:
>>>> user=0.03 sys=0.00, real=0.04 secs]
>>>> 2010-12-09T05:04:33.875+0000: 12304.840: [GC [PSYoungGen:
>>>> 251552K->7200K(242240K)] 605136K->367800K(672064K), 0.0336420 secs] [Times:
>>>> user=0.09 sys=0.01, real=0.03 secs]
>>>> Heap
>>>> PSYoungGen total 242240K, used 223390K [0x00007fcc640b0000,
>>>> 0x00007fcc77a90000, 0x00007fcc78e00000)
>>>> eden space 235008K, 91% used
>>>> [0x00007fcc640b0000,0x00007fcc713cfa30,0x00007fcc72630000)
>>>> from space 7232K, 99% used
>>>> [0x00007fcc769b0000,0x00007fcc770b8090,0x00007fcc770c0000)
>>>> to space 10048K, 0% used
>>>> [0x00007fcc770c0000,0x00007fcc770c0000,0x00007fcc77a90000)
>>>> PSOldGen total 429824K, used 360600K [0x00007fcc3a600000,
>>>> 0x00007fcc549c0000, 0x00007fcc640b0000)
>>>> object space 429824K, 83% used
>>>> [0x00007fcc3a600000,0x00007fcc50626060,0x00007fcc549c0000)
>>>> PSPermGen total 21248K, used 18131K [0x00007fcc35200000,
>>>> 0x00007fcc366c0000, 0x00007fcc3a600000)
>>>> object space 21248K, 85% used
>>>> [0x00007fcc35200000,0x00007fcc363b4d60,0x00007fcc366c0000)
>>>>
>>>> Now Dead / End of log
>>>>
>>>>
>>>> That said, I will continue down the path you pointed me on .
>>>>
>>>> Thanks,
>>>> Lance
>>>>
>>>> On Dec 9, 2010, at 11:40 AM, Jean-Daniel Cryans wrote:
>>>>
>>>>> Lance,
>>>>>
>>>>> Both those lines indicate the problem:
>>>>>
>>>>> IPC Server handler 13 on 60020 took 182416ms
>>>>> Client session timed out, have not heard from server in 182936ms
>>>>>
>>>>> It's very clear that your region servers are suffering from
>>>>> pause-of-the-world garbage collection issues. Basically this one GC'ed
>>>>> for 3 minutes, which is over the 1 minute timeout to consider a region
>>>>> server dead. The rest is just the side effect of that.
>>>>>
>>>>> This subject comes often on the mailing list, for example:
>>>>> http://search-hadoop.com/m/t97q027tr7f2
>>>>>
>>>>> Make sure you don't swap, give plenty of RAM to HBase, use LZO
>>>>> compression, don't underprovision your machines, etc
>>>>>
>>>>> J-D
>>>>>
>>>>> On Thu, Dec 9, 2010 at 11:27 AM, Lance Riedel <la...@gmail.com>
>>>> wrote:
>>>>>>
>>>>>> We have a 6 node cluster, 5 with region serves. 2 of the region servers
>>>> have been stable for days, but 3 of them keep crashing. Here are the logs
>>>> around around when the crash occurs. (btw, we are shoving approximately the
>>>> twitter firehose into hbase via flume) I'm an hbase newbie, but I have been
>>>> reading. Not sure what else is needed to help debug. When the problem
>>>> occurs, it is concurrent with issues that appear all the way down in the
>>>> data node. Also, interesting to note, all 3 servers seem to fail
>>>> differently somewhat:
>>>>>>
>>>>>> Servers (6):
>>>>>> 00.hadoop is the master
>>>>>> 01.hadoop, 03.hadoop (there is no 02.hadoop - flaky machine that got
>>>> taken out) - are stable Region Servers
>>>>>> 04-06.hadoop crash at different times, very different logs, Region
>>>> Servers
>>>>>>
>>>>>>
>>>>>> Master Attributes:
>>>>>> HBase Version 0.89.20100924+28, r
>>>>>> Hadoop Version 0.20.2+737, r98c55c28258aa6f42250569bd7fa431ac657bdbd
>>>>>>
>>>>>> HBase Root Directory hdfs://00.hadoop.****:54310/hbase
>>>>>>
>>>>>> Load average
>>>>>> 103.0
>>>>>> Average number of regions per regionserver. Naive computation.
>>>>>> Regions On FS
>>>>>> 577
>>>>>>
>>>>>>
>>>>>> (This is after recently restarting 04-06)
>>>>>>
>>>>>>
>>>>>> Address Start Code Load
>>>>>> 01.hadoop:60030 1291268910638 requests=223, regions=118,
>>>> usedHeap=519, maxHeap=987
>>>>>> 03.hadoop:60030 1291269219610 requests=23, regions=111, usedHeap=862,
>>>> maxHeap=987
>>>>>> 04.hadoop:60030 1291910665912 requests=169, regions=82, usedHeap=194,
>>>> maxHeap=888
>>>>>> 05.hadoop:60030 1291909584060 requests=232, regions=110,
>>>> usedHeap=477, maxHeap=888
>>>>>> 06.hadoop60030 1291909723787 requests=99, regions=94, usedHeap=394,
>>>> maxHeap=888
>>>>>> Total:
>>>>>> servers: 5
>>>>>> requests=746, regions=515
>>>>>>
>>>>>>
>>>>>>
>>>>>> Following are various logs around the time of the failures for 04-06
>>>>>>
>>>>>>
>>>> ************************************************************************************
>>>>>> 04.hadoop
>>>>>>
>>>> ************************************************************************************
>>>>>> REGION SERVER ->
>>>>>> 2010-12-09 05:01:58,640 INFO
>>>> org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region
>>>> article,a83858a08f2270d319f75a7b43c756c2453988e7,1291811929868.58f6d9fc80
>>>>>> c78f3ca490b0280b4f1226. in 0sec
>>>>>> 2010-12-09 05:04:30,898 DEBUG
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.46 MB,
>>>> free=176.33 MB, max=177.79 MB, blocks=0, accesses=200583, hits=0, hitRatio=
>>>>>> 0.00%%, evictions=0, evicted=0, evictedPerRun=NaN
>>>>>> 2010-12-09 05:04:59,612 INFO
>>>> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs
>>>> -- HDFS-200
>>>>>> 2010-12-09 05:04:59,618 INFO
>>>> org.apache.hadoop.hbase.regionserver.wal.HLog: Roll
>>>> /hbase/.logs/04.hadoop.****,60020,1291858770125/10.100.154.103%3A60020.1291870585253,
>>>> entries=5318
>>>>>> 6, filesize=63776311. New hlog
>>>> /hbase/.logs/04.hadoop.****,60020,1291858770125/10.100.154.103
>>>> %3A60020.1291871099582
>>>>>> 2010-12-09 05:08:02,033 DEBUG
>>>> org.apache.hadoop.hbase.regionserver.wal.HLog: Found 1 hlogs to remove out
>>>> of total 6; oldest outstanding sequenceid is 32891801 from region article
>>>>>>
>>>> ,4b2039b791e894dd479b90661ca97087f61645d3,1291813201039.7dff47c63bb5648bb1ee6670c60553a5.
>>>>>> 2010-12-09 05:08:02,034 WARN
>>>> org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler 13 on
>>>> 60020 took 182416ms appending an edit to hlog; editcount=0
>>>>>> 2010-12-09 05:08:02,066 INFO
>>>> org.apache.hadoop.hbase.regionserver.wal.HLog: moving old hlog file
>>>> /hbase/.logs/04.hadoop.****,60020,1291858770125/10.100.154.103
>>>> %3A60020.12918679476
>>>>>> 65 whose highest sequenceid is 32880227 to /hbase/.oldlogs/
>>>> 10.100.154.103%3A60020.1291867947665
>>>>>> 2010-12-09 05:08:02,066 INFO org.apache.zookeeper.ClientCnxn: Client
>>>> session timed out, have not heard from server in 182936ms for sessionid
>>>> 0x12ca58c08ff0048, closing socket conn
>>>>>> ection and attempting reconnect
>>>>>> 2010-12-09 05:08:02,089 FATAL
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Aborting region server
>>>> serverName=04.hadoop.****,60020,1291858770125, load=(requests=24, regions=
>>>>>> 64, usedHeap=548, maxHeap=888): Unhandled exception
>>>>>> org.apache.hadoop.hbase.YouAreDeadException:
>>>> org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected;
>>>> currently processing 04.hadoop.****,60020,1291858770125 as dead s
>>>>>> erver
>>>>>> at
>>>> org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:217)
>>>>>> at
>>>> org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:271)
>>>>>> at
>>>> org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:744)
>>>>>> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>> at
>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:561)
>>>>>> at
>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>>>>>>
>>>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>>> Method)
>>>>>> at
>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>>>>>> at
>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>>>>>> at
>>>> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
>>>>>> at
>>>> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
>>>>>> at
>>>> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
>>>>>> at
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.doRun(HRegionServer.java:635)
>>>>>> at
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.access$000(HRegionServer.java:126)
>>>>>> at
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer$1.run(HRegionServer.java:518)
>>>>>> at
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer$1.run(HRegionServer.java:516)
>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>> at javax.security.auth.Subject.doAs(Subject.java:337)
>>>>>> at
>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1046)
>>>>>> at
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:516)
>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>> 2010-12-09 05:08:02,090 INFO
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
>>>> request=0.0, regions=64, stores=64, storefiles=136, storefileIndexSize=27,
>>>> memsto
>>>>>> reSize=353, compactionQueueSize=0, usedHeap=549, maxHeap=888,
>>>> blockCacheSize=1530552, blockCacheFree=184893160, blockCacheCount=0,
>>>> blockCacheHitRatio=0
>>>>>> 2010-12-09 05:08:02,090 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
>>>> server on 60020
>>>>>> 2010-12-09 05:08:02,090 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>>> Server handler 0 on 60020: exiting2010-12-09 05:08:02,090 INFO
>>>> org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server listener on 60020
>>>>>>
>>>>>>
>>>>>> MASTER HBASE ->
>>>>>>
>>>>>> 2010-12-09 05:05:06,072 INFO org.apache.hadoop.hbase.master.BaseScanner:
>>>> All 1 .META. region(s) scanned
>>>>>> 2010-12-09 05:05:40,004 INFO
>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>> 04.hadoop.jive,60020,1291858770125 znode expired
>>>>>> 2010-12-09 05:05:40,005 DEBUG
>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>> Added=04.hadoop.jive,60020,1291858770125 to dead servers, added shutdown
>>>> processing operation
>>>>>> 2010-12-09 05:05:40,005 DEBUG
>>>> org.apache.hadoop.hbase.master.RegionServerOperationQueue: Processing todo:
>>>> ProcessServerShutdown of 04.hadoop.jive,60020,12918587701252010-12-09
>>>> 05:05:40,005 INFO org.apache.hadoop.hbase.master.RegionServerOperation:
>>>> Process shutdown of server 04.hadoop.jive,60020,1291858770125: logSplit:
>>>> false, rootRescanned: f
>>>>>> alse, numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
>>>>>> 2010-12-09 05:05:40,008 INFO
>>>> org.apache.hadoop.hbase.regionserver.wal.HLog: Splitting 7 hlog(s) in
>>>> hdfs://00.hadoop.jive:54310/hbase/.logs/04.hadoop.jive,60020,12918587701252010-12-09
>>>> 05:05:40,008 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: Splitting
>>>> hlog 1 of 7:
>>>> hdfs://00.hadoop.jive:54310/hbase/.logs/04.hadoop.jive,60020,1291858770125/10.1
>>>>>> 00.154.103%3A60020.1291867947665, length=637577092010-12-09 05:05:40,008
>>>> INFO org.apache.hadoop.hbase.util.FSUtils: Recovering
>>>> filehdfs://00.hadoop.jive:54310/hbase/.logs/04.hadoop.jive,60020,1291858770125/
>>>> 10.100.154.103%3A60020
>>>>>> .1291867947665
>>>>>>
>>>>>>
>>>>>> NAMENODE ->
>>>>>>
>>>>>> 2010-12-09 05:08:02,471 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
>>>> NameSystem.addStoredBlock: blockMap updated: 10.100.159.13:50010 is added
>>>> to blk_1531008743226086399_251615 size 63757709
>>>>>> 2010-12-09 05:08:02,473 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>> handler 9 on 54310, call
>>>> complete(/hbase/.logs/04.hadoop.jive,60020,1291858770125/10.100.154.103%3A60020.1291867947665,
>>>> DFSClient_hb_m_10.194.194.79:60000_1291788452343) from 10.194.194.79:44117:
>>>> error: org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No
>>>> lease on /hbase/.logs/04.hadoop.jive,60020,1291858770125/10.100.154.103%3A60020.1291867947665
>>>> File does not exist. [Lease. Holder:
>>>> DFSClient_hb_m_10.194.194.79:60000_1291788452343, pendingcreates: 1]
>>>>>> org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease
>>>> on /hbase/.logs/04.hadoop.jive,60020,1291858770125/10.100.154.103%3A60020.1291867947665
>>>> File does not exist. [Lease. Holder:
>>>> DFSClient_hb_m_10.194.194.79:60000_1291788452343, pendingcreates: 1]
>>>>>> at
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1488)
>>>>>> at
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1479)
>>>>>> at
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:1534)
>>>>>> at
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:1522)
>>>>>> at
>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.complete(NameNode.java:610)
>>>>>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>>>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:528)
>>>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1319)
>>>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1315)
>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>>>>> at
>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
>>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1313)
>>>>>> 2010-12-09 05:08:04,206 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
>>>> NameSystem.addStoredBlock: blockMap updated: 10.99.97.106:50010 is added
>>>> to blk_-734280257049179934_251614 size 7330
>>>>>>
>>>>>> DATANODE ->
>>>>>>
>>>>>> 2010-12-09 05:08:02,212 WARN
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in
>>>> BlockReceiver.lastNodeRun: java.io.IOException: Broken pipe
>>>>>> at sun.nio.ch.FileDispatcher.write0(Native Method)
>>>>>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
>>>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100) at
>>>> sun.nio.ch.IOUtil.write(IOUtil.java:71)
>>>>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
>>>>>> at
>>>> org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
>>>> at
>>>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
>>>> at
>>>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
>>>>>> at
>>>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
>>>> at java.io.DataOutputStream.writeLong(DataOutputStream.java:207)
>>>>>> at
>>>> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.write(DataTransferProtocol.java:133)
>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.lastDataNodeRun(BlockReceiver.java:840)
>>>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:870)
>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>> 2010-12-09 05:08:02,213 WARN
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: checkDiskError: exception:
>>>>>> 2010-12-09 05:08:02,213 WARN
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: checkDiskError: exception:
>>>> java.io.IOException: Broken pipe
>>>>>> at sun.nio.ch.FileDispatcher.write0(Native Method)
>>>>>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
>>>>>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100)
>>>>>> at sun.nio.ch.IOUtil.write(IOUtil.java:71)
>>>>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
>>>>>> at
>>>> org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
>>>>>> at
>>>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
>>>>>> at
>>>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
>>>>>> at
>>>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
>>>>>> at java.io.DataOutputStream.writeLong(DataOutputStream.java:207)
>>>>>> at
>>>> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.write(DataTransferProtocol.java:133)
>>>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.lastDataNodeRun(BlockReceiver.java:840)
>>>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:870)
>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>> 2010-12-09 05:08:02,221 INFO
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder
>>>> blk_-8817504198034990390_251613 0 Exception java.io.IOException: Broken pipe
>>>>>> at sun.nio.ch.FileDispatcher.write0(Native Method)
>>>>>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
>>>>>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100)
>>>>>> at sun.nio.ch.IOUtil.write(IOUtil.java:71)
>>>>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
>>>>>> at
>>>> org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
>>>>>> at
>>>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
>>>>>> at
>>>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
>>>>>> at
>>>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
>>>>>> at java.io.DataOutputStream.writeLong(DataOutputStream.java:207)
>>>>>> at
>>>> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.write(DataTransferProtocol.java:133)
>>>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.lastDataNodeRun(BlockReceiver.java:840)
>>>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:870)
>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> 2010-12-09 05:08:02,221 INFO
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block
>>>> blk_-8817504198034990390_251613 terminating
>>>>>> 2010-12-09 05:08:02,224 INFO
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock
>>>> blk_-8817504198034990390_251613 received exception java.io.IOException:
>>>> Connection reset by peer
>>>>>> 2010-12-09 05:08:02,224 INFO
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block
>>>> blk_4505306716668305567_250752 file
>>>> /dist1/data/hadoop-data/current/subdir19/blk_4505306716668305567 for
>>>> deletion
>>>>>> 2010-12-09 05:08:02,226 ERROR
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
>>>> 10.100.154.103:50010,
>>>> storageID=DS-866555524-10.100.154.103-50010-1291262739506, infoPort=50075,
>>>> ipcPort=50020):DataXceiver
>>>>>> java.io.IOException: Connection reset by peer
>>>>>> at sun.nio.ch.FileDispatcher.read0(Native Method)
>>>>>> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
>>>>>> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:237)
>>>>>> at sun.nio.ch.IOUtil.read(IOUtil.java:210)
>>>>>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
>>>>>> at
>>>> org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:55)
>>>>>> at
>>>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
>>>>>> at
>>>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>>>>> at
>>>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>>>>> at
>>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
>>>>>> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
>>>>>> at java.io.DataInputStream.read(DataInputStream.java:132)
>>>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:267)
>>>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:357)
>>>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:378)
>>>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:534)
>>>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:417)
>>>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:122)
>>>>>> 2010-12-09 05:08:02,226 INFO
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block
>>>> blk_1422641301942366074_250732 at file
>>>> /dist1/data/hadoop-data/current/subdir19/blk_1422641301942366074
>>>>>> 2010-12-09 05:08:02,227 INFO
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block
>>>> blk_4505306716668305567_250752 at file
>>>> /dist1/data/hadoop-data/current/subdir19/blk_4505306716668305567
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>> ************************************************************************************
>>>>>> 05.hadoop
>>>>>>
>>>> ************************************************************************************
>>>>>> REGION SERVER ->
>>>>>> 2010-12-09 04:29:25,047 DEBUG
>>>> org.apache.hadoop.hbase.regionserver.Store: Compaction size of data: 250.5m;
>>>> Skipped 1 file(s), size: 206896367
>>>>>> 2010-12-09 04:29:25,047 INFO org.apache.hadoop.hbase.regionserver.Store:
>>>> Started compaction of 2 file(s) in data of
>>>> article,060c4b75067378c045eead9edec2d4fb7d8f3e5e,1291812563166.95e92e97823e11e57850720a63433a62.
>>>> into
>>>> hdfs://00.hadoop.****:54310/hbase/article/95e92e97823e11e57850720a63433a62/.tmp,
>>>> sequenceid=33050221
>>>>>> 2010-12-09 04:29:26,328 INFO org.apache.hadoop.hbase.regionserver.Store:
>>>> Completed compaction of 2 file(s) in data of
>>>> article,060c4b75067378c045eead9edec2d4fb7d8f3e5e,1291812563166.95e92e97823e11e57850720a63433a62.;
>>>> new storefile is
>>>> hdfs://00.hadoop.****:54310/hbase/article/95e92e97823e11e57850720a63433a62/data/2787768957794924068;
>>>> store size is 250.5m
>>>>>> 2010-12-09 04:29:26,328 INFO
>>>> org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region
>>>> article,060c4b75067378c045eead9edec2d4fb7d8f3e5e,1291812563166.95e92e97823e11e57850720a63433a62.
>>>> in 1sec
>>>>>> 2010-12-09 04:29:26,328 INFO
>>>> org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on region
>>>> article,ed0cdc7ae6db73432e2ca47c973d545bee11ddd8,1291813400028.85a8d31d099f409dd36f3bcc4be1eba6.
>>>>>> 2010-12-09 04:29:26,330 INFO
>>>> org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region
>>>> article,ed0cdc7ae6db73432e2ca47c973d545bee11ddd8,1291813400028.85a8d31d099f409dd36f3bcc4be1eba6.
>>>> in 0sec
>>>>>> 2010-12-09 04:32:33,081 INFO
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGIONSERVER_STOP
>>>>>> 2010-12-09 04:32:33,081 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
>>>> server on 60020
>>>>>> 2010-12-09 04:32:33,089 WARN org.apache.hadoop.hdfs.DFSClient:
>>>> DFSOutputStream ResponseProcessor exception for block
>>>> blk_2012253533858803341_250248java.io.EOFException
>>>>>> at java.io.DataInputStream.readFully(DataInputStream.java:180)
>>>>>> at java.io.DataInputStream.readLong(DataInputStream.java:399)
>>>>>> at
>>>> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:120)
>>>>>> at
>>>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2621)
>>>>>>
>>>>>> 2010-12-09 04:32:33,089 INFO org.apache.zookeeper.ClientCnxn: Unable to
>>>> read additional data from server sessionid 0x12ca58c08ff0049, likely server
>>>> has closed socket, closing sock
>>>>>> et connection and attempting reconnect
>>>>>> 2010-12-09 04:32:33,089 WARN
>>>> org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler 5 on 60020
>>>> took 75446ms appending an edit to hlog; editcount=50505
>>>>>> 2010-12-09 04:32:33,090 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>>>> Server handler 18 on 60020: exiting
>>>>>> 2010-12-09 04:32:33,139 INFO org.apache.hadoop.hdfs.DFSClient: Error
>>>> Recovery for block blk_2012253533858803341_250248 waiting for responder to
>>>> exit.
>>>>>> 2010-12-09 04:32:33,139 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>>>> Server handler 8 on 60020: exiting
>>>>>> 2010-12-09 04:32:33,139 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>>> Server handler 1 on 60020: exiting
>>>>>> 2010-12-09 04:32:33,139 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>>>> Server handler 10 on 60020: exiting
>>>>>> 2010-12-09 04:32:33,140 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>>> Server handler 0 on 60020: exiting2010-12-09 04:32:33,140 INFO
>>>> org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 5 on 60020:
>>>> exiting
>>>>>> 2010-12-09 04:32:33,139 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>>> Server handler 8 on 60020: exiting2010-12-09 04:32:33,140 INFO
>>>> org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 4 on 60020:
>>>> exiting
>>>>>> 2010-12-09 04:32:33,140 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>>> Server handler 3 on 60020: exiting2010-12-09 04:32:33,142 INFO
>>>> org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 7 on 60020:
>>>> exiting
>>>>>> 2010-12-09 04:32:33,142 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>>> Server handler 9 on 60020: exiting2010-12-09 04:32:33,143 INFO
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
>>>>>> 2010-12-09 04:32:33,143 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>>> Server handler 2 on 60020: exiting
>>>>>> 2010-12-09 04:32:33,143 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
>>>> IPC Server Responder
>>>>>> 2010-12-09 04:32:33,143 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>>> Server handler 6 on 60020: exiting
>>>>>> 2010-12-09 04:32:33,144 INFO org.mortbay.log: Stopped
>>>> SelectChannelConnector@0.0.0.0:60030
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> NAMENODE ->
>>>>>> 2010-12-09 04:32:08,408 WARN org.apache.hadoop.hdfs.StateChange: DIR*
>>>> NameSystem.startFile: failed to create file
>>>> /hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555
>>>> for DFSClient_hb_m_10.194.194.79:60000_1291788452343 on client
>>>> 10.194.194.79, because this file is already being created by NN_Recovery on
>>>> 10.100.159.13
>>>>>> 2010-12-09 04:32:08,408 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>> handler 6 on 54310, call
>>>> append(/hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555,
>>>> DFSClient_hb_m_10.194.194.79:60000_1291788452343) from 10.194.194.79:59118:
>>>> error: org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed
>>>> to create file /hba
>>>>>> se/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555
>>>> for DFSClient_hb_m_10.194.194.79:60000_1291788452343 on client
>>>> 10.194.194.79, because this file is already being created by NN_Recovery on
>>>> 10.100.159.13
>>>>>> org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to
>>>> create file /hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555
>>>> for DFSClient_hb_m_10.194.194.79:60000_1291788452343 on client
>>>> 10.194.194.79, because this file is already being created by NN_Recovery on
>>>> 10.100.159.13
>>>>>> at
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1194)
>>>> at
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:1282)
>>>>>> at
>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.append(NameNode.java:541)
>>>> at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
>>>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:528) at
>>>> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1319)
>>>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1315)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>> at javax.security.auth.Subject.doAs(Subject.java:396) at
>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
>>>>>> at
>>>> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1313)2010-12-09
>>>> 04:32:09,411 WARN org.apache.hadoop.hdfs.StateChange: DIR*
>>>> NameSystem.startFile: failed to create file
>>>> /hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A6
>>>>>> 0020.1291868481555 for DFSClient_hb_m_10.194.194.79:60000_1291788452343
>>>> on client 10.194.194.79, because this file is already being created by
>>>> NN_Recovery on 10.100.159.132010-12-09 04:32:09,411 INFO
>>>> org.apache.hadoop.ipc.Server: IPC Server handler 9 on 54310, call
>>>> append(/hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13
>>>> %3A60020.1291868
>>>>>> 481555, DFSClient_hb_m_10.194.194.79:60000_1291788452343) from
>>>> 10.194.194.79:59118: error:
>>>> org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to
>>>> create file /hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555
>>>> for DFSClient_hb_m_10.194.194.79:60000_1291788452343 on client
>>>> 10.194.194.79, because this file is
>>>>>> already being created by NN_Recovery on
>>>> 10.100.159.13org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException:
>>>> failed to create file /hbase/.logs/05.hadoop.jive,60020,1291862294517/
>>>> 10.100.159.13%3A60020.1291868481555 for DFSClie
>>>>>> nt_hb_m_10.194.194.79:60000_1291788452343 on client 10.194.194.79,
>>>> because this file is already being created by NN_Recovery on 10.100.159.13
>>>> at
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1194)
>>>>>> at
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:1282)
>>>> at
>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.append(NameNode.java:541)
>>>>>> at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:597) at
>>>> org.apache.hadoop.ipc.RPC$Server.call(RPC.java:528)
>>>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1319)
>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1315)
>>>>>>
>>>>>>
>>>>>>
>>>>>> DATANODE
>>>>>>
>>>>>> 2010-12-09 04:32:09,534 INFO
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder
>>>> blk_2012253533858803341_250248 1 : Thread is interrupted.
>>>>>> 2010-12-09 04:32:09,534 ERROR
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
>>>> 10.100.159.13:50010,
>>>> storageID=DS-145997007-10.100.159.13-50010-1291262743182, infoPort=50075,
>>>> ipcPort=50020):DataXceiver
>>>>>> java.io.IOException: Interrupted receiveBlock
>>>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:579)
>>>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:417)
>>>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:122)
>>>>>> 2010-12-09 04:32:09,534 INFO
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 1 for block
>>>> blk_2012253533858803341_250248 terminating
>>>>>>
>>>>>>
>>>>>>
>>>> ************************************************************************************
>>>>>> 06.hadoop
>>>>>>
>>>> ************************************************************************************
>>>>>> REGION SERVER ->
>>>>>>
>>>>>> 2010-12-09 06:17:47,215 DEBUG
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>>> NotServingRegionException;
>>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>>> 2010-12-09 06:17:47,430 DEBUG
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>>> NotServingRegionException;
>>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>>> 2010-12-09 06:17:47,549 DEBUG
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>>> NotServingRegionException;
>>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>>> 2010-12-09 06:17:47,557 DEBUG
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>>> NotServingRegionException;
>>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>>> 2010-12-09 06:17:47,563 DEBUG
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>>> NotServingRegionException;
>>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>>> 2010-12-09 06:17:48,619 DEBUG
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>>> NotServingRegionException;
>>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>>> 2010-12-09 06:17:48,619 DEBUG
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>>> NotServingRegionException;
>>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>>> 2010-12-09 06:17:48,619 DEBUG
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>>> NotServingRegionException;
>>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>>> 2010-12-09 06:17:48,619 DEBUG
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>>> NotServingRegionException;
>>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>>> 2010-12-09 06:17:48,619 DEBUG
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>>> NotServingRegionException;
>>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>>> 2010-12-09 06:17:49,703 DEBUG
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>>> NotServingRegionException;
>>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>>> 2010-12-09 06:17:49,703 DEBUG
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>>> NotServingRegionException;
>>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>>> 2010-12-09 06:17:49,703 DEBUG
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>>> NotServingRegionException;
>>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>>> 2010-12-09 06:17:49,705 DEBUG
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>>> NotServingRegionException;
>>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>>> 2010-12-09 06:17:49,729 DEBUG
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>>> NotServingRegionException;
>>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>>>
>>>>>> A bunch of these ^
>>>>>>
>>>>>> Then a little later::
>>>>>> ....
>>>>>>
>>>>>> 2010-12-09 06:23:23,561 DEBUG
>>>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction
>>>> requested for region
>>>> article,993143fca3613014990cc6975cacdd62f4147e03,12918133247
>>>>>> 53.19413f33fc660536d9d4f315a1283440. because:
>>>> regionserver60020.cacheFlusher
>>>>>> 2010-12-09 06:23:27,058 INFO
>>>> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs
>>>> -- HDFS-200
>>>>>> 2010-12-09 06:23:27,080 INFO
>>>> org.apache.hadoop.hbase.regionserver.wal.HLog: Roll
>>>> /hbase/.logs/06.hadoop.****,60020,1291864620851/10.99.97.106%3A60020.1291875047947,
>>>> entries=53315,
>>>>>> filesize=63781925. New hlog
>>>> /hbase/.logs/06.hadoop.****,60020,1291864620851/10.99.97.106
>>>> %3A60020.1291875807006
>>>>>> 2010-12-09 06:23:27,080 DEBUG
>>>> org.apache.hadoop.hbase.regionserver.wal.HLog: Found 1 hlogs to remove out
>>>> of total 8; oldest outstanding sequenceid is 33526881 from region article
>>>>>>
>>>> ,1221b0013c61fd7446cb822f698d0bcafa487342,1291872610788.bb2aa6ae8c694a096a00e7bb91fc46c3.
>>>>>> 2010-12-09 06:23:27,080 INFO
>>>> org.apache.hadoop.hbase.regionserver.wal.HLog: moving old hlog file
>>>> /hbase/.logs/06.hadoop.****,60020,1291864620851/10.99.97.106
>>>> %3A60020.1291872064424
>>>>>> whose highest sequenceid is 33506639 to /hbase/.oldlogs/10.99.97.106
>>>> %3A60020.1291872064424
>>>>>> 2010-12-09 06:25:39,162 WARN
>>>> org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler 12 on
>>>> 60020 took 119034ms appending an edit to hlog; editcount=3683
>>>>>> 2010-12-09 06:25:39,162 WARN org.apache.hadoop.hdfs.DFSClient:
>>>> DataStreamer Exception: java.io.IOException: Broken pipe
>>>>>> at sun.nio.ch.FileDispatcher.write0(Native Method)
>>>>>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
>>>>>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100)
>>>>>> at sun.nio.ch.IOUtil.write(IOUtil.java:71)
>>>>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
>>>>>> at
>>>> org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
>>>>>> at
>>>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
>>>>>> at
>>>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
>>>>>> at
>>>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
>>>>>> at
>>>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
>>>>>> at java.io.DataOutputStream.write(DataOutputStream.java:90)
>>>>>> at
>>>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2509)
>>>>>> 2010-12-09 06:25:39,163 WARN org.apache.hadoop.hdfs.DFSClient:
>>>> DFSOutputStream ResponseProcessor exception for block
>>>> blk_1141147913164105086_258331java.io.EOFException
>>>>>> at java.io.DataInputStream.readFully(DataInputStream.java:180)
>>>>>> at java.io.DataInputStream.readLong(DataInputStream.java:399)
>>>>>> at
>>>> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:120)
>>>>>> at
>>>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2621)
>>>>>>
>>>>>> 2010-12-09 06:25:39,163 INFO org.apache.hadoop.hdfs.DFSClient: Error
>>>> Recovery for block blk_1141147913164105086_258331 waiting for responder to
>>>> exit.
>>>>>> 2010-12-09 06:25:39,164 WARN org.apache.hadoop.hdfs.DFSClient:
>>>> DFSOutputStream ResponseProcessor exception for block
>>>> blk_-7076574401277381592_258324java.io.EOFException
>>>>>> at java.io.DataInputStream.readFully(DataInputStream.java:180)
>>>>>> at java.io.DataInputStream.readLong(DataInputStream.java:399)
>>>>>> at
>>>> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:120)
>>>>>> at
>>>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2621)
>>>>>>
>>>>>> 2010-12-09 06:25:39,164 WARN org.apache.hadoop.hdfs.DFSClient: Error
>>>> Recovery for block blk_-7076574401277381592_258324 bad datanode[0]
>>>> 10.99.97.106:50010
>>>>>> 2010-12-09 06:25:39,164 INFO org.apache.zookeeper.ClientCnxn: Unable to
>>>> read additional data from server sessionid 0x12ca58c08ff004b, likely server
>>>> has closed socket, closing socket connection and attempting reconnect
>>>>>> 2010-12-09 06:25:39,164 WARN org.apache.hadoop.hdfs.DFSClient: Error
>>>> Recovery for block blk_-7076574401277381592_258324 in pipeline
>>>> 10.99.97.106:50010, 10.100.154.103:50010: bad datanode 10.99.97.106:50010
>>>>>> 2010-12-09 06:25:39,165 WARN org.apache.hadoop.hdfs.DFSClient: Error
>>>> Recovery for block blk_1141147913164105086_258331 bad datanode[0]
>>>> 10.99.97.106:50010
>>>>>> 2010-12-09 06:25:39,167 INFO org.apache.zookeeper.ClientCnxn: Unable to
>>>> read additional data from server sessionid 0x12ca58c08ff004c, likely server
>>>> has closed socket, closing socket connection and attempting reconnect
>>>>>> 2010-12-09 06:25:39,169 INFO org.apache.hadoop.ipc.Client: Retrying
>>>> connect to server: /10.100.154.103:50020. Already tried 0 time(s).
>>>>>> 2010-12-09 06:25:39,169 WARN org.apache.hadoop.hdfs.DFSClient: Error
>>>> Recovery for block blk_1141147913164105086_258331 in pipeline
>>>> 10.99.97.106:50010, 10.100.159.13:50010: bad datanode 10.99.97.106:50010
>>>>>> 2010-12-09 06:25:39,209 INFO
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGIONSERVER_STOP
>>>>>> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
>>>> server on 60020
>>>>>> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>>>> Server handler 3 on 60020: exiting
>>>>>> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>>>> Server handler 9 on 60020: exiting
>>>>>> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>>>> Server handler 24 on 60020: exiting
>>>>>> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>>> Server handler 1 on 60020: exiting
>>>>>> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>>>> Server handler 15 on 60020: exiting
>>>>>> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>>> Server handler 0 on 60020: exiting
>>>>>> 2010-12-09 06:25:39,214 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>>> Server handler 3 on 60020: exiting
>>>>>> 2010-12-09 06:25:39,214 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
>>>> IPC Server listener on 60020
>>>>>> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>>> Server handler 2 on 60020: exiting
>>>>>> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>>> Server handler 7 on 60020: exiting
>>>>>> 2010-12-09 06:25:39,213 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>>>> Server handler 21 on 60020: exiting
>>>>>> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>>> Server handler 4 on 60020: exiting
>>>>>> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
>>>> IPC Server Responder
>>>>>> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>>> Server handler 5 on 60020: exiting
>>>>>> 2010-12-09 06:25:39,216 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>>> Server handler 8 on 60020: exiting
>>>>>> 2010-12-09 06:25:39,216 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>>> Server handler 9 on 60020: exiting
>>>>>> 2010-12-09 06:25:39,216 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>>> Server handler 6 on 60020: exiting
>>>>>> 2010-12-09 06:25:39,216 INFO
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
>>>>>> 2010-12-09 06:25:39,223 WARN org.apache.hadoop.hdfs.DFSClient: Failed
>>>> recovery attempt #0 from primary datanode 10.100.159.13:50010
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>
>
>
Re: Crashing Region Servers
Posted by Lance Riedel <la...@gmail.com>.
Still having issues, so taking a step back here for some scale questions:
We are doing:
- inserts at the rate of up to 1k/sec with each entry just under 1k (roughly)
- roughly 60-100 megs/min - 5-10 gigs/day/ 100 million records per day
- rolling 30 days of data (so will kill off a days worth of data per day)
~ 300gigs of raw data per month (maybe some more, but not orders more)
What size of machines / number do you think we need?
Here is what we have been using with lots of issues for just 1 weeks worth of data:
-4 compute.xlarge = each with 4 cpus, 16 gigs ram
- Each regionserver is getting 6 gigs allocated
Problem we are having:
Region server crashes
Summary of problem:
We get a region server crashing after about 12 hours of runtime. The only load really is the inserts as described above. All the logs (master, region, datanode) are below during the hangup.
Where it gets weird:
The only way to startup the region server is to turn off concurrent sweep GC, otherwise it just dies on startup (see 2nd set of logs at bottom)
Below are 2 sets of logs, (Dies after 12 hours, and Dies on startup) - only difference is GC params
**********************************************************
DIES AFTER 12 HOURS LOGS:
Non Concurrent Sweep Death Logs
Runs for up to 12 hours, then here are the various logs when it dies
export HBASE_OPTS="$HBASE_OPTS -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:$HBASE_HOME/logs/gc-hbase.log"
**********************************************************
///////////REGION SERVER
2010-12-20 11:58:32,252 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Roll /hbase/.logs/01.hbase.jive,60020,1292805088631/10.101.25.203%3A6002
0.1292845076146, entries=53328, filesize=63945980. New hlog /hbase/.logs/01.hbase.jive,60020,1292805088631/10.101.25.203%3A60020.1292846312039
2010-12-20 11:58:32,252 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: Found 2 hlogs to remove out of total 34; oldest outstanding sequenceid
is 57439916 from region article,6addb2ace7f12175eb8ec92d16faa26f7592c64e,1292731695023.ec7bc70e76c6f4fc3b0779c412680375.
2010-12-20 11:58:32,252 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: moving old hlog file /hbase/.logs/01.hbase.jive,60020,1292805088631/10.1
01.25.203%3A60020.1292826985598 whose highest sequenceid is 57337628 to /hbase/.oldlogs/10.101.25.203%3A60020.1292826985598
2010-12-20 11:58:32,253 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: moving old hlog file /hbase/.logs/01.hbase.jive,60020,1292805088631/10.1
01.25.203%3A60020.1292827095006 whose highest sequenceid is 57390617 to /hbase/.oldlogs/10.101.25.203%3A60020.1292827095006
2010-12-20 11:59:24,061 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler 3 on 60020 took 5172ms appending an edit to hlog; edi
tcount=29066
****NOTE 14 minute gap here!!!! *********
2010-12-20 12:12:55,016 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=32.86 MB, free=1.01 GB, max=1.04 GB, blocks=1836, accesses=1250468, hits=775218, hitRatio=61.99%%, evictions=0, evicted=0, evictedPerRun=NaN
2010-12-20 12:12:55,016 WARN org.apache.hadoop.hdfs.DFSClient: DFSOutputStream ResponseProcessor exception for block blk_-6892804755545789592_78455java.net.SocketTimeoutException: 66000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.101.25.203:57216 remote=/10.101.25.203:50010]
2010-12-20 12:12:55,017 INFO org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_-6892804755545789592_78455 waiting for responder to exit.
2010-12-20 12:12:55,044 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 784310ms for sessionid 0x12ce6e6154d00a3, closing socket connection and attempting reconnect
2010-12-20 12:12:55,062 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_-6892804755545789592_78455 bad datanode[0] 10.101.25.203:50010
2010-12-20 12:12:55,062 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_-6892804755545789592_78455 in pipeline 10.101.25.203:50010, 10.124.189.106:50010: bad datanode 10.101.25.203:50010
2010-12-20 12:12:55,089 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: HDFS pipeline error detected. Found 1 replicas but expecting 2 replicas. Requesting close of hlog.
2010-12-20 12:12:55,148 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs -- HDFS-200
2010-12-20 12:12:55,155 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event, state: Disconnected, type: None, path: null
2010-12-20 12:12:55,155 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event, state: Disconnected, type: None, path: null
2010-12-20 12:12:55,175 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: Aborting region server serverName=01.hbase.jive,60020,1292805088631, load=(requests=486, regions=317, usedHeap=2185, maxHeap=5333): Unhandled exception
2010-12-20 12:12:55,176 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: request=0.0, regions=317, stores=318, storefiles=679, storefileIndexSize=88, memstoreSize=2096, compactionQueueSize=0, usedHeap=2192, maxHeap=5333, blockCacheSize=34460320, blockCacheFree=1084029536, blockCacheCount=1836, blockCacheHitRatio=61
2010-12-20 12:12:55,176 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60020
///////////////////MASTER
2010-12-20 12:12:55,022 DEBUG org.apache.hadoop.hbase.master.ServerManager: Server REPORT rejected; currently processing 01.hbase.jive,60020,1292805088631 as dead server
2010-12-20 12:12:55,761 INFO org.apache.hadoop.hbase.util.FSUtils: Finished lease recover attempt for hdfs://00.hbase.jive:54310/hbase/.logs/01.hbase.jive,60020,1292805088631/10.101.25.203%3A60020.1292827812330
2010-12-20 12:12:56,824 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: Pushed=52893 entries from hdfs://00.hbase.jive:54310/hbase/.logs/01.hbase.jive,60020,1292805088631/10.101.25.203%3A60020.1292827812330
2010-12-20 12:12:56,825 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: Splitting hlog 3 of 33: hdfs://00.hbase.jive:54310/hbase/.logs/01.hbase.jive,60020,1292805088631/10.101.25.203%3A60020.1292828346328, length=63757023
2010-12-20 12:12:56,825 INFO org.apache.hadoop.hbase.util.FSUtils: Recovering filehdfs://00.hbase.jive:54310/hbase/.logs/01.hbase.jive,60020,1292805088631/10.101.25.203%3A60020.1292828346328
2010-12-20 12:12:57,279 INFO org.apache.hadoop.hbase.util.FSUtils: Finished lease recover attempt for hdfs://00.hbase.jive:54310/hbase/.logs/01.hbase.jive,60020,1292805088631/10.101.25.203%3A60020.1292828346328
2010-12-20 12:12:58,735 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: Pushed=52918 entries from hdfs://00.hbase.jive:54310/hbase/.logs/01.hbase.jive,60020,1292805088631/10.101.25.203%3A60020.1292828346328
////////////////DATANODE
2010-12-20 12:12:55,067 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in receiveBlock for block blk_-6892804755545789592_78455 java.io.EOFException: while trying to read 4377 bytes
2010-12-20 12:12:55,068 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder blk_-6892804755545789592_78455 1 : Thread is interrupted.
2010-12-20 12:12:55,068 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 1 for block blk_-6892804755545789592_78455 terminating
2010-12-20 12:12:55,068 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock blk_-6892804755545789592_78455 received exception java.io.EOFException: while trying to read 4377 bytes
2010-12-20 12:12:55,069 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(10.101.25.203:50010, storageID=DS-1813182679-10.101.25.203-50010-1292355638641, infoPort=50075, ipcPort=50020):DataXceiver
2010-12-20 12:12:55,414 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_-702706422692129802_78497 src: /10.101.25.249:55244 dest: /10.101.25.203:50010
2010-12-20 12:12:55,462 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_7987291438907198920_78499 src: /10.101.25.203:33387 dest: /10.101.25.203:50010
2010-12-20 12:12:55,464 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_-2741441115648159702_78500 src: /10.101.25.203:33388 dest: /10.101.25.203:50010
2010-12-20 12:12:55,486 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected: wildcard ongoingCreates exists for block blk_-702706422692129802_78496
2010-12-20 12:12:55,486 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Reopen Block for append blk_-702706422692129802_78496
2010-12-20 12:12:55,487 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in receiveBlock for block blk_-702706422692129802_78496 java.io.EOFException: while trying to read 65557 bytes
2010-12-20 12:12:55,487 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block blk_-702706422692129802_78496 Interrupted.
2010-12-20 12:12:55,487 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block blk_-702706422692129802_78496 terminating
2010-12-20 12:12:55,487 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock blk_-702706422692129802_78496 received exception java.io.EOFException: while trying to read 65557 bytes
2010-12-20 12:12:55,487 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(10.101.25.203:50010, storageID=DS-1813182679-10.101.25.203-50010-1292355638641, infoPort=50075, ipcPort=50020):DataXceiver
2010-12-20 12:12:55,489 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: oldblock=blk_287777955370148653_76327(length=63761157), newblock=blk_287777955370148653_78501(length=63761157), datanode=10.101.25.203:50010
2010-12-20 12:12:55,501 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in receiveBlock for block blk_-702706422692129802_78497 java.io.EOFException: while trying to read 65536 bytes
2010-12-20 12:12:55,501 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Block blk_-702706422692129802_78497 unfinalized and removed.
2010-12-20 12:12:55,501 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock blk_-702706422692129802_78497 received exception java.io.EOFException: while trying to read 65536 bytes
2010-12-20 12:12:55,501 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(10.101.25.203:50010, storageID=DS-1813182679-10.101.25.203-50010-1292355638641, infoPort=50075, ipcPort=50020):DataXceiver
2010-12-20 12:12:55,610 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.203:33388, dest: /10.101.25.203:50010, bytes: 5736187, op: HDFS_WRITE, cliID: DFSClient_-1650596123, offset: 0, srvID: DS-1813182679-10.101.25.203-50010-1292355638641, blockid: blk_-2741441115648159702_78500, duration: 119469848
////////////////////GC
2010-12-20T11:48:25.248+0000: 40617.905: [GC [PSYoungGen: 312017K->31408K(354880K)] 2396181K->2115572K(2563648K), 0.0727710 secs] [Times: user=0.18 sys=0.00, real=0.07 secs]
2010-12-20T11:57:47.651+0000: 41180.308: [GC [PSYoungGen: 312048K->33351K(307712K)] 2396212K->2117515K(2516480K), 0.0643350 secs] [Times: user=0.16 sys=0.01, real=0.06 secs]
2010-12-20T11:57:54.525+0000: 41187.182: [GC [PSYoungGen: 307655K->37385K(341504K)] 2391819K->2121549K(2550272K), 0.0710110 secs] [Times: user=0.17 sys=0.00, real=0.07 secs]
2010-12-20T11:58:00.157+0000: 41192.814: [GC [PSYoungGen: 305545K->26719K(289024K)] 2389709K->2127955K(2497792K), 0.0779900 secs] [Times: user=0.18 sys=0.01, real=0.08 secs]
2010-12-20T11:58:04.530+0000: 41197.188: [GC [PSYoungGen: 288991K->11195K(339968K)] 2390227K->2127423K(2548736K), 0.0531710 secs] [Times: user=0.12 sys=0.00, real=0.06 secs]
2010-12-20T11:58:15.908+0000: 41208.565: [GC [PSYoungGen: 275835K->10768K(269696K)] 2392063K->2131139K(2478464K), 0.0256330 secs] [Times: user=0.06 sys=0.00, real=0.02 secs]
2010-12-20T11:58:20.325+0000: 41212.982: [GC [PSYoungGen: 269648K->14982K(311744K)] 2390019K->2142213K(2520512K), 0.0394690 secs] [Times: user=0.07 sys=0.00, real=0.04 secs]
2010-12-20T11:58:29.751+0000: 41222.409: [GC [PSYoungGen: 268258K->7168K(255104K)] 2395489K->2141035K(2463872K), 0.0367260 secs] [Times: user=0.09 sys=0.00, real=0.03 secs]
2010-12-20T11:58:34.476+0000: 41227.133: [GC [PSYoungGen: 255067K->7424K(298624K)] 2388934K->2148539K(2507392K), 0.0379800 secs] [Times: user=0.12 sys=0.00, real=0.03 secs]
2010-12-20T11:58:38.866+0000: 41231.523: [GC [PSYoungGen: 250094K->7104K(244800K)] 2391210K->2155495K(2453568K), 0.0414990 secs] [Times: user=0.09 sys=0.00, real=0.04 secs]
2010-12-20T11:58:43.223+0000: 41235.880: [GC [PSYoungGen: 244694K->7072K(301120K)] 2393086K->2162599K(2509888K), 0.0404940 secs] [Times: user=0.08 sy
s=0.00, real=0.04 secs]
2010-12-20T11:58:47.589+0000: 41240.246: [GC [PSYoungGen: 247054K->7264K(298752K)] 2402582K->2169607K(2507520K), 0.0322390 secs] [Times: user=0.08 sy
s=0.00, real=0.04 secs]
2010-12-20T11:58:52.521+0000: 41245.178: [GC [PSYoungGen: 247208K->7200K(326016K)] 2409552K->2176615K(2534784K), 0.0366060 secs] [Times: user=0.07 sy
s=0.00, real=0.03 secs]
2010-12-20T11:58:57.931+0000: 41250.588: [GC [PSYoungGen: 281759K->8128K(328448K)] 2451174K->2184519K(2537216K), 0.0359190 secs] [Times: user=0.09 sy
s=0.00, real=0.03 secs]
2010-12-20T11:59:09.681+0000: 41262.338: [GC [PSYoungGen: 282649K->8000K(276480K)] 2459041K->2192471K(2485248K), 0.0432830 secs] [Times: user=0.11 sys=0.01, real=0.05 secs]
2010-12-20T11:59:14.136+0000: 41266.793: [GC [PSYoungGen: 276405K->7872K(324416K)] 2460876K->2200295K(2533184K), 0.0565900 secs] [Times: user=0.11 sys=0.01, real=0.06 secs]
2010-12-20T11:59:18.889+0000: 41271.546: [GC [PSYoungGen: 283525K->8288K(356544K)] 2475949K->2208559K(2565312K), 0.0390340 secs] [Times: user=0.10 sys=0.01, real=0.04 secs]
2010-12-20T11:59:18.928+0000: 41271.585: [Full GC [PSYoungGen: 8288K->0K(356544K)] [PSOldGen: 2200271K->1991701K(2358656K)] 2208559K->1991701K(2715200K) [PSPermGen: 18540K->18540K(21248K)], 5.1322070 secs] [Times: user=5.07 sys=0.00, real=5.14 secs]
2010-12-20T11:59:29.253+0000: 41281.910: [GC [PSYoungGen: 310221K->9152K(312256K)] 2301922K->2000853K(2670912K), 0.0268860 secs] [Times: user=0.06 sys=0.00, real=0.04 secs]
2010-12-20T11:59:34.610+0000: 41287.267: [GC [PSYoungGen: 312172K->8928K(325888K)] 2303873K->2009821K(2684544K), 0.0403310 secs] [Times: user=0.09 sys=0.00, real=0.05 secs]
2010-12-20T11:59:41.074+0000: 41293.731: [GC [PSYoungGen: 304982K->8544K(297920K)] 2305875K->2018269K(2656576K), 0.0468460 secs] [Times: user=0.07 sys=0.00, real=0.05 secs]
2010-12-20T11:59:47.708+0000: 41300.365: [GC [PSYoungGen: 297888K->8448K(309504K)] 2307613K->2026869K(2668160K), 0.0409660 secs] [Times: user=0.09 sys=0.00, real=0.04 secs]
Heap
PSYoungGen total 309504K, used 243942K [0x00007f5556d50000, 0x00007f556c610000, 0x00007f55d3d50000)
eden space 282880K, 83% used [0x00007f5556d50000,0x00007f55653498f0,0x00007f5568190000)
from space 26624K, 31% used [0x00007f556ac10000,0x00007f556b4500c0,0x00007f556c610000)
to space 35072K, 0% used [0x00007f5568190000,0x00007f5568190000,0x00007f556a3d0000)
PSOldGen total 2358656K, used 2018421K [0x00007f545cd50000, 0x00007f54eccb0000, 0x00007f5556d50000)
object space 2358656K, 85% used [0x00007f545cd50000,0x00007f54d806d480,0x00007f54eccb0000)
PSPermGen total 21248K, used 18595K [0x00007f5457950000, 0x00007f5458e10000, 0x00007f545cd50000)
object space 21248K, 87% used [0x00007f5457950000,0x00007f5458b78c50,0x00007f5458e10000)
**********************************************************
DIES ON STARTUP LOGS:
Non Concurrent Sweep
export HBASE_OPTS="$HBASE_OPTS -verbose:gc -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:$HBASE_HOME/logs/gc-hbase.log"
**********************************************************
///////////REGION SERVER
2010-12-20 17:25:49,020 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region article,26b5ee07c80cf31d88719b1a510726591f2b01da,12927082
91003.aae953527ae2ce697fda39699e29f70e. in 0sec
2010-12-20 17:25:49,020 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on region article,c050050bf22e8f276a8a6bd3c900b34d6600d6d0,129273169
0903.0460d68f5f4fd423750dd6bf67937b7e.2010-12-20 17:25:49,022 DEBUG org.apache.hadoop.hbase.regionserver.Store: Compaction size of data: 186.0m; Skipped 2 file(s), size: 1894568042010-12-20 17:25:49,023 INFO org.apache.hadoop.hbase.regionserver.Store: Started compaction of 2 file(s) in data of article,c050050bf22e8f276a8a6bd3c900b34d6600d6d
0,1292731690903.0460d68f5f4fd423750dd6bf67937b7e. into hdfs://00.hbase.jive:54310/hbase/article/0460d68f5f4fd423750dd6bf67937b7e/.tmp, sequenceid=63434988
2010-12-20 17:25:49,324 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Applied 0, skipped 10953, firstSequenceidInLog=52141208, maxSequenceidInLog=63429846
2010-12-20 17:25:49,327 INFO org.apache.hadoop.hbase.regionserver.HRegion: Onlined article,150bfc2bd7a0effc9f3dfa22312878bf916d68df,1292597687450.f213d3923dca3a7e3
d020cf7dbf51918.; next sequenceid=63429847
2010-12-20 17:25:49,328 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 2 file(s) in data of article,c050050bf22e8f276a8a6bd3c900b34d6600d
6d0,1292731690903.0460d68f5f4fd423750dd6bf67937b7e.; new storefile is hdfs://00.hbase.jive:54310/hbase/article/0460d68f5f4fd423750dd6bf67937b7e/data/88724363503101
19241; store size is 186.0m
2010-12-20 17:25:49,329 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region article,c050050bf22e8f276a8a6bd3c900b34d6600d6d0,12927316
90903.0460d68f5f4fd423750dd6bf67937b7e. in 0sec
2010-12-20 17:25:49,329 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on region article,933fc87674f7132b5d91ce6e9c60bc68981a56a7,129270464
1421.2551c0c46387d1479d25fb359ef4c5c9.
2010-12-20 17:25:49,336 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: article,0643e00cb1b172034b2627567d00d26913a25f97,12927089
81266.5e0ac5bd27f34da31ded307e2ea932cf.2010-12-20 17:25:49,337 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Creating region article,0643e00cb1b172034b2627567d00d26913a25f97,1292708981266.5e0ac5bd
27f34da31ded307e2ea932cf.
2010-12-20 17:25:49,338 DEBUG org.apache.hadoop.hbase.regionserver.Store: Compaction size of data: 186.5m; Skipped 2 file(s), size: 190928927
2010-12-20 17:25:49,338 INFO org.apache.hadoop.hbase.regionserver.Store: Started compaction of 2 file(s) in data of article,933fc87674f7132b5d91ce6e9c60bc68981a56a
7,1292704641421.2551c0c46387d1479d25fb359ef4c5c9. into hdfs://00.hbase.jive:54310/hbase/article/2551c0c46387d1479d25fb359ef4c5c9/.tmp, sequenceid=63434974
2010-12-20 17:25:49,362 DEBUG org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://00.hbase.jive:54310/hbase/article/5e0ac5bd27f34da31ded307e2ea932cf/data/421
7589706037747905, isReference=false, isBulkLoadResult=false, seqid=53910828, majorCompaction=true
2010-12-20 17:25:49,372 DEBUG org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://00.hbase.jive:54310/hbase/article/5e0ac5bd27f34da31ded307e2ea932cf/data/540
8931691681201713, isReference=false, isBulkLoadResult=false, seqid=63429839, majorCompaction=false
2010-12-20 17:25:49,379 DEBUG org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://00.hbase.jive:54310/hbase/article/5e0ac5bd27f34da31ded307e2ea932cf/data/779
2081177431743589, isReference=false, isBulkLoadResult=false, seqid=62768007, majorCompaction=false
2010-12-20 17:25:49,402 DEBUG org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://00.hbase.jive:54310/hbase/article/5e0ac5bd27f34da31ded307e2ea932cf/data/793
5329939530860970, isReference=false, isBulkLoadResult=false, seqid=62300392, majorCompaction=false
2010-12-20 17:25:49,427 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://00.hbase.jive:54310/hbase/article/5e0ac5bd27f34da31ded307e2e
a932cf/recovered.edits/0000000000052141022; minSequenceid=63429839
2010-12-20 17:25:49,609 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region article,933fc87674f7132b5d91ce6e9c60bc68981a56a7,1292704641421.2551c0c46387d1479d25fb359ef4c5c9. in 0sec
2010-12-20 17:25:49,609 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on region article,fc36770defdbeba6e2841181a7da6f7bbae7f64f,1292701971082.399bf454e0c2a73adc038810e8fe4017.
2010-12-20 17:25:49,610 DEBUG org.apache.hadoop.hbase.regionserver.Store: Compaction size of data: 186.6m; Skipped 2 file(s), size: 192468499
2010-12-20 17:25:49,610 INFO org.apache.hadoop.hbase.regionserver.Store: Started compaction of 2 file(s) in data of article,fc36770defdbeba6e2841181a7da6f7bbae7f64f,1292701971082.399bf454e0c2a73adc038810e8fe4017. into hdfs://00.hbase.jive:54310/hbase/article/399bf454e0c2a73adc038810e8fe4017/.tmp, sequenceid=63427381
2010-12-20 17:26:37,543 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 56739ms for sessionid 0x12ce6e6154d00b3, closing socket connection and attempting reconnect
2010-12-20 17:26:37,632 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Applied 0, skipped 5388, firstSequenceidInLog=52141022, maxSequenceidInLog=63429839
2010-12-20 17:26:37,644 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: Aborting region server serverName=02.hbase.jive,60020,1292865913422, load=(requests=44, regions=142, usedHeap=97, maxHeap=5991): Unhandled exception
org.apache.hadoop.hbase.YouAreDeadException: org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing 02.hbase.jive,60020,1292865913422 as dead server
at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:217)
at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:271)
at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:744)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:561)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
at org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
at org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
at org.apache.hadoop.hbase.regionserver.HRegionServer.doRun(HRegionServer.java:635)
at org.apache.hadoop.hbase.regionserver.HRegionServer.access$000(HRegionServer.java:126)
at org.apache.hadoop.hbase.regionserver.HRegionServer$1.run(HRegionServer.java:518)
at org.apache.hadoop.hbase.regionserver.HRegionServer$1.run(HRegionServer.java:516)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:337)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1046)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:516)
at java.lang.Thread.run(Thread.java:662)
2010-12-20 17:26:37,645 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: request=0.0, regions=142, stores=142, storefiles=462, storefileIndexSize=43, memstoreSize=4, compactionQueueSize=36, usedHeap=105, maxHeap=5991, blockCacheSize=10308888, blockCacheFree=1246239080, blockCacheCount=0, blockCacheHitRatio=0
2010-12-20 17:26:37,645 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60020
2010-12-20 17:26:37,645 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 0 on 60020: exiting
2010-12-20 17:26:37,645 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 1 on 60020: exiting
2010-12-20 17:26:37,645 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 2 on 60020: exiting
////////////////DATANODE
2010-12-20 17:25:48,970 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 1 for block blk_7313855026629458364_85588 terminating
2010-12-20 17:25:49,012 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:50010, dest: /10.101.25.249:53108, bytes: 232, op: HDFS_READ, cliID: DFSClient_hb_rs_02.hbase.jive,60020,1292865913422_1292865913443, offset: 5637120, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_7313855026629458364_85588, duration: 168436
2010-12-20 17:25:49,013 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:50010, dest: /10.101.25.249:53109, bytes: 7456, op: HDFS_READ, cliID: DFSClient_hb_rs_02.hbase.jive,60020,1292865913422_1292865913443, offset: 5629952, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_7313855026629458364_85588, duration: 178950
2010-12-20 17:25:49,061 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_5943185653696604572_85589 src: /10.101.25.249:53112 dest: /10.101.25.249:50010
2010-12-20 17:25:49,103 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:50010, dest: /10.101.25.249:53111, bytes: 349508, op: HDFS_READ, cliID: DFSClient_hb_rs_02.hbase.jive,60020,1292865913422_1292865913443, offset: 0, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_7780046853952422303_85486, duration: 73022705
2010-12-20 17:25:49,307 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:53112, dest: /10.101.25.249:50010, bytes: 5564634, op: HDFS_WRITE, cliID: DFSClient_hb_rs_02.hbase.jive,60020,1292865913422_1292865913443, offset: 0, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_5943185653696604572_85589, duration: 242139544
2010-12-20 17:25:49,308 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 1 for block blk_5943185653696604572_85589 terminating
2010-12-20 17:25:49,324 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:50010, dest: /10.101.25.249:53114, bytes: 222, op: HDFS_READ, cliID: DFSClient_hb_rs_02.hbase.jive,60020,1292865913422_1292865913443, offset: 5564416, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_5943185653696604572_85589, duration: 144173
2010-12-20 17:25:49,325 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:50010, dest: /10.101.25.249:53115, bytes: 7446, op: HDFS_READ, cliID: DFSClient_hb_rs_02.hbase.jive,60020,1292865913422_1292865913443, offset: 5557248, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_5943185653696604572_85589, duration: 157056
2010-12-20 17:25:49,355 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_-5312992283942058338_85590 src: /10.101.25.249:53118 dest: /10.101.25.249:50010
2010-12-20 17:25:49,358 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:50010, dest: /10.101.25.249:53120, bytes: 199, op: HDFS_READ, cliID: DFSClient_hb_rs_02.hbase.jive,60020,1292865913422_1292865913443, offset: 25063424, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_-2003525593082351171_69708, duration: 138638
2010-12-20 17:25:49,359 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:50010, dest: /10.101.25.249:53121, bytes: 190603, op: HDFS_READ, cliID: DFSClient_hb_rs_02.hbase.jive,60020,1292865913422_1292865913443, offset: 24874496, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_-2003525593082351171_69708, duration: 735353
2010-12-20 17:25:49,411 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:50010, dest: /10.101.25.249:53117, bytes: 3070642, op: HDFS_READ, cliID: DFSClient_hb_rs_02.hbase.jive,60020,1292865913422_1292865913443, offset: 0, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_8585904126405165382_85484, duration: 68063489
2010-12-20 17:25:49,597 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:53118, dest: /10.101.25.249:50010, bytes: 4631054, op: HDFS_WRITE, cliID: DFSClient_hb_rs_02.hbase.jive,60020,1292865913422_1292865913443, offset: 0, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_-5312992283942058338_85590, duration: 234095142
2010-12-20 17:25:49,597 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 1 for block blk_-5312992283942058338_85590 terminating
2010-12-20 17:25:49,604 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:50010, dest: /10.101.25.249:53129, bytes: 534, op: HDFS_READ, cliID: DFSClient_hb_rs_02.hbase.jive,60020,1292865913422_1292865913443, offset: 4630528, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_-5312992283942058338_85590, duration: 186485
2010-12-20 17:25:49,605 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:50010, dest: /10.101.25.249:53130, bytes: 6210, op: HDFS_READ, cliID: DFSClient_hb_rs_02.hbase.jive,60020,1292865913422_1292865913443, offset: 4624896, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_-5312992283942058338_85590, duration: 155935
2010-12-20 17:25:49,672 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_-7865815939426325649_85591 src: /10.101.25.249:53133 dest: /10.101.25.249:50010
2010-12-20 17:26:22,770 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:50010, dest: /10.112.79.14:41575, bytes: 166, op: HDFS_READ, cliID: DFSClient_-1843172643, offset: 39720448, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_-2090804821592279293_75264, duration: 12128121
2010-12-20 17:26:22,810 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:50010, dest: /10.112.79.14:41576, bytes: 208114, op: HDFS_READ, cliID: DFSClient_-1843172643, offset: 39514112, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_-2090804821592279293_75264, duration: 38720643
2010-12-20 17:26:23,107 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:50010, dest: /10.112.79.14:41581, bytes: 312, op: HDFS_READ, cliID: DFSClient_-1843172643, offset: 16006656, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_4633255788259970659_77483, duration: 250202855
2010-12-20 17:26:37,538 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-8913559857952133277_85447 file /dist2/data/hadoop-data/current/subdir34/blk_-8913559857952133277 for deletion
2010-12-20 17:26:37,547 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.101.25.249:50010, dest: /10.112.79.14:41582, bytes: 19920, op: HDFS_READ, cliID: DFSClient_-1843172643, offset: 15987200, srvID: DS-1979430320-10.101.25.249-50010-1292359320669, blockid: blk_4633255788259970659_77483, duration: 4042558
2010-12-20 17:26:37,547 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-8511507105977516601_84587 file /dist3/data/hadoop-data/current/subdir31/blk_-8511507105977516601 for deletion
2010-12-20 17:26:37,549 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block blk_-8913559857952133277_85447 at file /dist2/data/hadoop-data/current/subdir34/blk_-8913559857952133277
2010-12-20 17:26:37,555 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-8508054838724986047_84580 file /dist2/data/hadoop-data/current/subdir41/blk_-8508054838724986047 for deletion
2010-12-20 17:26:37,556 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-7845924152894793652_85398 file /dist3/data/hadoop-data/current/subdir43/blk_-7845924152894793652 for deletion
2010-12-20 17:26:37,556 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-7088372163619556394_84570 file /dist2/data/hadoop-data/current/subdir19/blk_-7088372163619556394 for deletion
2010-12-20 17:26:37,556 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-6970445900444688558_85387 file /dist1/data/hadoop-data/current/subdir57/blk_-6970445900444688558 for deletion
2010-12-20 17:26:37,557 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-6453226657933735816_84505 file /dist1/data/hadoop-data/current/subdir43/blk_-6453226657933735816 for deletion
2010-12-20 17:26:37,557 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-6417800567255755267_84658 file /dist2/data/hadoop-data/current/subdir41/blk_-6417800567255755267 for deletion
2010-12-20 17:26:37,557 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-6349908890584705232_84612 file /dist1/data/hadoop-data/current/subdir57/blk_-6349908890584705232 for deletion
2010-12-20 17:26:37,557 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-6260136960291589474_84706 file /dist2/data/hadoop-data/current/subdir41/blk_-6260136960291589474 for deletion
2010-12-20 17:26:37,557 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-6236925753292021036_84691 file /dist2/data/hadoop-data/current/subdir41/blk_-6236925753292021036 for deletion
2010-12-20 17:26:37,557 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-6108807505647576258_84708 file /dist3/data/hadoop-data/current/subdir59/blk_-6108807505647576258 for deletion
2010-12-20 17:26:37,557 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-5322405269255073859_84530 file /dist2/data/hadoop-data/current/subdir19/blk_-5322405269255073859 for deletion
2010-12-20 17:26:37,558 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-4932349484140214063_84528 file /dist3/data/hadoop-data/current/subdir31/blk_-4932349484140214063 for deletion
2010-12-20 17:26:37,558 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-4903335785153601370_84648 file /dist3/data/hadoop-data/current/subdir59/blk_-4903335785153601370 for deletion
2010-12-20 17:26:37,558 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-4036825098230519223_84531 file /dist3/data/hadoop-data/current/subdir31/blk_-4036825098230519223 for deletion
2010-12-20 17:26:37,558 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-3437992750138243050_85434 file /dist2/data/hadoop-data/current/subdir34/blk_-3437992750138243050 for deletion
2010-12-20 17:26:37,558 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block blk_-6970445900444688558_85387 at file /dist1/data/hadoop-data/current/subdir57/blk_-6970445900444688558
2010-12-20 17:26:37,558 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-3364192718686291807_85433 file /dist1/data/hadoop-data/current/subdir48/blk_-3364192718686291807 for deletion
2010-12-20 17:26:37,558 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-3357432203841337074_84713 file /dist1/data/hadoop-data/current/subdir57/blk_-3357432203841337074 for deletion
2010-12-20 17:26:37,558 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-3238846993152973042_84497 file /dist1/data/hadoop-data/current/subdir43/blk_-3238846993152973042 for deletion
2010-12-20 17:26:37,558 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-2996249640811055720_83272 file /dist1/data/hadoop-data/current/subdir31/blk_-2996249640811055720 for deletion
2010-12-20 17:26:37,558 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-2790489595075270318_85390 file /dist3/data/hadoop-data/current/subdir43/blk_-2790489595075270318 for deletion
2010-12-20 17:26:37,558 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-2781143678253104315_84697 file /dist3/data/hadoop-data/current/subdir59/blk_-2781143678253104315 for deletion
2010-12-20 17:26:37,558 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-2640743692778440261_84536 file /dist2/data/hadoop-data/current/subdir19/blk_-2640743692778440261 for deletion
2010-12-20 17:26:37,558 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-2220284194502441495_85381 file /dist3/data/hadoop-data/current/subdir59/blk_-2220284194502441495 for deletion
2010-12-20 17:26:37,559 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-2126728115231716918_84647 file /dist2/data/hadoop-data/current/subdir41/blk_-2126728115231716918 for deletion
2010-12-20 17:26:37,559 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-2101072572804865450_84585 file /dist1/data/hadoop-data/current/subdir43/blk_-2101072572804865450 for deletion
2010-12-20 17:26:37,559 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-1725324590496466521_84695 file /dist1/data/hadoop-data/current/subdir57/blk_-1725324590496466521 for deletion
2010-12-20 17:26:37,559 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-1569098086020238107_83275 file /dist2/data/hadoop-data/current/subdir57/blk_-1569098086020238107 for deletion
2010-12-20 17:26:37,559 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-638559696544756508_85446 file /dist1/data/hadoop-data/current/subdir48/blk_-638559696544756508 for deletion
2010-12-20 17:26:37,559 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-306662003767169937_84555 file /dist1/data/hadoop-data/current/subdir43/blk_-306662003767169937 for deletion
2010-12-20 17:26:37,559 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-278326848888782043_84594 file /dist1/data/hadoop-data/current/subdir57/blk_-278326848888782043 for deletion
2010-12-20 17:26:37,559 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block blk_-6453226657933735816_84505 at file /dist1/data/hadoop-data/current/subdir43/blk_-6453226657933735816
2010-12-20 17:26:37,559 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_-91580339643929179_85459 file /dist3/data/hadoop-data/current/subdir43/blk_-91580339643929179 for deletion
2010-12-20 17:26:37,559 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_9419128090539524_84662 file /dist1/data/hadoop-data/current/subdir57/blk_9419128090539524 for deletion
2010-12-20 17:26:37,559 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_679544100533753423_84719 file /dist1/data/hadoop-data/current/subdir57/blk_679544100533753423 for deletion
2010-12-20 17:26:37,559 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_730403277102726249_84590 file /dist3/data/hadoop-data/current/subdir31/blk_730403277102726249 for deletion
2010-12-20 17:26:37,572 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_797603508126576279_84551 file /dist3/data/hadoop-data/current/subdir31/blk_797603508126576279 for deletion
2010-12-20 17:26:37,572 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_867345031868208479_85424 file /dist3/data/hadoop-data/current/subdir43/blk_867345031868208479 for deletion
2010-12-20 17:26:37,572 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_1054277086916165125_84630 file /dist2/data/hadoop-data/current/subdir41/blk_1054277086916165125 for deletion
2010-12-20 17:26:37,572 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_1276716639229193075_85420 file /dist1/data/hadoop-data/current/subdir57/blk_1276716639229193075 for deletion
2010-12-20 17:26:37,572 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_1747684879678419666_84627 file /dist1/data/hadoop-data/current/subdir57/blk_1747684879678419666 for deletion
2010-12-20 17:26:37,572 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_1962685174146987659_84561 file /dist3/data/hadoop-data/current/subdir31/blk_1962685174146987659 for deletion
2010-12-20 17:26:37,572 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_2025768890877930088_84552 file /dist1/data/hadoop-data/current/subdir43/blk_2025768890877930088 for deletion
2010-12-20 17:26:37,572 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_2156546746312159503_84617 file /dist3/data/hadoop-data/current/subdir59/blk_2156546746312159503 for deletion
2010-12-20 17:26:37,572 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_2580899260621247350_85413 file /dist2/data/hadoop-data/current/subdir34/blk_2580899260621247350 for deletion
2010-12-20 17:26:37,572 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_2600394775430572452_85458 file /dist2/data/hadoop-data/current/subdir34/blk_2600394775430572452 for deletion
2010-12-20 17:26:37,572 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_2906704921361114164_84593 file /dist3/data/hadoop-data/current/subdir31/blk_2906704921361114164 for deletion
2010-12-20 17:26:37,572 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_3088125590653329452_85466 file /dist1/data/hadoop-data/current/subdir48/blk_3088125590653329452 for deletion
2010-12-20 17:26:37,572 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_3263574674943605427_84527 file /dist2/data/hadoop-data/current/subdir19/blk_3263574674943605427 for deletion
2010-12-20 17:26:37,573 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_3271569822852131416_85399 file /dist1/data/hadoop-data/current/subdir57/blk_3271569822852131416 for deletion
2010-12-20 17:26:37,573 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_3629755351194789263_84618 file /dist1/data/hadoop-data/current/subdir57/blk_3629755351194789263 for deletion
2010-12-20 17:26:37,573 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_3872473819820471284_84646 file /dist1/data/hadoop-data/current/subdir57/blk_3872473819820471284 for deletion
2010-12-20 17:26:37,573 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_3967429886843094718_84586 file /dist2/data/hadoop-data/current/subdir41/blk_3967429886843094718 for deletion
2010-12-20 17:26:37,573 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_4177484748478595642_84600 file /dist1/data/hadoop-data/current/subdir57/blk_4177484748478595642 for deletion
2010-12-20 17:26:37,573 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_4741750624588255690_85455 file /dist3/data/hadoop-data/current/subdir43/blk_4741750624588255690 for deletion
2010-12-20 17:26:37,573 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_5189334311200664471_84687 file /dist3/data/hadoop-data/current/subdir59/blk_5189334311200664471 for deletion
2010-12-20 17:26:37,573 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_5554094926453315166_84616 file /dist2/data/hadoop-data/current/subdir41/blk_5554094926453315166 for deletion
2010-12-20 17:26:37,573 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_5630019573499502243_85460 file /dist1/data/hadoop-data/current/subdir48/blk_5630019573499502243 for deletion
2010-12-20 17:26:37,573 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_6462688363000457793_84620 file /dist3/data/hadoop-data/current/subdir59/blk_6462688363000457793 for deletion
2010-12-20 17:26:37,573 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_6470741303722852096_84592 file /dist2/data/hadoop-data/current/subdir41/blk_6470741303722852096 for deletion
2010-12-20 17:26:37,573 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_6605638457633532879_84632 file /dist3/data/hadoop-data/current/subdir59/blk_6605638457633532879 for deletion
2010-12-20 17:26:37,573 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_6794523298330118718_77314 file /dist3/data/hadoop-data/current/subdir11/blk_6794523298330118718 for deletion
2010-12-20 17:26:37,574 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_6926053514624905958_84695 file /dist3/data/hadoop-data/current/subdir59/blk_6926053514624905958 for deletion
2010-12-20 17:26:37,574 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_7116690323560112908_85418 file /dist2/data/hadoop-data/current/subdir34/blk_7116690323560112908 for deletion
2010-12-20 17:26:37,574 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_7202997804403926942_84705 file /dist1/data/hadoop-data/current/subdir57/blk_7202997804403926942 for deletion
2010-12-20 17:26:37,574 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_7434927287494785191_85394 file /dist2/data/hadoop-data/current/subdir34/blk_7434927287494785191 for deletion
2010-12-20 17:26:37,574 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_7594986426526893113_84520 file /dist3/data/hadoop-data/current/subdir31/blk_7594986426526893113 for deletion
2010-12-20 17:26:37,574 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_7661061963483856377_85463 file /dist3/data/hadoop-data/current/subdir43/blk_7661061963483856377 for deletion
2010-12-20 17:26:37,574 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_7780046853952422303_85486 file /dist3/data/hadoop-data/current/subdir43/blk_7780046853952422303 for deletion
2010-12-20 17:26:37,574 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_7789033250788024385_83279 file /dist1/data/hadoop-data/current/subdir31/blk_7789033250788024385 for deletion
2010-12-20 17:26:37,574 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_7805477460105158814_84523 file /dist1/data/hadoop-data/current/subdir43/blk_7805477460105158814 for deletion
2010-12-20 17:26:37,574 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_7852099683471709790_84591 file /dist1/data/hadoop-data/current/subdir57/blk_7852099683471709790 for deletion
2010-12-20 17:26:37,574 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_8229638397827768666_85415 file /dist3/data/hadoop-data/current/subdir43/blk_8229638397827768666 for deletion
2010-12-20 17:26:37,574 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_8541283056878555347_85477 file /dist1/data/hadoop-data/current/subdir48/blk_8541283056878555347 for deletion
2010-12-20 17:26:37,574 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_8585904126405165382_85484 file /dist1/data/hadoop-data/current/subdir48/blk_8585904126405165382 for deletion
2010-12-20 17:26:37,575 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_8606266970652887102_84483 file /dist1/data/hadoop-data/current/subdir43/blk_8606266970652887102 for deletion
2010-12-20 17:26:37,575 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_9082358445568727636_85437 file /dist2/data/hadoop-data/current/subdir34/blk_9082358445568727636 for deletion
2010-12-20 17:26:37,575 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block blk_-6349908890584705232_84612 at file /dist1/data/hadoop-data/current/subdir57/blk_-6349908890584705232
//////////////////////GC
2010-12-20T17:25:33.242+0000: 20.929: [CMS-concurrent-mark: 0.079/0.380 secs] [Times: user=0.36 sys=0.04, real=0.38 secs]
2010-12-20T17:25:33.242+0000: 20.929: [CMS-concurrent-preclean-start]
2010-12-20T17:25:33.245+0000: 20.932: [CMS-concurrent-preclean: 0.002/0.003 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:33.245+0000: 20.932: [CMS-concurrent-abortable-preclean-start]
2010-12-20T17:25:33.302+0000: 20.989: [GC 20.989: [ParNew: 17796K->1070K(19136K), 0.0025870 secs] 68068K->51343K(101612K) icms_dc=9 , 0.0027060 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:33.441+0000: 21.127: [GC 21.127: [ParNew: 18034K->2086K(19136K), 0.0042130 secs] 68307K->53943K(101612K) icms_dc=9 , 0.0043330 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2010-12-20T17:25:33.514+0000: 21.201: [CMS-concurrent-abortable-preclean: 0.007/0.269 secs] [Times: user=0.19 sys=0.05, real=0.27 secs]
2010-12-20T17:25:33.515+0000: 21.202: [GC[YG occupancy: 11042 K (19136 K)]21.202: [Rescan (parallel) , 0.0158670 secs]21.218: [weak refs processing, 0.0000950 secs] [1 CMS-remark: 51857K(82476K)] 62900K(101612K), 0.0160770 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2010-12-20T17:25:33.531+0000: 21.218: [CMS-concurrent-sweep-start]
2010-12-20T17:25:33.566+0000: 21.253: [GC 21.253: [ParNew: 19070K->1999K(19136K), 0.0027670 secs] 70927K->54085K(101612K) icms_dc=9 , 0.0028940 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:33.648+0000: 21.335: [GC 21.335: [ParNew: 19023K->2112K(19136K), 0.0046590 secs] 70961K->55368K(101612K) icms_dc=9 , 0.0047840 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2010-12-20T17:25:33.713+0000: 21.400: [CMS-concurrent-sweep: 0.013/0.182 secs] [Times: user=0.14 sys=0.03, real=0.18 secs]
2010-12-20T17:25:33.714+0000: 21.401: [CMS-concurrent-reset-start]
2010-12-20T17:25:33.893+0000: 21.580: [GC 21.580: [ParNew: 19123K->598K(19136K), 0.0015670 secs] 72284K->53771K(107740K) icms_dc=9 , 0.0016740 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:33.945+0000: 21.632: [CMS-concurrent-reset: 0.020/0.231 secs] [Times: user=0.19 sys=0.07, real=0.23 secs]
2010-12-20T17:25:34.078+0000: 21.764: [GC 21.765: [ParNew: 17622K->1665K(19136K), 0.0021650 secs] 70795K->54839K(107740K) icms_dc=4 , 0.0022950 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:34.127+0000: 21.814: [GC [1 CMS-initial-mark: 53173K(88604K)] 63030K(107740K), 0.0031010 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:34.130+0000: 21.817: [CMS-concurrent-mark-start]
2010-12-20T17:25:34.181+0000: 21.867: [GC 21.867: [ParNew: 18633K->1360K(19136K), 0.0044440 secs] 71806K->55614K(107740K) icms_dc=5 , 0.0045740 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:34.387+0000: 22.074: [GC 22.074: [ParNew: 18370K->1923K(19136K), 0.0020250 secs] 72624K->56177K(107740K) icms_dc=5 , 0.0021580 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:34.557+0000: 22.244: [GC 22.244: [ParNew: 18947K->2112K(19136K), 0.0225430 secs] 73201K->58091K(107740K) icms_dc=6 , 0.0226640 secs] [Times: user=0.02 sys=0.00, real=0.03 secs]
2010-12-20T17:25:34.681+0000: 22.367: [GC 22.367: [ParNew: 19098K->811K(19136K), 0.0031310 secs] 75077K->57280K(107740K) icms_dc=6 , 0.0032510 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:34.751+0000: 22.437: [CMS-concurrent-mark: 0.087/0.620 secs] [Times: user=0.42 sys=0.04, real=0.63 secs]
2010-12-20T17:25:34.751+0000: 22.438: [CMS-concurrent-preclean-start]
2010-12-20T17:25:34.757+0000: 22.444: [CMS-concurrent-preclean: 0.005/0.006 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
2010-12-20T17:25:34.757+0000: 22.444: [CMS-concurrent-abortable-preclean-start]
2010-12-20T17:25:34.865+0000: 22.552: [GC 22.552: [ParNew: 17835K->1114K(19136K), 0.0016200 secs] 74304K->57582K(107740K) icms_dc=6 , 0.0017310 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:34.966+0000: 22.652: [CMS-concurrent-abortable-preclean: 0.003/0.209 secs] [Times: user=0.10 sys=0.02, real=0.21 secs]
2010-12-20T17:25:34.966+0000: 22.653: [GC[YG occupancy: 9657 K (19136 K)]22.653: [Rescan (parallel) , 0.0020570 secs]22.655: [weak refs processing, 0.0000610 secs] [1 CMS-remark: 56468K(88604K)] 66126K(107740K), 0.0022320 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2010-12-20T17:25:34.969+0000: 22.656: [CMS-concurrent-sweep-start]
2010-12-20T17:25:35.101+0000: 22.787: [GC 22.787: [ParNew: 18138K->2112K(19136K), 0.0038570 secs] 74584K->60794K(107740K) icms_dc=6 , 0.0039870 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:35.214+0000: 22.900: [CMS-concurrent-sweep: 0.013/0.245 secs] [Times: user=0.07 sys=0.01, real=0.24 secs]
2010-12-20T17:25:35.214+0000: 22.900: [CMS-concurrent-reset-start]
2010-12-20T17:25:35.354+0000: 23.041: [GC 23.041: [ParNew: 19136K->1909K(19136K), 0.0035100 secs] 75099K->58683K(112412K) icms_dc=6 , 0.0036250 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2010-12-20T17:25:35.449+0000: 23.136: [GC 23.136: [ParNew: 18931K->1384K(19136K), 0.0027830 secs] 75706K->59022K(112412K) icms_dc=6 , 0.0029070 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:35.499+0000: 23.186: [CMS-concurrent-reset: 0.018/0.285 secs] [Times: user=18446743517.92 sys=0.04, real=0.29 secs]
2010-12-20T17:25:35.606+0000: 23.292: [GC 23.292: [ParNew: 18352K->1675K(19136K), 0.0019650 secs] 75990K->59313K(112412K) icms_dc=1 , 0.0020900 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2010-12-20T17:25:35.661+0000: 23.348: [GC [1 CMS-initial-mark: 57637K(93276K)] 67851K(112412K), 0.0032750 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:35.664+0000: 23.351: [CMS-concurrent-mark-start]
2010-12-20T17:25:35.779+0000: 23.466: [GC 23.466: [ParNew: 18699K->2112K(19136K), 0.0049320 secs] 76337K->60788K(112412K) icms_dc=4 , 0.0050890 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2010-12-20T17:25:35.963+0000: 23.650: [GC 23.650: [ParNew: 19073K->1947K(19136K), 0.0080830 secs] 77750K->61429K(112412K) icms_dc=4 , 0.0082030 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2010-12-20T17:25:36.237+0000: 23.924: [GC 23.924: [ParNew: 18964K->1547K(19136K), 0.0041330 secs] 78447K->61930K(112412K) icms_dc=4 , 0.0042610 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2010-12-20T17:25:36.361+0000: 24.048: [GC 24.048: [ParNew: 18514K->987K(19136K), 0.0038260 secs] 78897K->61370K(112412K) icms_dc=4 , 0.0039330 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2010-12-20T17:25:36.454+0000: 24.141: [CMS-concurrent-mark: 0.089/0.790 secs] [Times: user=36893488016.88 sys=0.06, real=0.79 secs]
2010-12-20T17:25:36.454+0000: 24.141: [CMS-concurrent-preclean-start]
2010-12-20T17:25:36.460+0000: 24.147: [CMS-concurrent-preclean: 0.005/0.006 secs] [Times: user=0.00 sys=0.02, real=0.01 secs]
2010-12-20T17:25:36.460+0000: 24.147: [CMS-concurrent-abortable-preclean-start]
2010-12-20T17:25:36.520+0000: 24.207: [GC 24.207: [ParNew: 17969K->1341K(19136K), 0.0023940 secs] 78352K->61724K(112412K) icms_dc=4 , 0.0025130 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:36.728+0000: 24.414: [GC 24.414: [ParNew: 18356K->1010K(19136K), 0.0025740 secs] 78739K->61870K(112412K) icms_dc=4 , 0.0026990 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2010-12-20T17:25:36.800+0000: 24.487: [CMS-concurrent-abortable-preclean: 0.009/0.340 secs] [Times: user=0.26 sys=0.02, real=0.34 secs]
2010-12-20T17:25:36.869+0000: 24.555: [GC[YG occupancy: 9861 K (19136 K)]24.555: [Rescan (parallel) , 0.0012070 secs]24.557: [weak refs processing, 0.0000890 secs] [1 CMS-remark: 60859K(93276K)] 70720K(112412K), 0.0014050 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
2010-12-20T17:25:36.870+0000: 24.557: [CMS-concurrent-sweep-start]
2010-12-20T17:25:36.948+0000: 24.634: [GC 24.634: [ParNew: 18027K->1183K(19136K), 0.0035250 secs] 78793K->61949K(112412K) icms_dc=4 , 0.0036410 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:37.088+0000: 24.775: [CMS-concurrent-sweep: 0.015/0.218 secs] [Times: user=0.05 sys=0.00, real=0.21 secs]
2010-12-20T17:25:37.088+0000: 24.775: [CMS-concurrent-reset-start]
2010-12-20T17:25:37.105+0000: 24.791: [CMS-concurrent-reset: 0.016/0.016 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2010-12-20T17:25:37.155+0000: 24.842: [GC 24.842: [ParNew: 18153K->961K(19136K), 0.0233150 secs] 76441K->59351K(116284K) icms_dc=0 , 0.0234410 secs] [Times: user=0.02 sys=0.00, real=0.03 secs]
2010-12-20T17:25:37.293+0000: 24.980: [GC 24.980: [ParNew: 17985K->1566K(19136K), 0.0078520 secs] 76375K->59957K(116284K) icms_dc=0 , 0.0079770 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2010-12-20T17:25:37.470+0000: 25.157: [GC 25.157: [ParNew: 18533K->2095K(19136K), 0.0028390 secs] 76923K->60833K(116284K) icms_dc=0 , 0.0029650 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:37.628+0000: 25.315: [GC 25.315: [ParNew: 19119K->2112K(19136K), 0.0032590 secs] 77857K->61573K(116284K) icms_dc=0 , 0.0033810 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2010-12-20T17:25:37.692+0000: 25.379: [GC 25.379: [ParNew: 19092K->1664K(19136K), 0.0027200 secs] 78554K->61695K(116284K) icms_dc=0 , 0.0028400 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:37.814+0000: 25.501: [GC 25.501: [ParNew: 18688K->1461K(19136K), 0.0017560 secs] 78719K->61492K(116284K) icms_dc=0 , 0.0018670 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:37.946+0000: 25.632: [GC 25.632: [ParNew: 18449K->1309K(19136K), 0.0052360 secs] 78481K->61761K(116284K) icms_dc=0 , 0.0053520 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2010-12-20T17:25:38.114+0000: 25.800: [GC 25.800: [ParNew: 18333K->2112K(19136K), 0.0046840 secs] 78785K->64510K(116284K) icms_dc=0 , 0.0048140 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
2010-12-20T17:25:38.229+0000: 25.915: [GC 25.915: [ParNew: 19132K->1927K(19136K), 0.0301210 secs] 81531K->65315K(116284K) icms_dc=0 , 0.0302420 secs] [Times: user=0.03 sys=0.00, real=0.03 secs]
2010-12-20T17:25:38.404+0000: 26.091: [GC 26.091: [ParNew: 18887K->1639K(19136K), 0.0029570 secs] 82275K->65027K(116284K) icms_dc=0 , 0.0030810 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:38.551+0000: 26.237: [GC 26.237: [ParNew: 18655K->2092K(19136K), 0.0040070 secs] 82042K->66522K(116284K) icms_dc=0 , 0.0041290 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2010-12-20T17:25:38.810+0000: 26.497: [GC 26.497: [ParNew: 19101K->1244K(19136K), 0.0036340 secs] 83531K->66458K(116284K) icms_dc=0 , 0.0037620 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:38.920+0000: 26.607: [GC 26.607: [ParNew: 18253K->684K(19136K), 0.0014730 secs] 83468K->65898K(116284K) icms_dc=0 , 0.0015700 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:39.065+0000: 26.751: [GC 26.751: [ParNew: 17669K->782K(19136K), 0.0015630 secs] 82884K->65996K(116284K) icms_dc=0 , 0.0016660 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2010-12-20T17:25:39.213+0000: 26.900: [GC 26.900: [ParNew: 17756K->783K(19136K), 0.0015390 secs] 82970K->65998K(116284K) icms_dc=0 , 0.0016400 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:39.374+0000: 27.061: [GC 27.061: [ParNew: 17787K->1548K(19136K), 0.0023500 secs] 83001K->66975K(116284K) icms_dc=0 , 0.0024690 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2010-12-20T17:25:39.624+0000: 27.311: [GC 27.311: [ParNew: 18571K->798K(19136K), 0.0015980 secs] 83998K->66324K(116284K) icms_dc=0 , 0.0016950 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:39.736+0000: 27.423: [GC 27.423: [ParNew: 17771K->936K(19136K), 0.0014100 secs] 83297K->66463K(116284K) icms_dc=0 , 0.0015130 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:39.906+0000: 27.592: [GC 27.592: [ParNew: 17920K->1886K(19136K), 0.0020160 secs] 83447K->67413K(116284K) icms_dc=0 , 0.0021370 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2010-12-20T17:25:40.042+0000: 27.729: [GC 27.729: [ParNew: 18905K->1886K(19136K), 0.0040910 secs] 84431K->68678K(116284K) icms_dc=0 , 0.0042270 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:40.213+0000: 27.900: [GC 27.900: [ParNew: 18855K->1500K(19136K), 0.0031110 secs] 85647K->68745K(116284K) icms_dc=0 , 0.0032440 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:40.323+0000: 28.010: [GC 28.010: [ParNew: 18524K->2112K(19136K), 0.0060030 secs] 85769K->72197K(116284K) icms_dc=4 , 0.0061260 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:40.484+0000: 28.170: [GC [1 CMS-initial-mark: 70085K(97148K)] 80410K(116284K), 0.0038900 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2010-12-20T17:25:40.488+0000: 28.175: [CMS-concurrent-mark-start]
2010-12-20T17:25:40.567+0000: 28.254: [GC 28.254: [ParNew: 19136K->1886K(19136K), 0.0043620 secs] 89221K->72619K(116284K) icms_dc=10 , 0.0044950 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:40.752+0000: 28.439: [GC 28.439: [ParNew: 18883K->1596K(19136K), 0.0045560 secs] 89617K->73198K(116284K) icms_dc=10 , 0.0046680 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2010-12-20T17:25:41.018+0000: 28.704: [GC 28.705: [ParNew: 18590K->650K(19136K), 0.0046350 secs] 90191K->72706K(116284K) icms_dc=10 , 0.0047500 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:41.129+0000: 28.816: [GC 28.816: [ParNew: 17674K->399K(19136K), 0.0014520 secs] 89730K->72455K(116284K) icms_dc=10 , 0.0015540 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:41.290+0000: 28.977: [GC 28.977: [ParNew: 17396K->581K(19136K), 0.0021900 secs] 89452K->72637K(116284K) icms_dc=10 , 0.0022930 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2010-12-20T17:25:41.478+0000: 29.164: [GC 29.165: [ParNew: 17569K->929K(19136K), 0.0018590 secs] 89625K->72986K(116284K) icms_dc=10 , 0.0019820 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:41.653+0000: 29.340: [GC 29.340: [ParNew: 17894K->1013K(19136K), 0.0333480 secs] 89950K->73172K(116284K) icms_dc=10 , 0.0334710 secs] [Times: user=0.07 sys=0.00, real=0.04 secs]
2010-12-20T17:25:41.763+0000: 29.450: [CMS-concurrent-mark: 0.101/1.275 secs] [Times: user=18446743473.28 sys=0.08, real=1.28 secs]
2010-12-20T17:25:41.763+0000: 29.450: [CMS-concurrent-preclean-start]
2010-12-20T17:25:41.887+0000: 29.574: [GC 29.574: [ParNew: 18024K->1580K(19136K), 0.0025720 secs] 90183K->73783K(116284K) icms_dc=10 , 0.0026960 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:41.948+0000: 29.634: [CMS-concurrent-preclean: 0.006/0.184 secs] [Times: user=0.11 sys=0.01, real=0.18 secs]
2010-12-20T17:25:41.948+0000: 29.634: [CMS-concurrent-abortable-preclean-start]
2010-12-20T17:25:42.010+0000: 29.697: [GC 29.697: [ParNew: 18571K->1529K(19136K), 0.0261550 secs] 90773K->74878K(116284K) icms_dc=10 , 0.0262850 secs] [Times: user=0.01 sys=0.01, real=0.03 secs]
2010-12-20T17:25:42.037+0000: 29.723: [CMS-concurrent-abortable-preclean: 0.002/0.089 secs] [Times: user=0.05 sys=0.01, real=0.09 secs]
2010-12-20T17:25:42.037+0000: 29.724: [GC[YG occupancy: 1657 K (19136 K)]29.724: [Rescan (parallel) , 0.0021770 secs]29.726: [weak refs processing, 0.0002560 secs] [1 CMS-remark: 73349K(97148K)] 75006K(116284K), 0.0025180 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:42.040+0000: 29.727: [CMS-concurrent-sweep-start]
2010-12-20T17:25:42.143+0000: 29.830: [GC 29.830: [ParNew: 18553K->2112K(19136K), 0.0039990 secs] 91513K->76482K(116284K) icms_dc=10 , 0.0041230 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:42.159+0000: 29.846: [CMS-concurrent-sweep: 0.021/0.119 secs] [Times: user=0.11 sys=0.02, real=0.12 secs]
2010-12-20T17:25:42.159+0000: 29.846: [CMS-concurrent-reset-start]
2010-12-20T17:25:42.180+0000: 29.867: [CMS-concurrent-reset: 0.021/0.021 secs] [Times: user=0.02 sys=0.01, real=0.02 secs]
2010-12-20T17:25:42.220+0000: 29.907: [GC [1 CMS-initial-mark: 72204K(120344K)] 82008K(139480K), 0.0031690 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2010-12-20T17:25:42.223+0000: 29.910: [CMS-concurrent-mark-start]
2010-12-20T17:25:42.363+0000: 30.050: [GC 30.050: [ParNew: 19089K->1911K(19136K), 0.0287550 secs] 91294K->74871K(139480K) icms_dc=5 , 0.0288700 secs] [Times: user=0.02 sys=0.00, real=0.03 secs]
2010-12-20T17:25:42.520+0000: 30.207: [GC 30.207: [ParNew: 18890K->2095K(19136K), 0.0026020 secs] 91850K->76245K(139480K) icms_dc=5 , 0.0027290 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:42.626+0000: 30.313: [GC 30.313: [ParNew: 19116K->1072K(19136K), 0.0021620 secs] 93267K->75735K(139480K) icms_dc=5 , 0.0022790 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:42.794+0000: 30.480: [GC 30.480: [ParNew: 18095K->1985K(19136K), 0.0224570 secs] 92758K->76648K(139480K) icms_dc=5 , 0.0225750 secs] [Times: user=0.02 sys=0.00, real=0.03 secs]
2010-12-20T17:25:43.028+0000: 30.715: [GC 30.715: [ParNew: 18975K->1230K(19136K), 0.0029720 secs] 93638K->76922K(139480K) icms_dc=5 , 0.0031070 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:43.127+0000: 30.814: [CMS-concurrent-mark: 0.101/0.904 secs] [Times: user=45.86 sys=0.07, real=0.90 secs]
2010-12-20T17:25:43.127+0000: 30.814: [CMS-concurrent-preclean-start]
2010-12-20T17:25:43.140+0000: 30.826: [CMS-concurrent-preclean: 0.010/0.012 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2010-12-20T17:25:43.140+0000: 30.826: [CMS-concurrent-abortable-preclean-start]
2010-12-20T17:25:43.263+0000: 30.950: [GC 30.950: [ParNew: 18242K->1343K(19136K), 0.0013570 secs] 93934K->77035K(139480K) icms_dc=5 , 0.0014830 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:43.439+0000: 31.126: [GC 31.126: [ParNew: 18367K->1276K(19136K), 0.0013350 secs] 94059K->76968K(139480K) icms_dc=5 , 0.0014590 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:43.539+0000: 31.225: [CMS-concurrent-abortable-preclean: 0.008/0.399 secs] [Times: user=0.16 sys=0.02, real=0.40 secs]
2010-12-20T17:25:43.539+0000: 31.226: [GC[YG occupancy: 11143 K (19136 K)]31.226: [Rescan (parallel) , 0.0012840 secs]31.227: [weak refs processing, 0.0000800 secs] [1 CMS-remark: 75692K(120344K)] 86835K(139480K), 0.0014660 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:43.566+0000: 31.252: [CMS-concurrent-sweep-start]
2010-12-20T17:25:43.588+0000: 31.275: [CMS-concurrent-sweep: 0.022/0.022 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
2010-12-20T17:25:43.588+0000: 31.275: [CMS-concurrent-reset-start]
2010-12-20T17:25:43.606+0000: 31.292: [CMS-concurrent-reset: 0.017/0.017 secs] [Times: user=0.01 sys=0.00, real=0.02 secs]
2010-12-20T17:25:43.617+0000: 31.303: [GC 31.303: [ParNew: 18237K->1557K(19136K), 0.0016650 secs] 91506K->74826K(141252K) icms_dc=0 , 0.0017980 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:43.766+0000: 31.452: [GC 31.452: [ParNew: 18525K->986K(19136K), 0.0019760 secs] 91794K->74744K(141252K) icms_dc=0 , 0.0021000 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
2010-12-20T17:25:43.984+0000: 31.671: [GC 31.671: [ParNew: 18010K->1004K(19136K), 0.0011630 secs] 91768K->74762K(141252K) icms_dc=0 , 0.0012740 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:44.155+0000: 31.842: [GC 31.842: [ParNew: 17986K->1535K(19136K), 0.0020850 secs] 91744K->75294K(141252K) icms_dc=0 , 0.0022080 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:44.279+0000: 31.965: [GC 31.965: [ParNew: 18525K->1844K(19136K), 0.0036450 secs] 92284K->75963K(141252K) icms_dc=0 , 0.0037850 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2010-12-20T17:25:44.446+0000: 32.132: [GC 32.133: [ParNew: 18868K->2112K(19136K), 0.0027860 secs] 92987K->76905K(141252K) icms_dc=0 , 0.0029120 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2010-12-20T17:25:44.560+0000: 32.247: [GC 32.247: [ParNew: 19136K->2062K(19136K), 0.0035670 secs] 93929K->77864K(141252K) icms_dc=0 , 0.0036890 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2010-12-20T17:25:44.786+0000: 32.473: [GC 32.473: [ParNew: 19081K->2099K(19136K), 0.0042870 secs] 94883K->79044K(141252K) icms_dc=0 , 0.0044120 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:45.051+0000: 32.737: [GC 32.738: [ParNew: 19082K->2112K(19136K), 0.0057110 secs] 96026K->79932K(141252K) icms_dc=0 , 0.0058410 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
2010-12-20T17:25:45.159+0000: 32.845: [GC 32.845: [ParNew: 19132K->1902K(19136K), 0.0048970 secs] 96953K->80775K(141252K) icms_dc=0 , 0.0050310 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
2010-12-20T17:25:45.315+0000: 33.001: [GC 33.001: [ParNew: 18888K->1172K(19136K), 0.0016980 secs] 97761K->80045K(141252K) icms_dc=0 , 0.0018120 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2010-12-20T17:25:45.416+0000: 33.103: [GC 33.103: [ParNew: 18143K->1113K(19136K), 0.0016580 secs] 97016K->79987K(141252K) icms_dc=0 , 0.0017670 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:45.571+0000: 33.258: [GC 33.258: [ParNew: 18117K->1175K(19136K), 0.0018150 secs] 96990K->80049K(141252K) icms_dc=0 , 0.0019290 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:45.716+0000: 33.403: [GC 33.403: [ParNew: 18193K->2092K(19136K), 0.0243240 secs] 97067K->83760K(141252K) icms_dc=0 , 0.0244520 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2010-12-20T17:25:46.275+0000: 33.961: [GC 33.961: [ParNew: 19116K->1282K(19136K), 0.0251550 secs] 100784K->83201K(141252K) icms_dc=0 , 0.0252890 secs] [Times: user=0.06 sys=0.00, real=0.03 secs]
2010-12-20T17:25:46.421+0000: 34.108: [GC 34.108: [ParNew: 18286K->1295K(19136K), 0.0186440 secs] 100205K->83214K(141252K) icms_dc=0 , 0.0187610 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
2010-12-20T17:25:46.676+0000: 34.363: [GC 34.363: [ParNew: 18319K->2112K(19136K), 0.0062280 secs] 100238K->84044K(141252K) icms_dc=0 , 0.0063550 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:46.825+0000: 34.512: [GC 34.512: [ParNew: 19136K->2112K(19136K), 0.0235520 secs] 101068K->87526K(141252K) icms_dc=0 , 0.0236890 secs] [Times: user=0.02 sys=0.00, real=0.03 secs]
2010-12-20T17:25:47.010+0000: 34.696: [GC 34.696: [ParNew: 19136K->1126K(19136K), 0.0036520 secs] 104550K->86998K(141252K) icms_dc=0 , 0.0037680 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:47.179+0000: 34.866: [GC 34.866: [ParNew: 18094K->2112K(19136K), 0.0036740 secs] 103967K->89106K(141252K) icms_dc=0 , 0.0037950 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:47.354+0000: 35.041: [GC 35.041: [ParNew: 19136K->1976K(19136K), 0.0042910 secs] 106130K->90305K(141252K) icms_dc=0 , 0.0044170 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:47.523+0000: 35.210: [GC 35.210: [ParNew: 18973K->2112K(19136K), 0.0020850 secs] 107303K->90449K(141252K) icms_dc=0 , 0.0022090 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:47.757+0000: 35.443: [GC 35.443: [ParNew: 19086K->1740K(19136K), 0.0047220 secs] 107424K->91042K(141252K) icms_dc=0 , 0.0048520 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
2010-12-20T17:25:47.949+0000: 35.635: [GC 35.635: [ParNew: 18758K->1349K(19136K), 0.0100480 secs] 108060K->90651K(141252K) icms_dc=0 , 0.0101550 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2010-12-20T17:25:48.160+0000: 35.846: [GC 35.847: [ParNew: 18373K->1062K(19136K), 0.0035050 secs] 107675K->90707K(141252K) icms_dc=0 , 0.0036170 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:48.363+0000: 36.050: [GC 36.050: [ParNew: 18057K->746K(19136K), 0.0150130 secs] 107702K->90391K(141252K) icms_dc=0 , 0.0151200 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2010-12-20T17:25:48.547+0000: 36.234: [GC 36.234: [ParNew: 17752K->2057K(19136K), 0.0048080 secs] 107397K->92215K(141252K) icms_dc=0 , 0.0049410 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
2010-12-20T17:25:48.755+0000: 36.442: [GC 36.442: [ParNew: 19042K->1221K(19136K), 0.0231150 secs] 109199K->91615K(141252K) icms_dc=0 , 0.0232640 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2010-12-20T17:25:48.867+0000: 36.554: [GC 36.554: [ParNew: 18227K->2102K(19136K), 0.0218400 secs] 108621K->95180K(141252K) icms_dc=0 , 0.0219730 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
2010-12-20T17:25:49.086+0000: 36.773: [GC 36.773: [ParNew: 19090K->1186K(19136K), 0.0040110 secs] 112167K->94679K(141252K) icms_dc=0 , 0.0041370 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:49.186+0000: 36.873: [GC 36.873: [ParNew: 18171K->1369K(19136K), 0.0018640 secs] 111664K->94862K(141252K) icms_dc=0 , 0.0019870 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:49.311+0000: 36.998: [GC 36.998: [ParNew: 18393K->154K(19136K), 0.0049150 secs] 111886K->94007K(141252K) icms_dc=0 , 0.0050180 secs] [Times: user=0.00 sys=0.01, real=0.01 secs]
2010-12-20T17:25:49.433+0000: 37.120: [GC 37.120: [ParNew: 17165K->2079K(19136K), 0.0081480 secs] 111018K->98366K(141252K) icms_dc=4 , 0.0082710 secs] [Times: user=0.00 sys=0.01, real=0.01 secs]
2010-12-20T17:25:49.544+0000: 37.230: [GC [1 CMS-initial-mark: 96287K(122116K)] 106652K(141252K), 0.0041720 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:25:49.548+0000: 37.235: [CMS-concurrent-mark-start]
2010-12-20T17:25:49.706+0000: 37.393: [GC 37.393: [ParNew: 19059K->818K(19136K), 47.8334700 secs] 115346K->97662K(141252K) icms_dc=10 , 47.8335900 secs] [Times: user=90.47 sys=0.01, real=47.86 secs]
2010-12-20T17:26:37.884+0000: 85.570: [GC 85.571: [ParNew: 17836K->541K(19136K), 0.0405420 secs] 114680K->97386K(141252K) icms_dc=10 , 0.0406480 secs] [Times: user=0.00 sys=0.00, real=0.04 secs]
2010-12-20T17:26:47.779+0000: 95.466: [CMS-concurrent-mark: 0.103/58.231 secs] [Times: user=36893487576.24 sys=0.98, real=58.22 secs]
2010-12-20T17:26:47.779+0000: 95.466: [CMS-concurrent-preclean-start]
2010-12-20T17:26:47.790+0000: 95.476: [CMS-concurrent-preclean: 0.008/0.010 secs] [Times: user=0.00 sys=0.01, real=0.01 secs]
2010-12-20T17:26:47.790+0000: 95.476: [CMS-concurrent-abortable-preclean-start]
2010-12-20T17:26:47.790+0000: 95.476: [CMS-concurrent-abortable-preclean: 0.000/0.000 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-12-20T17:26:47.790+0000: 95.477: [GC[YG occupancy: 3204 K (19136 K)]95.477: [Rescan (parallel) , 0.0018430 secs]95.479: [weak refs processing, 0.0006910 secs] [1 CMS-remark: 96844K(122116K)] 100048K(141252K), 0.0026330 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
2010-12-20T17:26:47.793+0000: 95.479: [CMS-concurrent-sweep-start]
2010-12-20T17:26:47.826+0000: 95.513: [CMS-concurrent-sweep: 0.033/0.033 secs] [Times: user=0.03 sys=0.00, real=0.03 secs]
2010-12-20T17:26:47.826+0000: 95.513: [CMS-concurrent-reset-start]
2010-12-20T17:26:47.842+0000: 95.528: [CMS-concurrent-reset: 0.016/0.016 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
Heap
par new generation total 19136K, used 3204K [0x00007f611e340000, 0x00007f611f800000, 0x00007f6123670000)
eden space 17024K, 15% used [0x00007f611e340000, 0x00007f611e5d9b38, 0x00007f611f3e0000)
from space 2112K, 25% used [0x00007f611f3e0000, 0x00007f611f467768, 0x00007f611f5f0000)
to space 2112K, 0% used [0x00007f611f5f0000, 0x00007f611f5f0000, 0x00007f611f800000)
concurrent mark-sweep generation total 122116K, used 73206K [0x00007f6123670000, 0x00007f612adb1000, 0x00007f6295340000)
concurrent-mark-sweep perm gen total 29708K, used 17810K [0x00007f6295340000, 0x00007f6297043000, 0x00007f629a740000)
On Dec 9, 2010, at 4:09 PM, Lance Riedel wrote:
> I came across just probably the culprit - I just discovered that the machines that were dying had OS swap turned OFF.
>
> So, I added swap to those machines, but also reduced the memory configs for tasks. (was at 1 gig, now at 512m). Seems to be stable right now, but the nightly longer term jobs run at night, so we will see how they do.
>
> This is an ec2 c1.xlarge, so the memory is at 8 gigs. I had 4 mappers per node, which was fine when all they had on them were Tasktrackers and datanodes. But with the hbase regionservers, the machines could definitely get over-allocated.
>
> Lance
>
> On Dec 9, 2010, at 3:57 PM, Ted Dunning wrote:
>
>> This could indicate swapping during GC.
>>
>> On Thu, Dec 9, 2010 at 12:13 PM, Lance Riedel <la...@gmail.com> wrote:
>>
>>> Seems reasonable, but having trouble making sense of the GC logs I had
>>> turned on. Basically since there was a full GC a minute before this happens
>>> on that server that lasts less than a second.
>>>
>>>
>>> Example:
>>> So, here is what the last of the GC logs say for that Regionserver
>>> (04.hadoop on 10.100.154.10) (note the last full GC only 1 minutes before @
>>> 5:03:29.850+0000):
>>>
>>> 2010-12-09T05:02:19.472+0000: 12170.437: [GC [PSYoungGen:
>>> 278229K->5024K(287360K)] 694269K->421693K(721856K), 0.0274840 secs] [Times:
>>> user=0.04 sys=0.00, real=0.03 secs]
>>> 2010-12-09T05:02:55.479+0000: 12206.444: [GC [PSYoungGen:
>>> 281450K->7968K(273344K)] 698120K->429658K(707840K), 0.0503000 secs] [Times:
>>> user=0.10 sys=0.00, real=0.05 secs]
>>> 2010-12-09T05:03:29.800+0000: 12240.766: [GC [PSYoungGen:
>>> 273295K->7712K(262464K)] 694986K->437306K(696960K), 0.0490770 secs] [Times:
>>> user=0.12 sys=0.01, real=0.05 secs]
>>> 2010-12-09T05:03:29.850+0000: 12240.815: [Full GC [PSYoungGen:
>>> 7712K->0K(262464K)] [PSOldGen: 429594K->353152K(429824K)]
>>> 437306K->353152K(692288K) [PSPermGen: 18016K->18016K(21248K)], 0.8960760
>>> secs] [Times: user=0.89 sys=0.00, real=0.90 secs]
>>> 2010-12-09T05:04:03.345+0000: 12274.310: [GC [PSYoungGen:
>>> 254720K->6944K(251584K)] 607872K->360528K(681408K), 0.0331060 secs] [Times:
>>> user=0.03 sys=0.00, real=0.04 secs]
>>> 2010-12-09T05:04:33.875+0000: 12304.840: [GC [PSYoungGen:
>>> 251552K->7200K(242240K)] 605136K->367800K(672064K), 0.0336420 secs] [Times:
>>> user=0.09 sys=0.01, real=0.03 secs]
>>> Heap
>>> PSYoungGen total 242240K, used 223390K [0x00007fcc640b0000,
>>> 0x00007fcc77a90000, 0x00007fcc78e00000)
>>> eden space 235008K, 91% used
>>> [0x00007fcc640b0000,0x00007fcc713cfa30,0x00007fcc72630000)
>>> from space 7232K, 99% used
>>> [0x00007fcc769b0000,0x00007fcc770b8090,0x00007fcc770c0000)
>>> to space 10048K, 0% used
>>> [0x00007fcc770c0000,0x00007fcc770c0000,0x00007fcc77a90000)
>>> PSOldGen total 429824K, used 360600K [0x00007fcc3a600000,
>>> 0x00007fcc549c0000, 0x00007fcc640b0000)
>>> object space 429824K, 83% used
>>> [0x00007fcc3a600000,0x00007fcc50626060,0x00007fcc549c0000)
>>> PSPermGen total 21248K, used 18131K [0x00007fcc35200000,
>>> 0x00007fcc366c0000, 0x00007fcc3a600000)
>>> object space 21248K, 85% used
>>> [0x00007fcc35200000,0x00007fcc363b4d60,0x00007fcc366c0000)
>>>
>>> Now Dead / End of log
>>>
>>>
>>> That said, I will continue down the path you pointed me on .
>>>
>>> Thanks,
>>> Lance
>>>
>>> On Dec 9, 2010, at 11:40 AM, Jean-Daniel Cryans wrote:
>>>
>>>> Lance,
>>>>
>>>> Both those lines indicate the problem:
>>>>
>>>> IPC Server handler 13 on 60020 took 182416ms
>>>> Client session timed out, have not heard from server in 182936ms
>>>>
>>>> It's very clear that your region servers are suffering from
>>>> pause-of-the-world garbage collection issues. Basically this one GC'ed
>>>> for 3 minutes, which is over the 1 minute timeout to consider a region
>>>> server dead. The rest is just the side effect of that.
>>>>
>>>> This subject comes often on the mailing list, for example:
>>>> http://search-hadoop.com/m/t97q027tr7f2
>>>>
>>>> Make sure you don't swap, give plenty of RAM to HBase, use LZO
>>>> compression, don't underprovision your machines, etc
>>>>
>>>> J-D
>>>>
>>>> On Thu, Dec 9, 2010 at 11:27 AM, Lance Riedel <la...@gmail.com>
>>> wrote:
>>>>>
>>>>> We have a 6 node cluster, 5 with region serves. 2 of the region servers
>>> have been stable for days, but 3 of them keep crashing. Here are the logs
>>> around around when the crash occurs. (btw, we are shoving approximately the
>>> twitter firehose into hbase via flume) I'm an hbase newbie, but I have been
>>> reading. Not sure what else is needed to help debug. When the problem
>>> occurs, it is concurrent with issues that appear all the way down in the
>>> data node. Also, interesting to note, all 3 servers seem to fail
>>> differently somewhat:
>>>>>
>>>>> Servers (6):
>>>>> 00.hadoop is the master
>>>>> 01.hadoop, 03.hadoop (there is no 02.hadoop - flaky machine that got
>>> taken out) - are stable Region Servers
>>>>> 04-06.hadoop crash at different times, very different logs, Region
>>> Servers
>>>>>
>>>>>
>>>>> Master Attributes:
>>>>> HBase Version 0.89.20100924+28, r
>>>>> Hadoop Version 0.20.2+737, r98c55c28258aa6f42250569bd7fa431ac657bdbd
>>>>>
>>>>> HBase Root Directory hdfs://00.hadoop.****:54310/hbase
>>>>>
>>>>> Load average
>>>>> 103.0
>>>>> Average number of regions per regionserver. Naive computation.
>>>>> Regions On FS
>>>>> 577
>>>>>
>>>>>
>>>>> (This is after recently restarting 04-06)
>>>>>
>>>>>
>>>>> Address Start Code Load
>>>>> 01.hadoop:60030 1291268910638 requests=223, regions=118,
>>> usedHeap=519, maxHeap=987
>>>>> 03.hadoop:60030 1291269219610 requests=23, regions=111, usedHeap=862,
>>> maxHeap=987
>>>>> 04.hadoop:60030 1291910665912 requests=169, regions=82, usedHeap=194,
>>> maxHeap=888
>>>>> 05.hadoop:60030 1291909584060 requests=232, regions=110,
>>> usedHeap=477, maxHeap=888
>>>>> 06.hadoop60030 1291909723787 requests=99, regions=94, usedHeap=394,
>>> maxHeap=888
>>>>> Total:
>>>>> servers: 5
>>>>> requests=746, regions=515
>>>>>
>>>>>
>>>>>
>>>>> Following are various logs around the time of the failures for 04-06
>>>>>
>>>>>
>>> ************************************************************************************
>>>>> 04.hadoop
>>>>>
>>> ************************************************************************************
>>>>> REGION SERVER ->
>>>>> 2010-12-09 05:01:58,640 INFO
>>> org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region
>>> article,a83858a08f2270d319f75a7b43c756c2453988e7,1291811929868.58f6d9fc80
>>>>> c78f3ca490b0280b4f1226. in 0sec
>>>>> 2010-12-09 05:04:30,898 DEBUG
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.46 MB,
>>> free=176.33 MB, max=177.79 MB, blocks=0, accesses=200583, hits=0, hitRatio=
>>>>> 0.00%%, evictions=0, evicted=0, evictedPerRun=NaN
>>>>> 2010-12-09 05:04:59,612 INFO
>>> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs
>>> -- HDFS-200
>>>>> 2010-12-09 05:04:59,618 INFO
>>> org.apache.hadoop.hbase.regionserver.wal.HLog: Roll
>>> /hbase/.logs/04.hadoop.****,60020,1291858770125/10.100.154.103%3A60020.1291870585253,
>>> entries=5318
>>>>> 6, filesize=63776311. New hlog
>>> /hbase/.logs/04.hadoop.****,60020,1291858770125/10.100.154.103
>>> %3A60020.1291871099582
>>>>> 2010-12-09 05:08:02,033 DEBUG
>>> org.apache.hadoop.hbase.regionserver.wal.HLog: Found 1 hlogs to remove out
>>> of total 6; oldest outstanding sequenceid is 32891801 from region article
>>>>>
>>> ,4b2039b791e894dd479b90661ca97087f61645d3,1291813201039.7dff47c63bb5648bb1ee6670c60553a5.
>>>>> 2010-12-09 05:08:02,034 WARN
>>> org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler 13 on
>>> 60020 took 182416ms appending an edit to hlog; editcount=0
>>>>> 2010-12-09 05:08:02,066 INFO
>>> org.apache.hadoop.hbase.regionserver.wal.HLog: moving old hlog file
>>> /hbase/.logs/04.hadoop.****,60020,1291858770125/10.100.154.103
>>> %3A60020.12918679476
>>>>> 65 whose highest sequenceid is 32880227 to /hbase/.oldlogs/
>>> 10.100.154.103%3A60020.1291867947665
>>>>> 2010-12-09 05:08:02,066 INFO org.apache.zookeeper.ClientCnxn: Client
>>> session timed out, have not heard from server in 182936ms for sessionid
>>> 0x12ca58c08ff0048, closing socket conn
>>>>> ection and attempting reconnect
>>>>> 2010-12-09 05:08:02,089 FATAL
>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Aborting region server
>>> serverName=04.hadoop.****,60020,1291858770125, load=(requests=24, regions=
>>>>> 64, usedHeap=548, maxHeap=888): Unhandled exception
>>>>> org.apache.hadoop.hbase.YouAreDeadException:
>>> org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected;
>>> currently processing 04.hadoop.****,60020,1291858770125 as dead s
>>>>> erver
>>>>> at
>>> org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:217)
>>>>> at
>>> org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:271)
>>>>> at
>>> org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:744)
>>>>> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>> at
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:561)
>>>>> at
>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>>>>>
>>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>> Method)
>>>>> at
>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>>>>> at
>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>>>>> at
>>> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
>>>>> at
>>> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
>>>>> at
>>> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
>>>>> at
>>> org.apache.hadoop.hbase.regionserver.HRegionServer.doRun(HRegionServer.java:635)
>>>>> at
>>> org.apache.hadoop.hbase.regionserver.HRegionServer.access$000(HRegionServer.java:126)
>>>>> at
>>> org.apache.hadoop.hbase.regionserver.HRegionServer$1.run(HRegionServer.java:518)
>>>>> at
>>> org.apache.hadoop.hbase.regionserver.HRegionServer$1.run(HRegionServer.java:516)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at javax.security.auth.Subject.doAs(Subject.java:337)
>>>>> at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1046)
>>>>> at
>>> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:516)
>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>> 2010-12-09 05:08:02,090 INFO
>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
>>> request=0.0, regions=64, stores=64, storefiles=136, storefileIndexSize=27,
>>> memsto
>>>>> reSize=353, compactionQueueSize=0, usedHeap=549, maxHeap=888,
>>> blockCacheSize=1530552, blockCacheFree=184893160, blockCacheCount=0,
>>> blockCacheHitRatio=0
>>>>> 2010-12-09 05:08:02,090 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
>>> server on 60020
>>>>> 2010-12-09 05:08:02,090 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>> Server handler 0 on 60020: exiting2010-12-09 05:08:02,090 INFO
>>> org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server listener on 60020
>>>>>
>>>>>
>>>>> MASTER HBASE ->
>>>>>
>>>>> 2010-12-09 05:05:06,072 INFO org.apache.hadoop.hbase.master.BaseScanner:
>>> All 1 .META. region(s) scanned
>>>>> 2010-12-09 05:05:40,004 INFO
>>> org.apache.hadoop.hbase.master.ServerManager:
>>> 04.hadoop.jive,60020,1291858770125 znode expired
>>>>> 2010-12-09 05:05:40,005 DEBUG
>>> org.apache.hadoop.hbase.master.ServerManager:
>>> Added=04.hadoop.jive,60020,1291858770125 to dead servers, added shutdown
>>> processing operation
>>>>> 2010-12-09 05:05:40,005 DEBUG
>>> org.apache.hadoop.hbase.master.RegionServerOperationQueue: Processing todo:
>>> ProcessServerShutdown of 04.hadoop.jive,60020,12918587701252010-12-09
>>> 05:05:40,005 INFO org.apache.hadoop.hbase.master.RegionServerOperation:
>>> Process shutdown of server 04.hadoop.jive,60020,1291858770125: logSplit:
>>> false, rootRescanned: f
>>>>> alse, numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
>>>>> 2010-12-09 05:05:40,008 INFO
>>> org.apache.hadoop.hbase.regionserver.wal.HLog: Splitting 7 hlog(s) in
>>> hdfs://00.hadoop.jive:54310/hbase/.logs/04.hadoop.jive,60020,12918587701252010-12-09
>>> 05:05:40,008 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: Splitting
>>> hlog 1 of 7:
>>> hdfs://00.hadoop.jive:54310/hbase/.logs/04.hadoop.jive,60020,1291858770125/10.1
>>>>> 00.154.103%3A60020.1291867947665, length=637577092010-12-09 05:05:40,008
>>> INFO org.apache.hadoop.hbase.util.FSUtils: Recovering
>>> filehdfs://00.hadoop.jive:54310/hbase/.logs/04.hadoop.jive,60020,1291858770125/
>>> 10.100.154.103%3A60020
>>>>> .1291867947665
>>>>>
>>>>>
>>>>> NAMENODE ->
>>>>>
>>>>> 2010-12-09 05:08:02,471 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
>>> NameSystem.addStoredBlock: blockMap updated: 10.100.159.13:50010 is added
>>> to blk_1531008743226086399_251615 size 63757709
>>>>> 2010-12-09 05:08:02,473 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 9 on 54310, call
>>> complete(/hbase/.logs/04.hadoop.jive,60020,1291858770125/10.100.154.103%3A60020.1291867947665,
>>> DFSClient_hb_m_10.194.194.79:60000_1291788452343) from 10.194.194.79:44117:
>>> error: org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No
>>> lease on /hbase/.logs/04.hadoop.jive,60020,1291858770125/10.100.154.103%3A60020.1291867947665
>>> File does not exist. [Lease. Holder:
>>> DFSClient_hb_m_10.194.194.79:60000_1291788452343, pendingcreates: 1]
>>>>> org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease
>>> on /hbase/.logs/04.hadoop.jive,60020,1291858770125/10.100.154.103%3A60020.1291867947665
>>> File does not exist. [Lease. Holder:
>>> DFSClient_hb_m_10.194.194.79:60000_1291788452343, pendingcreates: 1]
>>>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1488)
>>>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1479)
>>>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:1534)
>>>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:1522)
>>>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.complete(NameNode.java:610)
>>>>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>>>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:528)
>>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1319)
>>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1315)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>>>> at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1313)
>>>>> 2010-12-09 05:08:04,206 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
>>> NameSystem.addStoredBlock: blockMap updated: 10.99.97.106:50010 is added
>>> to blk_-734280257049179934_251614 size 7330
>>>>>
>>>>> DATANODE ->
>>>>>
>>>>> 2010-12-09 05:08:02,212 WARN
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in
>>> BlockReceiver.lastNodeRun: java.io.IOException: Broken pipe
>>>>> at sun.nio.ch.FileDispatcher.write0(Native Method)
>>>>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
>>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100) at
>>> sun.nio.ch.IOUtil.write(IOUtil.java:71)
>>>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
>>>>> at
>>> org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
>>> at
>>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
>>> at
>>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
>>>>> at
>>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
>>> at java.io.DataOutputStream.writeLong(DataOutputStream.java:207)
>>>>> at
>>> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.write(DataTransferProtocol.java:133)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.lastDataNodeRun(BlockReceiver.java:840)
>>>>> at
>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:870)
>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>> 2010-12-09 05:08:02,213 WARN
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: checkDiskError: exception:
>>>>> 2010-12-09 05:08:02,213 WARN
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: checkDiskError: exception:
>>> java.io.IOException: Broken pipe
>>>>> at sun.nio.ch.FileDispatcher.write0(Native Method)
>>>>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
>>>>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100)
>>>>> at sun.nio.ch.IOUtil.write(IOUtil.java:71)
>>>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
>>>>> at
>>> org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
>>>>> at
>>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
>>>>> at
>>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
>>>>> at
>>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
>>>>> at java.io.DataOutputStream.writeLong(DataOutputStream.java:207)
>>>>> at
>>> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.write(DataTransferProtocol.java:133)
>>>>> at
>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.lastDataNodeRun(BlockReceiver.java:840)
>>>>> at
>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:870)
>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>> 2010-12-09 05:08:02,221 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder
>>> blk_-8817504198034990390_251613 0 Exception java.io.IOException: Broken pipe
>>>>> at sun.nio.ch.FileDispatcher.write0(Native Method)
>>>>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
>>>>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100)
>>>>> at sun.nio.ch.IOUtil.write(IOUtil.java:71)
>>>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
>>>>> at
>>> org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
>>>>> at
>>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
>>>>> at
>>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
>>>>> at
>>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
>>>>> at java.io.DataOutputStream.writeLong(DataOutputStream.java:207)
>>>>> at
>>> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.write(DataTransferProtocol.java:133)
>>>>> at
>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.lastDataNodeRun(BlockReceiver.java:840)
>>>>> at
>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:870)
>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> 2010-12-09 05:08:02,221 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block
>>> blk_-8817504198034990390_251613 terminating
>>>>> 2010-12-09 05:08:02,224 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock
>>> blk_-8817504198034990390_251613 received exception java.io.IOException:
>>> Connection reset by peer
>>>>> 2010-12-09 05:08:02,224 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block
>>> blk_4505306716668305567_250752 file
>>> /dist1/data/hadoop-data/current/subdir19/blk_4505306716668305567 for
>>> deletion
>>>>> 2010-12-09 05:08:02,226 ERROR
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
>>> 10.100.154.103:50010,
>>> storageID=DS-866555524-10.100.154.103-50010-1291262739506, infoPort=50075,
>>> ipcPort=50020):DataXceiver
>>>>> java.io.IOException: Connection reset by peer
>>>>> at sun.nio.ch.FileDispatcher.read0(Native Method)
>>>>> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
>>>>> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:237)
>>>>> at sun.nio.ch.IOUtil.read(IOUtil.java:210)
>>>>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
>>>>> at
>>> org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:55)
>>>>> at
>>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
>>>>> at
>>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>>>> at
>>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>>>> at
>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
>>>>> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
>>>>> at java.io.DataInputStream.read(DataInputStream.java:132)
>>>>> at
>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:267)
>>>>> at
>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:357)
>>>>> at
>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:378)
>>>>> at
>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:534)
>>>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:417)
>>>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:122)
>>>>> 2010-12-09 05:08:02,226 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block
>>> blk_1422641301942366074_250732 at file
>>> /dist1/data/hadoop-data/current/subdir19/blk_1422641301942366074
>>>>> 2010-12-09 05:08:02,227 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block
>>> blk_4505306716668305567_250752 at file
>>> /dist1/data/hadoop-data/current/subdir19/blk_4505306716668305567
>>>>>
>>>>>
>>>>>
>>>>>
>>> ************************************************************************************
>>>>> 05.hadoop
>>>>>
>>> ************************************************************************************
>>>>> REGION SERVER ->
>>>>> 2010-12-09 04:29:25,047 DEBUG
>>> org.apache.hadoop.hbase.regionserver.Store: Compaction size of data: 250.5m;
>>> Skipped 1 file(s), size: 206896367
>>>>> 2010-12-09 04:29:25,047 INFO org.apache.hadoop.hbase.regionserver.Store:
>>> Started compaction of 2 file(s) in data of
>>> article,060c4b75067378c045eead9edec2d4fb7d8f3e5e,1291812563166.95e92e97823e11e57850720a63433a62.
>>> into
>>> hdfs://00.hadoop.****:54310/hbase/article/95e92e97823e11e57850720a63433a62/.tmp,
>>> sequenceid=33050221
>>>>> 2010-12-09 04:29:26,328 INFO org.apache.hadoop.hbase.regionserver.Store:
>>> Completed compaction of 2 file(s) in data of
>>> article,060c4b75067378c045eead9edec2d4fb7d8f3e5e,1291812563166.95e92e97823e11e57850720a63433a62.;
>>> new storefile is
>>> hdfs://00.hadoop.****:54310/hbase/article/95e92e97823e11e57850720a63433a62/data/2787768957794924068;
>>> store size is 250.5m
>>>>> 2010-12-09 04:29:26,328 INFO
>>> org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region
>>> article,060c4b75067378c045eead9edec2d4fb7d8f3e5e,1291812563166.95e92e97823e11e57850720a63433a62.
>>> in 1sec
>>>>> 2010-12-09 04:29:26,328 INFO
>>> org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on region
>>> article,ed0cdc7ae6db73432e2ca47c973d545bee11ddd8,1291813400028.85a8d31d099f409dd36f3bcc4be1eba6.
>>>>> 2010-12-09 04:29:26,330 INFO
>>> org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region
>>> article,ed0cdc7ae6db73432e2ca47c973d545bee11ddd8,1291813400028.85a8d31d099f409dd36f3bcc4be1eba6.
>>> in 0sec
>>>>> 2010-12-09 04:32:33,081 INFO
>>> org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGIONSERVER_STOP
>>>>> 2010-12-09 04:32:33,081 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
>>> server on 60020
>>>>> 2010-12-09 04:32:33,089 WARN org.apache.hadoop.hdfs.DFSClient:
>>> DFSOutputStream ResponseProcessor exception for block
>>> blk_2012253533858803341_250248java.io.EOFException
>>>>> at java.io.DataInputStream.readFully(DataInputStream.java:180)
>>>>> at java.io.DataInputStream.readLong(DataInputStream.java:399)
>>>>> at
>>> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:120)
>>>>> at
>>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2621)
>>>>>
>>>>> 2010-12-09 04:32:33,089 INFO org.apache.zookeeper.ClientCnxn: Unable to
>>> read additional data from server sessionid 0x12ca58c08ff0049, likely server
>>> has closed socket, closing sock
>>>>> et connection and attempting reconnect
>>>>> 2010-12-09 04:32:33,089 WARN
>>> org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler 5 on 60020
>>> took 75446ms appending an edit to hlog; editcount=50505
>>>>> 2010-12-09 04:32:33,090 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>>> Server handler 18 on 60020: exiting
>>>>> 2010-12-09 04:32:33,139 INFO org.apache.hadoop.hdfs.DFSClient: Error
>>> Recovery for block blk_2012253533858803341_250248 waiting for responder to
>>> exit.
>>>>> 2010-12-09 04:32:33,139 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>>> Server handler 8 on 60020: exiting
>>>>> 2010-12-09 04:32:33,139 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>> Server handler 1 on 60020: exiting
>>>>> 2010-12-09 04:32:33,139 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>>> Server handler 10 on 60020: exiting
>>>>> 2010-12-09 04:32:33,140 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>> Server handler 0 on 60020: exiting2010-12-09 04:32:33,140 INFO
>>> org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 5 on 60020:
>>> exiting
>>>>> 2010-12-09 04:32:33,139 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>> Server handler 8 on 60020: exiting2010-12-09 04:32:33,140 INFO
>>> org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 4 on 60020:
>>> exiting
>>>>> 2010-12-09 04:32:33,140 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>> Server handler 3 on 60020: exiting2010-12-09 04:32:33,142 INFO
>>> org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 7 on 60020:
>>> exiting
>>>>> 2010-12-09 04:32:33,142 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>> Server handler 9 on 60020: exiting2010-12-09 04:32:33,143 INFO
>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
>>>>> 2010-12-09 04:32:33,143 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>> Server handler 2 on 60020: exiting
>>>>> 2010-12-09 04:32:33,143 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
>>> IPC Server Responder
>>>>> 2010-12-09 04:32:33,143 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>> Server handler 6 on 60020: exiting
>>>>> 2010-12-09 04:32:33,144 INFO org.mortbay.log: Stopped
>>> SelectChannelConnector@0.0.0.0:60030
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> NAMENODE ->
>>>>> 2010-12-09 04:32:08,408 WARN org.apache.hadoop.hdfs.StateChange: DIR*
>>> NameSystem.startFile: failed to create file
>>> /hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555
>>> for DFSClient_hb_m_10.194.194.79:60000_1291788452343 on client
>>> 10.194.194.79, because this file is already being created by NN_Recovery on
>>> 10.100.159.13
>>>>> 2010-12-09 04:32:08,408 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 6 on 54310, call
>>> append(/hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555,
>>> DFSClient_hb_m_10.194.194.79:60000_1291788452343) from 10.194.194.79:59118:
>>> error: org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed
>>> to create file /hba
>>>>> se/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555
>>> for DFSClient_hb_m_10.194.194.79:60000_1291788452343 on client
>>> 10.194.194.79, because this file is already being created by NN_Recovery on
>>> 10.100.159.13
>>>>> org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to
>>> create file /hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555
>>> for DFSClient_hb_m_10.194.194.79:60000_1291788452343 on client
>>> 10.194.194.79, because this file is already being created by NN_Recovery on
>>> 10.100.159.13
>>>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1194)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:1282)
>>>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.append(NameNode.java:541)
>>> at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
>>>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:528) at
>>> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1319)
>>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1315)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at javax.security.auth.Subject.doAs(Subject.java:396) at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
>>>>> at
>>> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1313)2010-12-09
>>> 04:32:09,411 WARN org.apache.hadoop.hdfs.StateChange: DIR*
>>> NameSystem.startFile: failed to create file
>>> /hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A6
>>>>> 0020.1291868481555 for DFSClient_hb_m_10.194.194.79:60000_1291788452343
>>> on client 10.194.194.79, because this file is already being created by
>>> NN_Recovery on 10.100.159.132010-12-09 04:32:09,411 INFO
>>> org.apache.hadoop.ipc.Server: IPC Server handler 9 on 54310, call
>>> append(/hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13
>>> %3A60020.1291868
>>>>> 481555, DFSClient_hb_m_10.194.194.79:60000_1291788452343) from
>>> 10.194.194.79:59118: error:
>>> org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to
>>> create file /hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555
>>> for DFSClient_hb_m_10.194.194.79:60000_1291788452343 on client
>>> 10.194.194.79, because this file is
>>>>> already being created by NN_Recovery on
>>> 10.100.159.13org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException:
>>> failed to create file /hbase/.logs/05.hadoop.jive,60020,1291862294517/
>>> 10.100.159.13%3A60020.1291868481555 for DFSClie
>>>>> nt_hb_m_10.194.194.79:60000_1291788452343 on client 10.194.194.79,
>>> because this file is already being created by NN_Recovery on 10.100.159.13
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1194)
>>>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:1282)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.append(NameNode.java:541)
>>>>> at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>> at java.lang.reflect.Method.invoke(Method.java:597) at
>>> org.apache.hadoop.ipc.RPC$Server.call(RPC.java:528)
>>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1319)
>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1315)
>>>>>
>>>>>
>>>>>
>>>>> DATANODE
>>>>>
>>>>> 2010-12-09 04:32:09,534 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder
>>> blk_2012253533858803341_250248 1 : Thread is interrupted.
>>>>> 2010-12-09 04:32:09,534 ERROR
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
>>> 10.100.159.13:50010,
>>> storageID=DS-145997007-10.100.159.13-50010-1291262743182, infoPort=50075,
>>> ipcPort=50020):DataXceiver
>>>>> java.io.IOException: Interrupted receiveBlock
>>>>> at
>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:579)
>>>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:417)
>>>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:122)
>>>>> 2010-12-09 04:32:09,534 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 1 for block
>>> blk_2012253533858803341_250248 terminating
>>>>>
>>>>>
>>>>>
>>> ************************************************************************************
>>>>> 06.hadoop
>>>>>
>>> ************************************************************************************
>>>>> REGION SERVER ->
>>>>>
>>>>> 2010-12-09 06:17:47,215 DEBUG
>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>> NotServingRegionException;
>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>> 2010-12-09 06:17:47,430 DEBUG
>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>> NotServingRegionException;
>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>> 2010-12-09 06:17:47,549 DEBUG
>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>> NotServingRegionException;
>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>> 2010-12-09 06:17:47,557 DEBUG
>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>> NotServingRegionException;
>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>> 2010-12-09 06:17:47,563 DEBUG
>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>> NotServingRegionException;
>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>> 2010-12-09 06:17:48,619 DEBUG
>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>> NotServingRegionException;
>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>> 2010-12-09 06:17:48,619 DEBUG
>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>> NotServingRegionException;
>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>> 2010-12-09 06:17:48,619 DEBUG
>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>> NotServingRegionException;
>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>> 2010-12-09 06:17:48,619 DEBUG
>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>> NotServingRegionException;
>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>> 2010-12-09 06:17:48,619 DEBUG
>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>> NotServingRegionException;
>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>> 2010-12-09 06:17:49,703 DEBUG
>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>> NotServingRegionException;
>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>> 2010-12-09 06:17:49,703 DEBUG
>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>> NotServingRegionException;
>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>> 2010-12-09 06:17:49,703 DEBUG
>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>> NotServingRegionException;
>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>> 2010-12-09 06:17:49,705 DEBUG
>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>> NotServingRegionException;
>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>> 2010-12-09 06:17:49,729 DEBUG
>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>> NotServingRegionException;
>>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>>
>>>>> A bunch of these ^
>>>>>
>>>>> Then a little later::
>>>>> ....
>>>>>
>>>>> 2010-12-09 06:23:23,561 DEBUG
>>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction
>>> requested for region
>>> article,993143fca3613014990cc6975cacdd62f4147e03,12918133247
>>>>> 53.19413f33fc660536d9d4f315a1283440. because:
>>> regionserver60020.cacheFlusher
>>>>> 2010-12-09 06:23:27,058 INFO
>>> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs
>>> -- HDFS-200
>>>>> 2010-12-09 06:23:27,080 INFO
>>> org.apache.hadoop.hbase.regionserver.wal.HLog: Roll
>>> /hbase/.logs/06.hadoop.****,60020,1291864620851/10.99.97.106%3A60020.1291875047947,
>>> entries=53315,
>>>>> filesize=63781925. New hlog
>>> /hbase/.logs/06.hadoop.****,60020,1291864620851/10.99.97.106
>>> %3A60020.1291875807006
>>>>> 2010-12-09 06:23:27,080 DEBUG
>>> org.apache.hadoop.hbase.regionserver.wal.HLog: Found 1 hlogs to remove out
>>> of total 8; oldest outstanding sequenceid is 33526881 from region article
>>>>>
>>> ,1221b0013c61fd7446cb822f698d0bcafa487342,1291872610788.bb2aa6ae8c694a096a00e7bb91fc46c3.
>>>>> 2010-12-09 06:23:27,080 INFO
>>> org.apache.hadoop.hbase.regionserver.wal.HLog: moving old hlog file
>>> /hbase/.logs/06.hadoop.****,60020,1291864620851/10.99.97.106
>>> %3A60020.1291872064424
>>>>> whose highest sequenceid is 33506639 to /hbase/.oldlogs/10.99.97.106
>>> %3A60020.1291872064424
>>>>> 2010-12-09 06:25:39,162 WARN
>>> org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler 12 on
>>> 60020 took 119034ms appending an edit to hlog; editcount=3683
>>>>> 2010-12-09 06:25:39,162 WARN org.apache.hadoop.hdfs.DFSClient:
>>> DataStreamer Exception: java.io.IOException: Broken pipe
>>>>> at sun.nio.ch.FileDispatcher.write0(Native Method)
>>>>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
>>>>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100)
>>>>> at sun.nio.ch.IOUtil.write(IOUtil.java:71)
>>>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
>>>>> at
>>> org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
>>>>> at
>>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
>>>>> at
>>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
>>>>> at
>>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
>>>>> at
>>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
>>>>> at java.io.DataOutputStream.write(DataOutputStream.java:90)
>>>>> at
>>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2509)
>>>>> 2010-12-09 06:25:39,163 WARN org.apache.hadoop.hdfs.DFSClient:
>>> DFSOutputStream ResponseProcessor exception for block
>>> blk_1141147913164105086_258331java.io.EOFException
>>>>> at java.io.DataInputStream.readFully(DataInputStream.java:180)
>>>>> at java.io.DataInputStream.readLong(DataInputStream.java:399)
>>>>> at
>>> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:120)
>>>>> at
>>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2621)
>>>>>
>>>>> 2010-12-09 06:25:39,163 INFO org.apache.hadoop.hdfs.DFSClient: Error
>>> Recovery for block blk_1141147913164105086_258331 waiting for responder to
>>> exit.
>>>>> 2010-12-09 06:25:39,164 WARN org.apache.hadoop.hdfs.DFSClient:
>>> DFSOutputStream ResponseProcessor exception for block
>>> blk_-7076574401277381592_258324java.io.EOFException
>>>>> at java.io.DataInputStream.readFully(DataInputStream.java:180)
>>>>> at java.io.DataInputStream.readLong(DataInputStream.java:399)
>>>>> at
>>> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:120)
>>>>> at
>>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2621)
>>>>>
>>>>> 2010-12-09 06:25:39,164 WARN org.apache.hadoop.hdfs.DFSClient: Error
>>> Recovery for block blk_-7076574401277381592_258324 bad datanode[0]
>>> 10.99.97.106:50010
>>>>> 2010-12-09 06:25:39,164 INFO org.apache.zookeeper.ClientCnxn: Unable to
>>> read additional data from server sessionid 0x12ca58c08ff004b, likely server
>>> has closed socket, closing socket connection and attempting reconnect
>>>>> 2010-12-09 06:25:39,164 WARN org.apache.hadoop.hdfs.DFSClient: Error
>>> Recovery for block blk_-7076574401277381592_258324 in pipeline
>>> 10.99.97.106:50010, 10.100.154.103:50010: bad datanode 10.99.97.106:50010
>>>>> 2010-12-09 06:25:39,165 WARN org.apache.hadoop.hdfs.DFSClient: Error
>>> Recovery for block blk_1141147913164105086_258331 bad datanode[0]
>>> 10.99.97.106:50010
>>>>> 2010-12-09 06:25:39,167 INFO org.apache.zookeeper.ClientCnxn: Unable to
>>> read additional data from server sessionid 0x12ca58c08ff004c, likely server
>>> has closed socket, closing socket connection and attempting reconnect
>>>>> 2010-12-09 06:25:39,169 INFO org.apache.hadoop.ipc.Client: Retrying
>>> connect to server: /10.100.154.103:50020. Already tried 0 time(s).
>>>>> 2010-12-09 06:25:39,169 WARN org.apache.hadoop.hdfs.DFSClient: Error
>>> Recovery for block blk_1141147913164105086_258331 in pipeline
>>> 10.99.97.106:50010, 10.100.159.13:50010: bad datanode 10.99.97.106:50010
>>>>> 2010-12-09 06:25:39,209 INFO
>>> org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGIONSERVER_STOP
>>>>> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
>>> server on 60020
>>>>> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>>> Server handler 3 on 60020: exiting
>>>>> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>>> Server handler 9 on 60020: exiting
>>>>> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>>> Server handler 24 on 60020: exiting
>>>>> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>> Server handler 1 on 60020: exiting
>>>>> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>>> Server handler 15 on 60020: exiting
>>>>> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>> Server handler 0 on 60020: exiting
>>>>> 2010-12-09 06:25:39,214 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>> Server handler 3 on 60020: exiting
>>>>> 2010-12-09 06:25:39,214 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
>>> IPC Server listener on 60020
>>>>> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>> Server handler 2 on 60020: exiting
>>>>> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>> Server handler 7 on 60020: exiting
>>>>> 2010-12-09 06:25:39,213 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>>> Server handler 21 on 60020: exiting
>>>>> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>> Server handler 4 on 60020: exiting
>>>>> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
>>> IPC Server Responder
>>>>> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>> Server handler 5 on 60020: exiting
>>>>> 2010-12-09 06:25:39,216 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>> Server handler 8 on 60020: exiting
>>>>> 2010-12-09 06:25:39,216 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>> Server handler 9 on 60020: exiting
>>>>> 2010-12-09 06:25:39,216 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>>> Server handler 6 on 60020: exiting
>>>>> 2010-12-09 06:25:39,216 INFO
>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
>>>>> 2010-12-09 06:25:39,223 WARN org.apache.hadoop.hdfs.DFSClient: Failed
>>> recovery attempt #0 from primary datanode 10.100.159.13:50010
>>>>>
>>>>>
>>>>>
>>>>>
>>>
>>>
>
Re: Crashing Region Servers
Posted by Lance Riedel <la...@gmail.com>.
I came across just probably the culprit - I just discovered that the machines that were dying had OS swap turned OFF.
So, I added swap to those machines, but also reduced the memory configs for tasks. (was at 1 gig, now at 512m). Seems to be stable right now, but the nightly longer term jobs run at night, so we will see how they do.
This is an ec2 c1.xlarge, so the memory is at 8 gigs. I had 4 mappers per node, which was fine when all they had on them were Tasktrackers and datanodes. But with the hbase regionservers, the machines could definitely get over-allocated.
Lance
On Dec 9, 2010, at 3:57 PM, Ted Dunning wrote:
> This could indicate swapping during GC.
>
> On Thu, Dec 9, 2010 at 12:13 PM, Lance Riedel <la...@gmail.com> wrote:
>
>> Seems reasonable, but having trouble making sense of the GC logs I had
>> turned on. Basically since there was a full GC a minute before this happens
>> on that server that lasts less than a second.
>>
>>
>> Example:
>> So, here is what the last of the GC logs say for that Regionserver
>> (04.hadoop on 10.100.154.10) (note the last full GC only 1 minutes before @
>> 5:03:29.850+0000):
>>
>> 2010-12-09T05:02:19.472+0000: 12170.437: [GC [PSYoungGen:
>> 278229K->5024K(287360K)] 694269K->421693K(721856K), 0.0274840 secs] [Times:
>> user=0.04 sys=0.00, real=0.03 secs]
>> 2010-12-09T05:02:55.479+0000: 12206.444: [GC [PSYoungGen:
>> 281450K->7968K(273344K)] 698120K->429658K(707840K), 0.0503000 secs] [Times:
>> user=0.10 sys=0.00, real=0.05 secs]
>> 2010-12-09T05:03:29.800+0000: 12240.766: [GC [PSYoungGen:
>> 273295K->7712K(262464K)] 694986K->437306K(696960K), 0.0490770 secs] [Times:
>> user=0.12 sys=0.01, real=0.05 secs]
>> 2010-12-09T05:03:29.850+0000: 12240.815: [Full GC [PSYoungGen:
>> 7712K->0K(262464K)] [PSOldGen: 429594K->353152K(429824K)]
>> 437306K->353152K(692288K) [PSPermGen: 18016K->18016K(21248K)], 0.8960760
>> secs] [Times: user=0.89 sys=0.00, real=0.90 secs]
>> 2010-12-09T05:04:03.345+0000: 12274.310: [GC [PSYoungGen:
>> 254720K->6944K(251584K)] 607872K->360528K(681408K), 0.0331060 secs] [Times:
>> user=0.03 sys=0.00, real=0.04 secs]
>> 2010-12-09T05:04:33.875+0000: 12304.840: [GC [PSYoungGen:
>> 251552K->7200K(242240K)] 605136K->367800K(672064K), 0.0336420 secs] [Times:
>> user=0.09 sys=0.01, real=0.03 secs]
>> Heap
>> PSYoungGen total 242240K, used 223390K [0x00007fcc640b0000,
>> 0x00007fcc77a90000, 0x00007fcc78e00000)
>> eden space 235008K, 91% used
>> [0x00007fcc640b0000,0x00007fcc713cfa30,0x00007fcc72630000)
>> from space 7232K, 99% used
>> [0x00007fcc769b0000,0x00007fcc770b8090,0x00007fcc770c0000)
>> to space 10048K, 0% used
>> [0x00007fcc770c0000,0x00007fcc770c0000,0x00007fcc77a90000)
>> PSOldGen total 429824K, used 360600K [0x00007fcc3a600000,
>> 0x00007fcc549c0000, 0x00007fcc640b0000)
>> object space 429824K, 83% used
>> [0x00007fcc3a600000,0x00007fcc50626060,0x00007fcc549c0000)
>> PSPermGen total 21248K, used 18131K [0x00007fcc35200000,
>> 0x00007fcc366c0000, 0x00007fcc3a600000)
>> object space 21248K, 85% used
>> [0x00007fcc35200000,0x00007fcc363b4d60,0x00007fcc366c0000)
>>
>> Now Dead / End of log
>>
>>
>> That said, I will continue down the path you pointed me on .
>>
>> Thanks,
>> Lance
>>
>> On Dec 9, 2010, at 11:40 AM, Jean-Daniel Cryans wrote:
>>
>>> Lance,
>>>
>>> Both those lines indicate the problem:
>>>
>>> IPC Server handler 13 on 60020 took 182416ms
>>> Client session timed out, have not heard from server in 182936ms
>>>
>>> It's very clear that your region servers are suffering from
>>> pause-of-the-world garbage collection issues. Basically this one GC'ed
>>> for 3 minutes, which is over the 1 minute timeout to consider a region
>>> server dead. The rest is just the side effect of that.
>>>
>>> This subject comes often on the mailing list, for example:
>>> http://search-hadoop.com/m/t97q027tr7f2
>>>
>>> Make sure you don't swap, give plenty of RAM to HBase, use LZO
>>> compression, don't underprovision your machines, etc
>>>
>>> J-D
>>>
>>> On Thu, Dec 9, 2010 at 11:27 AM, Lance Riedel <la...@gmail.com>
>> wrote:
>>>>
>>>> We have a 6 node cluster, 5 with region serves. 2 of the region servers
>> have been stable for days, but 3 of them keep crashing. Here are the logs
>> around around when the crash occurs. (btw, we are shoving approximately the
>> twitter firehose into hbase via flume) I'm an hbase newbie, but I have been
>> reading. Not sure what else is needed to help debug. When the problem
>> occurs, it is concurrent with issues that appear all the way down in the
>> data node. Also, interesting to note, all 3 servers seem to fail
>> differently somewhat:
>>>>
>>>> Servers (6):
>>>> 00.hadoop is the master
>>>> 01.hadoop, 03.hadoop (there is no 02.hadoop - flaky machine that got
>> taken out) - are stable Region Servers
>>>> 04-06.hadoop crash at different times, very different logs, Region
>> Servers
>>>>
>>>>
>>>> Master Attributes:
>>>> HBase Version 0.89.20100924+28, r
>>>> Hadoop Version 0.20.2+737, r98c55c28258aa6f42250569bd7fa431ac657bdbd
>>>>
>>>> HBase Root Directory hdfs://00.hadoop.****:54310/hbase
>>>>
>>>> Load average
>>>> 103.0
>>>> Average number of regions per regionserver. Naive computation.
>>>> Regions On FS
>>>> 577
>>>>
>>>>
>>>> (This is after recently restarting 04-06)
>>>>
>>>>
>>>> Address Start Code Load
>>>> 01.hadoop:60030 1291268910638 requests=223, regions=118,
>> usedHeap=519, maxHeap=987
>>>> 03.hadoop:60030 1291269219610 requests=23, regions=111, usedHeap=862,
>> maxHeap=987
>>>> 04.hadoop:60030 1291910665912 requests=169, regions=82, usedHeap=194,
>> maxHeap=888
>>>> 05.hadoop:60030 1291909584060 requests=232, regions=110,
>> usedHeap=477, maxHeap=888
>>>> 06.hadoop60030 1291909723787 requests=99, regions=94, usedHeap=394,
>> maxHeap=888
>>>> Total:
>>>> servers: 5
>>>> requests=746, regions=515
>>>>
>>>>
>>>>
>>>> Following are various logs around the time of the failures for 04-06
>>>>
>>>>
>> ************************************************************************************
>>>> 04.hadoop
>>>>
>> ************************************************************************************
>>>> REGION SERVER ->
>>>> 2010-12-09 05:01:58,640 INFO
>> org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region
>> article,a83858a08f2270d319f75a7b43c756c2453988e7,1291811929868.58f6d9fc80
>>>> c78f3ca490b0280b4f1226. in 0sec
>>>> 2010-12-09 05:04:30,898 DEBUG
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.46 MB,
>> free=176.33 MB, max=177.79 MB, blocks=0, accesses=200583, hits=0, hitRatio=
>>>> 0.00%%, evictions=0, evicted=0, evictedPerRun=NaN
>>>> 2010-12-09 05:04:59,612 INFO
>> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs
>> -- HDFS-200
>>>> 2010-12-09 05:04:59,618 INFO
>> org.apache.hadoop.hbase.regionserver.wal.HLog: Roll
>> /hbase/.logs/04.hadoop.****,60020,1291858770125/10.100.154.103%3A60020.1291870585253,
>> entries=5318
>>>> 6, filesize=63776311. New hlog
>> /hbase/.logs/04.hadoop.****,60020,1291858770125/10.100.154.103
>> %3A60020.1291871099582
>>>> 2010-12-09 05:08:02,033 DEBUG
>> org.apache.hadoop.hbase.regionserver.wal.HLog: Found 1 hlogs to remove out
>> of total 6; oldest outstanding sequenceid is 32891801 from region article
>>>>
>> ,4b2039b791e894dd479b90661ca97087f61645d3,1291813201039.7dff47c63bb5648bb1ee6670c60553a5.
>>>> 2010-12-09 05:08:02,034 WARN
>> org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler 13 on
>> 60020 took 182416ms appending an edit to hlog; editcount=0
>>>> 2010-12-09 05:08:02,066 INFO
>> org.apache.hadoop.hbase.regionserver.wal.HLog: moving old hlog file
>> /hbase/.logs/04.hadoop.****,60020,1291858770125/10.100.154.103
>> %3A60020.12918679476
>>>> 65 whose highest sequenceid is 32880227 to /hbase/.oldlogs/
>> 10.100.154.103%3A60020.1291867947665
>>>> 2010-12-09 05:08:02,066 INFO org.apache.zookeeper.ClientCnxn: Client
>> session timed out, have not heard from server in 182936ms for sessionid
>> 0x12ca58c08ff0048, closing socket conn
>>>> ection and attempting reconnect
>>>> 2010-12-09 05:08:02,089 FATAL
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Aborting region server
>> serverName=04.hadoop.****,60020,1291858770125, load=(requests=24, regions=
>>>> 64, usedHeap=548, maxHeap=888): Unhandled exception
>>>> org.apache.hadoop.hbase.YouAreDeadException:
>> org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected;
>> currently processing 04.hadoop.****,60020,1291858770125 as dead s
>>>> erver
>>>> at
>> org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:217)
>>>> at
>> org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:271)
>>>> at
>> org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:744)
>>>> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:561)
>>>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>>>>
>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> Method)
>>>> at
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>>>> at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>>>> at
>> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
>>>> at
>> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
>>>> at
>> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
>>>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.doRun(HRegionServer.java:635)
>>>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.access$000(HRegionServer.java:126)
>>>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer$1.run(HRegionServer.java:518)
>>>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer$1.run(HRegionServer.java:516)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at javax.security.auth.Subject.doAs(Subject.java:337)
>>>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1046)
>>>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:516)
>>>> at java.lang.Thread.run(Thread.java:662)
>>>> 2010-12-09 05:08:02,090 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
>> request=0.0, regions=64, stores=64, storefiles=136, storefileIndexSize=27,
>> memsto
>>>> reSize=353, compactionQueueSize=0, usedHeap=549, maxHeap=888,
>> blockCacheSize=1530552, blockCacheFree=184893160, blockCacheCount=0,
>> blockCacheHitRatio=0
>>>> 2010-12-09 05:08:02,090 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
>> server on 60020
>>>> 2010-12-09 05:08:02,090 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>> Server handler 0 on 60020: exiting2010-12-09 05:08:02,090 INFO
>> org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server listener on 60020
>>>>
>>>>
>>>> MASTER HBASE ->
>>>>
>>>> 2010-12-09 05:05:06,072 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> All 1 .META. region(s) scanned
>>>> 2010-12-09 05:05:40,004 INFO
>> org.apache.hadoop.hbase.master.ServerManager:
>> 04.hadoop.jive,60020,1291858770125 znode expired
>>>> 2010-12-09 05:05:40,005 DEBUG
>> org.apache.hadoop.hbase.master.ServerManager:
>> Added=04.hadoop.jive,60020,1291858770125 to dead servers, added shutdown
>> processing operation
>>>> 2010-12-09 05:05:40,005 DEBUG
>> org.apache.hadoop.hbase.master.RegionServerOperationQueue: Processing todo:
>> ProcessServerShutdown of 04.hadoop.jive,60020,12918587701252010-12-09
>> 05:05:40,005 INFO org.apache.hadoop.hbase.master.RegionServerOperation:
>> Process shutdown of server 04.hadoop.jive,60020,1291858770125: logSplit:
>> false, rootRescanned: f
>>>> alse, numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
>>>> 2010-12-09 05:05:40,008 INFO
>> org.apache.hadoop.hbase.regionserver.wal.HLog: Splitting 7 hlog(s) in
>> hdfs://00.hadoop.jive:54310/hbase/.logs/04.hadoop.jive,60020,12918587701252010-12-09
>> 05:05:40,008 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: Splitting
>> hlog 1 of 7:
>> hdfs://00.hadoop.jive:54310/hbase/.logs/04.hadoop.jive,60020,1291858770125/10.1
>>>> 00.154.103%3A60020.1291867947665, length=637577092010-12-09 05:05:40,008
>> INFO org.apache.hadoop.hbase.util.FSUtils: Recovering
>> filehdfs://00.hadoop.jive:54310/hbase/.logs/04.hadoop.jive,60020,1291858770125/
>> 10.100.154.103%3A60020
>>>> .1291867947665
>>>>
>>>>
>>>> NAMENODE ->
>>>>
>>>> 2010-12-09 05:08:02,471 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
>> NameSystem.addStoredBlock: blockMap updated: 10.100.159.13:50010 is added
>> to blk_1531008743226086399_251615 size 63757709
>>>> 2010-12-09 05:08:02,473 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 9 on 54310, call
>> complete(/hbase/.logs/04.hadoop.jive,60020,1291858770125/10.100.154.103%3A60020.1291867947665,
>> DFSClient_hb_m_10.194.194.79:60000_1291788452343) from 10.194.194.79:44117:
>> error: org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No
>> lease on /hbase/.logs/04.hadoop.jive,60020,1291858770125/10.100.154.103%3A60020.1291867947665
>> File does not exist. [Lease. Holder:
>> DFSClient_hb_m_10.194.194.79:60000_1291788452343, pendingcreates: 1]
>>>> org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease
>> on /hbase/.logs/04.hadoop.jive,60020,1291858770125/10.100.154.103%3A60020.1291867947665
>> File does not exist. [Lease. Holder:
>> DFSClient_hb_m_10.194.194.79:60000_1291788452343, pendingcreates: 1]
>>>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1488)
>>>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1479)
>>>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:1534)
>>>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:1522)
>>>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.complete(NameNode.java:610)
>>>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>>>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:528)
>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1319)
>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1315)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1313)
>>>> 2010-12-09 05:08:04,206 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
>> NameSystem.addStoredBlock: blockMap updated: 10.99.97.106:50010 is added
>> to blk_-734280257049179934_251614 size 7330
>>>>
>>>> DATANODE ->
>>>>
>>>> 2010-12-09 05:08:02,212 WARN
>> org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in
>> BlockReceiver.lastNodeRun: java.io.IOException: Broken pipe
>>>> at sun.nio.ch.FileDispatcher.write0(Native Method)
>>>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100) at
>> sun.nio.ch.IOUtil.write(IOUtil.java:71)
>>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
>>>> at
>> org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
>> at
>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
>>>> at
>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
>> at java.io.DataOutputStream.writeLong(DataOutputStream.java:207)
>>>> at
>> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.write(DataTransferProtocol.java:133)
>> at
>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.lastDataNodeRun(BlockReceiver.java:840)
>>>> at
>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:870)
>>>> at java.lang.Thread.run(Thread.java:662)
>>>> 2010-12-09 05:08:02,213 WARN
>> org.apache.hadoop.hdfs.server.datanode.DataNode: checkDiskError: exception:
>>>> 2010-12-09 05:08:02,213 WARN
>> org.apache.hadoop.hdfs.server.datanode.DataNode: checkDiskError: exception:
>> java.io.IOException: Broken pipe
>>>> at sun.nio.ch.FileDispatcher.write0(Native Method)
>>>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
>>>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100)
>>>> at sun.nio.ch.IOUtil.write(IOUtil.java:71)
>>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
>>>> at
>> org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
>>>> at
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
>>>> at
>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
>>>> at
>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
>>>> at java.io.DataOutputStream.writeLong(DataOutputStream.java:207)
>>>> at
>> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.write(DataTransferProtocol.java:133)
>>>> at
>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.lastDataNodeRun(BlockReceiver.java:840)
>>>> at
>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:870)
>>>> at java.lang.Thread.run(Thread.java:662)
>>>> 2010-12-09 05:08:02,221 INFO
>> org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder
>> blk_-8817504198034990390_251613 0 Exception java.io.IOException: Broken pipe
>>>> at sun.nio.ch.FileDispatcher.write0(Native Method)
>>>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
>>>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100)
>>>> at sun.nio.ch.IOUtil.write(IOUtil.java:71)
>>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
>>>> at
>> org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
>>>> at
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
>>>> at
>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
>>>> at
>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
>>>> at java.io.DataOutputStream.writeLong(DataOutputStream.java:207)
>>>> at
>> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.write(DataTransferProtocol.java:133)
>>>> at
>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.lastDataNodeRun(BlockReceiver.java:840)
>>>> at
>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:870)
>>>> at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> 2010-12-09 05:08:02,221 INFO
>> org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block
>> blk_-8817504198034990390_251613 terminating
>>>> 2010-12-09 05:08:02,224 INFO
>> org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock
>> blk_-8817504198034990390_251613 received exception java.io.IOException:
>> Connection reset by peer
>>>> 2010-12-09 05:08:02,224 INFO
>> org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block
>> blk_4505306716668305567_250752 file
>> /dist1/data/hadoop-data/current/subdir19/blk_4505306716668305567 for
>> deletion
>>>> 2010-12-09 05:08:02,226 ERROR
>> org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
>> 10.100.154.103:50010,
>> storageID=DS-866555524-10.100.154.103-50010-1291262739506, infoPort=50075,
>> ipcPort=50020):DataXceiver
>>>> java.io.IOException: Connection reset by peer
>>>> at sun.nio.ch.FileDispatcher.read0(Native Method)
>>>> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
>>>> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:237)
>>>> at sun.nio.ch.IOUtil.read(IOUtil.java:210)
>>>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
>>>> at
>> org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:55)
>>>> at
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
>>>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
>>>> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
>>>> at java.io.DataInputStream.read(DataInputStream.java:132)
>>>> at
>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:267)
>>>> at
>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:357)
>>>> at
>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:378)
>>>> at
>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:534)
>>>> at
>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:417)
>>>> at
>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:122)
>>>> 2010-12-09 05:08:02,226 INFO
>> org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block
>> blk_1422641301942366074_250732 at file
>> /dist1/data/hadoop-data/current/subdir19/blk_1422641301942366074
>>>> 2010-12-09 05:08:02,227 INFO
>> org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block
>> blk_4505306716668305567_250752 at file
>> /dist1/data/hadoop-data/current/subdir19/blk_4505306716668305567
>>>>
>>>>
>>>>
>>>>
>> ************************************************************************************
>>>> 05.hadoop
>>>>
>> ************************************************************************************
>>>> REGION SERVER ->
>>>> 2010-12-09 04:29:25,047 DEBUG
>> org.apache.hadoop.hbase.regionserver.Store: Compaction size of data: 250.5m;
>> Skipped 1 file(s), size: 206896367
>>>> 2010-12-09 04:29:25,047 INFO org.apache.hadoop.hbase.regionserver.Store:
>> Started compaction of 2 file(s) in data of
>> article,060c4b75067378c045eead9edec2d4fb7d8f3e5e,1291812563166.95e92e97823e11e57850720a63433a62.
>> into
>> hdfs://00.hadoop.****:54310/hbase/article/95e92e97823e11e57850720a63433a62/.tmp,
>> sequenceid=33050221
>>>> 2010-12-09 04:29:26,328 INFO org.apache.hadoop.hbase.regionserver.Store:
>> Completed compaction of 2 file(s) in data of
>> article,060c4b75067378c045eead9edec2d4fb7d8f3e5e,1291812563166.95e92e97823e11e57850720a63433a62.;
>> new storefile is
>> hdfs://00.hadoop.****:54310/hbase/article/95e92e97823e11e57850720a63433a62/data/2787768957794924068;
>> store size is 250.5m
>>>> 2010-12-09 04:29:26,328 INFO
>> org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region
>> article,060c4b75067378c045eead9edec2d4fb7d8f3e5e,1291812563166.95e92e97823e11e57850720a63433a62.
>> in 1sec
>>>> 2010-12-09 04:29:26,328 INFO
>> org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on region
>> article,ed0cdc7ae6db73432e2ca47c973d545bee11ddd8,1291813400028.85a8d31d099f409dd36f3bcc4be1eba6.
>>>> 2010-12-09 04:29:26,330 INFO
>> org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region
>> article,ed0cdc7ae6db73432e2ca47c973d545bee11ddd8,1291813400028.85a8d31d099f409dd36f3bcc4be1eba6.
>> in 0sec
>>>> 2010-12-09 04:32:33,081 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGIONSERVER_STOP
>>>> 2010-12-09 04:32:33,081 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
>> server on 60020
>>>> 2010-12-09 04:32:33,089 WARN org.apache.hadoop.hdfs.DFSClient:
>> DFSOutputStream ResponseProcessor exception for block
>> blk_2012253533858803341_250248java.io.EOFException
>>>> at java.io.DataInputStream.readFully(DataInputStream.java:180)
>>>> at java.io.DataInputStream.readLong(DataInputStream.java:399)
>>>> at
>> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:120)
>>>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2621)
>>>>
>>>> 2010-12-09 04:32:33,089 INFO org.apache.zookeeper.ClientCnxn: Unable to
>> read additional data from server sessionid 0x12ca58c08ff0049, likely server
>> has closed socket, closing sock
>>>> et connection and attempting reconnect
>>>> 2010-12-09 04:32:33,089 WARN
>> org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler 5 on 60020
>> took 75446ms appending an edit to hlog; editcount=50505
>>>> 2010-12-09 04:32:33,090 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>> Server handler 18 on 60020: exiting
>>>> 2010-12-09 04:32:33,139 INFO org.apache.hadoop.hdfs.DFSClient: Error
>> Recovery for block blk_2012253533858803341_250248 waiting for responder to
>> exit.
>>>> 2010-12-09 04:32:33,139 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>> Server handler 8 on 60020: exiting
>>>> 2010-12-09 04:32:33,139 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>> Server handler 1 on 60020: exiting
>>>> 2010-12-09 04:32:33,139 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>> Server handler 10 on 60020: exiting
>>>> 2010-12-09 04:32:33,140 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>> Server handler 0 on 60020: exiting2010-12-09 04:32:33,140 INFO
>> org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 5 on 60020:
>> exiting
>>>> 2010-12-09 04:32:33,139 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>> Server handler 8 on 60020: exiting2010-12-09 04:32:33,140 INFO
>> org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 4 on 60020:
>> exiting
>>>> 2010-12-09 04:32:33,140 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>> Server handler 3 on 60020: exiting2010-12-09 04:32:33,142 INFO
>> org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 7 on 60020:
>> exiting
>>>> 2010-12-09 04:32:33,142 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>> Server handler 9 on 60020: exiting2010-12-09 04:32:33,143 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
>>>> 2010-12-09 04:32:33,143 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>> Server handler 2 on 60020: exiting
>>>> 2010-12-09 04:32:33,143 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
>> IPC Server Responder
>>>> 2010-12-09 04:32:33,143 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>> Server handler 6 on 60020: exiting
>>>> 2010-12-09 04:32:33,144 INFO org.mortbay.log: Stopped
>> SelectChannelConnector@0.0.0.0:60030
>>>>
>>>>
>>>>
>>>>
>>>> NAMENODE ->
>>>> 2010-12-09 04:32:08,408 WARN org.apache.hadoop.hdfs.StateChange: DIR*
>> NameSystem.startFile: failed to create file
>> /hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555
>> for DFSClient_hb_m_10.194.194.79:60000_1291788452343 on client
>> 10.194.194.79, because this file is already being created by NN_Recovery on
>> 10.100.159.13
>>>> 2010-12-09 04:32:08,408 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 6 on 54310, call
>> append(/hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555,
>> DFSClient_hb_m_10.194.194.79:60000_1291788452343) from 10.194.194.79:59118:
>> error: org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed
>> to create file /hba
>>>> se/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555
>> for DFSClient_hb_m_10.194.194.79:60000_1291788452343 on client
>> 10.194.194.79, because this file is already being created by NN_Recovery on
>> 10.100.159.13
>>>> org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to
>> create file /hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555
>> for DFSClient_hb_m_10.194.194.79:60000_1291788452343 on client
>> 10.194.194.79, because this file is already being created by NN_Recovery on
>> 10.100.159.13
>>>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1194)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:1282)
>>>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.append(NameNode.java:541)
>> at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
>>>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:528) at
>> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1319)
>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1315)
>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at javax.security.auth.Subject.doAs(Subject.java:396) at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
>>>> at
>> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1313)2010-12-09
>> 04:32:09,411 WARN org.apache.hadoop.hdfs.StateChange: DIR*
>> NameSystem.startFile: failed to create file
>> /hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A6
>>>> 0020.1291868481555 for DFSClient_hb_m_10.194.194.79:60000_1291788452343
>> on client 10.194.194.79, because this file is already being created by
>> NN_Recovery on 10.100.159.132010-12-09 04:32:09,411 INFO
>> org.apache.hadoop.ipc.Server: IPC Server handler 9 on 54310, call
>> append(/hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13
>> %3A60020.1291868
>>>> 481555, DFSClient_hb_m_10.194.194.79:60000_1291788452343) from
>> 10.194.194.79:59118: error:
>> org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to
>> create file /hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555
>> for DFSClient_hb_m_10.194.194.79:60000_1291788452343 on client
>> 10.194.194.79, because this file is
>>>> already being created by NN_Recovery on
>> 10.100.159.13org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException:
>> failed to create file /hbase/.logs/05.hadoop.jive,60020,1291862294517/
>> 10.100.159.13%3A60020.1291868481555 for DFSClie
>>>> nt_hb_m_10.194.194.79:60000_1291788452343 on client 10.194.194.79,
>> because this file is already being created by NN_Recovery on 10.100.159.13
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1194)
>>>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:1282)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.append(NameNode.java:541)
>>>> at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597) at
>> org.apache.hadoop.ipc.RPC$Server.call(RPC.java:528)
>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1319)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1315)
>>>>
>>>>
>>>>
>>>> DATANODE
>>>>
>>>> 2010-12-09 04:32:09,534 INFO
>> org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder
>> blk_2012253533858803341_250248 1 : Thread is interrupted.
>>>> 2010-12-09 04:32:09,534 ERROR
>> org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
>> 10.100.159.13:50010,
>> storageID=DS-145997007-10.100.159.13-50010-1291262743182, infoPort=50075,
>> ipcPort=50020):DataXceiver
>>>> java.io.IOException: Interrupted receiveBlock
>>>> at
>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:579)
>>>> at
>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:417)
>>>> at
>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:122)
>>>> 2010-12-09 04:32:09,534 INFO
>> org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 1 for block
>> blk_2012253533858803341_250248 terminating
>>>>
>>>>
>>>>
>> ************************************************************************************
>>>> 06.hadoop
>>>>
>> ************************************************************************************
>>>> REGION SERVER ->
>>>>
>>>> 2010-12-09 06:17:47,215 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>> NotServingRegionException;
>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>> 2010-12-09 06:17:47,430 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>> NotServingRegionException;
>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>> 2010-12-09 06:17:47,549 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>> NotServingRegionException;
>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>> 2010-12-09 06:17:47,557 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>> NotServingRegionException;
>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>> 2010-12-09 06:17:47,563 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>> NotServingRegionException;
>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>> 2010-12-09 06:17:48,619 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>> NotServingRegionException;
>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>> 2010-12-09 06:17:48,619 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>> NotServingRegionException;
>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>> 2010-12-09 06:17:48,619 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>> NotServingRegionException;
>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>> 2010-12-09 06:17:48,619 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>> NotServingRegionException;
>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>> 2010-12-09 06:17:48,619 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>> NotServingRegionException;
>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>> 2010-12-09 06:17:49,703 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>> NotServingRegionException;
>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>> 2010-12-09 06:17:49,703 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>> NotServingRegionException;
>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>> 2010-12-09 06:17:49,703 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>> NotServingRegionException;
>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>> 2010-12-09 06:17:49,705 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>> NotServingRegionException;
>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>> 2010-12-09 06:17:49,729 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>> NotServingRegionException;
>> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>>>
>>>> A bunch of these ^
>>>>
>>>> Then a little later::
>>>> ....
>>>>
>>>> 2010-12-09 06:23:23,561 DEBUG
>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction
>> requested for region
>> article,993143fca3613014990cc6975cacdd62f4147e03,12918133247
>>>> 53.19413f33fc660536d9d4f315a1283440. because:
>> regionserver60020.cacheFlusher
>>>> 2010-12-09 06:23:27,058 INFO
>> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs
>> -- HDFS-200
>>>> 2010-12-09 06:23:27,080 INFO
>> org.apache.hadoop.hbase.regionserver.wal.HLog: Roll
>> /hbase/.logs/06.hadoop.****,60020,1291864620851/10.99.97.106%3A60020.1291875047947,
>> entries=53315,
>>>> filesize=63781925. New hlog
>> /hbase/.logs/06.hadoop.****,60020,1291864620851/10.99.97.106
>> %3A60020.1291875807006
>>>> 2010-12-09 06:23:27,080 DEBUG
>> org.apache.hadoop.hbase.regionserver.wal.HLog: Found 1 hlogs to remove out
>> of total 8; oldest outstanding sequenceid is 33526881 from region article
>>>>
>> ,1221b0013c61fd7446cb822f698d0bcafa487342,1291872610788.bb2aa6ae8c694a096a00e7bb91fc46c3.
>>>> 2010-12-09 06:23:27,080 INFO
>> org.apache.hadoop.hbase.regionserver.wal.HLog: moving old hlog file
>> /hbase/.logs/06.hadoop.****,60020,1291864620851/10.99.97.106
>> %3A60020.1291872064424
>>>> whose highest sequenceid is 33506639 to /hbase/.oldlogs/10.99.97.106
>> %3A60020.1291872064424
>>>> 2010-12-09 06:25:39,162 WARN
>> org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler 12 on
>> 60020 took 119034ms appending an edit to hlog; editcount=3683
>>>> 2010-12-09 06:25:39,162 WARN org.apache.hadoop.hdfs.DFSClient:
>> DataStreamer Exception: java.io.IOException: Broken pipe
>>>> at sun.nio.ch.FileDispatcher.write0(Native Method)
>>>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
>>>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100)
>>>> at sun.nio.ch.IOUtil.write(IOUtil.java:71)
>>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
>>>> at
>> org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
>>>> at
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
>>>> at
>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
>>>> at
>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
>>>> at
>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
>>>> at java.io.DataOutputStream.write(DataOutputStream.java:90)
>>>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2509)
>>>> 2010-12-09 06:25:39,163 WARN org.apache.hadoop.hdfs.DFSClient:
>> DFSOutputStream ResponseProcessor exception for block
>> blk_1141147913164105086_258331java.io.EOFException
>>>> at java.io.DataInputStream.readFully(DataInputStream.java:180)
>>>> at java.io.DataInputStream.readLong(DataInputStream.java:399)
>>>> at
>> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:120)
>>>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2621)
>>>>
>>>> 2010-12-09 06:25:39,163 INFO org.apache.hadoop.hdfs.DFSClient: Error
>> Recovery for block blk_1141147913164105086_258331 waiting for responder to
>> exit.
>>>> 2010-12-09 06:25:39,164 WARN org.apache.hadoop.hdfs.DFSClient:
>> DFSOutputStream ResponseProcessor exception for block
>> blk_-7076574401277381592_258324java.io.EOFException
>>>> at java.io.DataInputStream.readFully(DataInputStream.java:180)
>>>> at java.io.DataInputStream.readLong(DataInputStream.java:399)
>>>> at
>> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:120)
>>>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2621)
>>>>
>>>> 2010-12-09 06:25:39,164 WARN org.apache.hadoop.hdfs.DFSClient: Error
>> Recovery for block blk_-7076574401277381592_258324 bad datanode[0]
>> 10.99.97.106:50010
>>>> 2010-12-09 06:25:39,164 INFO org.apache.zookeeper.ClientCnxn: Unable to
>> read additional data from server sessionid 0x12ca58c08ff004b, likely server
>> has closed socket, closing socket connection and attempting reconnect
>>>> 2010-12-09 06:25:39,164 WARN org.apache.hadoop.hdfs.DFSClient: Error
>> Recovery for block blk_-7076574401277381592_258324 in pipeline
>> 10.99.97.106:50010, 10.100.154.103:50010: bad datanode 10.99.97.106:50010
>>>> 2010-12-09 06:25:39,165 WARN org.apache.hadoop.hdfs.DFSClient: Error
>> Recovery for block blk_1141147913164105086_258331 bad datanode[0]
>> 10.99.97.106:50010
>>>> 2010-12-09 06:25:39,167 INFO org.apache.zookeeper.ClientCnxn: Unable to
>> read additional data from server sessionid 0x12ca58c08ff004c, likely server
>> has closed socket, closing socket connection and attempting reconnect
>>>> 2010-12-09 06:25:39,169 INFO org.apache.hadoop.ipc.Client: Retrying
>> connect to server: /10.100.154.103:50020. Already tried 0 time(s).
>>>> 2010-12-09 06:25:39,169 WARN org.apache.hadoop.hdfs.DFSClient: Error
>> Recovery for block blk_1141147913164105086_258331 in pipeline
>> 10.99.97.106:50010, 10.100.159.13:50010: bad datanode 10.99.97.106:50010
>>>> 2010-12-09 06:25:39,209 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGIONSERVER_STOP
>>>> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
>> server on 60020
>>>> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>> Server handler 3 on 60020: exiting
>>>> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>> Server handler 9 on 60020: exiting
>>>> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>> Server handler 24 on 60020: exiting
>>>> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>> Server handler 1 on 60020: exiting
>>>> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>> Server handler 15 on 60020: exiting
>>>> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>> Server handler 0 on 60020: exiting
>>>> 2010-12-09 06:25:39,214 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>> Server handler 3 on 60020: exiting
>>>> 2010-12-09 06:25:39,214 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
>> IPC Server listener on 60020
>>>> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>> Server handler 2 on 60020: exiting
>>>> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>> Server handler 7 on 60020: exiting
>>>> 2010-12-09 06:25:39,213 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>> Server handler 21 on 60020: exiting
>>>> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>> Server handler 4 on 60020: exiting
>>>> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
>> IPC Server Responder
>>>> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>> Server handler 5 on 60020: exiting
>>>> 2010-12-09 06:25:39,216 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>> Server handler 8 on 60020: exiting
>>>> 2010-12-09 06:25:39,216 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>> Server handler 9 on 60020: exiting
>>>> 2010-12-09 06:25:39,216 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
>> Server handler 6 on 60020: exiting
>>>> 2010-12-09 06:25:39,216 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
>>>> 2010-12-09 06:25:39,223 WARN org.apache.hadoop.hdfs.DFSClient: Failed
>> recovery attempt #0 from primary datanode 10.100.159.13:50010
>>>>
>>>>
>>>>
>>>>
>>
>>
Re: Crashing Region Servers
Posted by Ted Dunning <td...@maprtech.com>.
This could indicate swapping during GC.
On Thu, Dec 9, 2010 at 12:13 PM, Lance Riedel <la...@gmail.com> wrote:
> Seems reasonable, but having trouble making sense of the GC logs I had
> turned on. Basically since there was a full GC a minute before this happens
> on that server that lasts less than a second.
>
>
> Example:
> So, here is what the last of the GC logs say for that Regionserver
> (04.hadoop on 10.100.154.10) (note the last full GC only 1 minutes before @
> 5:03:29.850+0000):
>
> 2010-12-09T05:02:19.472+0000: 12170.437: [GC [PSYoungGen:
> 278229K->5024K(287360K)] 694269K->421693K(721856K), 0.0274840 secs] [Times:
> user=0.04 sys=0.00, real=0.03 secs]
> 2010-12-09T05:02:55.479+0000: 12206.444: [GC [PSYoungGen:
> 281450K->7968K(273344K)] 698120K->429658K(707840K), 0.0503000 secs] [Times:
> user=0.10 sys=0.00, real=0.05 secs]
> 2010-12-09T05:03:29.800+0000: 12240.766: [GC [PSYoungGen:
> 273295K->7712K(262464K)] 694986K->437306K(696960K), 0.0490770 secs] [Times:
> user=0.12 sys=0.01, real=0.05 secs]
> 2010-12-09T05:03:29.850+0000: 12240.815: [Full GC [PSYoungGen:
> 7712K->0K(262464K)] [PSOldGen: 429594K->353152K(429824K)]
> 437306K->353152K(692288K) [PSPermGen: 18016K->18016K(21248K)], 0.8960760
> secs] [Times: user=0.89 sys=0.00, real=0.90 secs]
> 2010-12-09T05:04:03.345+0000: 12274.310: [GC [PSYoungGen:
> 254720K->6944K(251584K)] 607872K->360528K(681408K), 0.0331060 secs] [Times:
> user=0.03 sys=0.00, real=0.04 secs]
> 2010-12-09T05:04:33.875+0000: 12304.840: [GC [PSYoungGen:
> 251552K->7200K(242240K)] 605136K->367800K(672064K), 0.0336420 secs] [Times:
> user=0.09 sys=0.01, real=0.03 secs]
> Heap
> PSYoungGen total 242240K, used 223390K [0x00007fcc640b0000,
> 0x00007fcc77a90000, 0x00007fcc78e00000)
> eden space 235008K, 91% used
> [0x00007fcc640b0000,0x00007fcc713cfa30,0x00007fcc72630000)
> from space 7232K, 99% used
> [0x00007fcc769b0000,0x00007fcc770b8090,0x00007fcc770c0000)
> to space 10048K, 0% used
> [0x00007fcc770c0000,0x00007fcc770c0000,0x00007fcc77a90000)
> PSOldGen total 429824K, used 360600K [0x00007fcc3a600000,
> 0x00007fcc549c0000, 0x00007fcc640b0000)
> object space 429824K, 83% used
> [0x00007fcc3a600000,0x00007fcc50626060,0x00007fcc549c0000)
> PSPermGen total 21248K, used 18131K [0x00007fcc35200000,
> 0x00007fcc366c0000, 0x00007fcc3a600000)
> object space 21248K, 85% used
> [0x00007fcc35200000,0x00007fcc363b4d60,0x00007fcc366c0000)
>
> Now Dead / End of log
>
>
> That said, I will continue down the path you pointed me on .
>
> Thanks,
> Lance
>
> On Dec 9, 2010, at 11:40 AM, Jean-Daniel Cryans wrote:
>
> > Lance,
> >
> > Both those lines indicate the problem:
> >
> > IPC Server handler 13 on 60020 took 182416ms
> > Client session timed out, have not heard from server in 182936ms
> >
> > It's very clear that your region servers are suffering from
> > pause-of-the-world garbage collection issues. Basically this one GC'ed
> > for 3 minutes, which is over the 1 minute timeout to consider a region
> > server dead. The rest is just the side effect of that.
> >
> > This subject comes often on the mailing list, for example:
> > http://search-hadoop.com/m/t97q027tr7f2
> >
> > Make sure you don't swap, give plenty of RAM to HBase, use LZO
> > compression, don't underprovision your machines, etc
> >
> > J-D
> >
> > On Thu, Dec 9, 2010 at 11:27 AM, Lance Riedel <la...@gmail.com>
> wrote:
> >>
> >> We have a 6 node cluster, 5 with region serves. 2 of the region servers
> have been stable for days, but 3 of them keep crashing. Here are the logs
> around around when the crash occurs. (btw, we are shoving approximately the
> twitter firehose into hbase via flume) I'm an hbase newbie, but I have been
> reading. Not sure what else is needed to help debug. When the problem
> occurs, it is concurrent with issues that appear all the way down in the
> data node. Also, interesting to note, all 3 servers seem to fail
> differently somewhat:
> >>
> >> Servers (6):
> >> 00.hadoop is the master
> >> 01.hadoop, 03.hadoop (there is no 02.hadoop - flaky machine that got
> taken out) - are stable Region Servers
> >> 04-06.hadoop crash at different times, very different logs, Region
> Servers
> >>
> >>
> >> Master Attributes:
> >> HBase Version 0.89.20100924+28, r
> >> Hadoop Version 0.20.2+737, r98c55c28258aa6f42250569bd7fa431ac657bdbd
> >>
> >> HBase Root Directory hdfs://00.hadoop.****:54310/hbase
> >>
> >> Load average
> >> 103.0
> >> Average number of regions per regionserver. Naive computation.
> >> Regions On FS
> >> 577
> >>
> >>
> >> (This is after recently restarting 04-06)
> >>
> >>
> >> Address Start Code Load
> >> 01.hadoop:60030 1291268910638 requests=223, regions=118,
> usedHeap=519, maxHeap=987
> >> 03.hadoop:60030 1291269219610 requests=23, regions=111, usedHeap=862,
> maxHeap=987
> >> 04.hadoop:60030 1291910665912 requests=169, regions=82, usedHeap=194,
> maxHeap=888
> >> 05.hadoop:60030 1291909584060 requests=232, regions=110,
> usedHeap=477, maxHeap=888
> >> 06.hadoop60030 1291909723787 requests=99, regions=94, usedHeap=394,
> maxHeap=888
> >> Total:
> >> servers: 5
> >> requests=746, regions=515
> >>
> >>
> >>
> >> Following are various logs around the time of the failures for 04-06
> >>
> >>
> ************************************************************************************
> >> 04.hadoop
> >>
> ************************************************************************************
> >> REGION SERVER ->
> >> 2010-12-09 05:01:58,640 INFO
> org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region
> article,a83858a08f2270d319f75a7b43c756c2453988e7,1291811929868.58f6d9fc80
> >> c78f3ca490b0280b4f1226. in 0sec
> >> 2010-12-09 05:04:30,898 DEBUG
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.46 MB,
> free=176.33 MB, max=177.79 MB, blocks=0, accesses=200583, hits=0, hitRatio=
> >> 0.00%%, evictions=0, evicted=0, evictedPerRun=NaN
> >> 2010-12-09 05:04:59,612 INFO
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs
> -- HDFS-200
> >> 2010-12-09 05:04:59,618 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLog: Roll
> /hbase/.logs/04.hadoop.****,60020,1291858770125/10.100.154.103%3A60020.1291870585253,
> entries=5318
> >> 6, filesize=63776311. New hlog
> /hbase/.logs/04.hadoop.****,60020,1291858770125/10.100.154.103
> %3A60020.1291871099582
> >> 2010-12-09 05:08:02,033 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLog: Found 1 hlogs to remove out
> of total 6; oldest outstanding sequenceid is 32891801 from region article
> >>
> ,4b2039b791e894dd479b90661ca97087f61645d3,1291813201039.7dff47c63bb5648bb1ee6670c60553a5.
> >> 2010-12-09 05:08:02,034 WARN
> org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler 13 on
> 60020 took 182416ms appending an edit to hlog; editcount=0
> >> 2010-12-09 05:08:02,066 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLog: moving old hlog file
> /hbase/.logs/04.hadoop.****,60020,1291858770125/10.100.154.103
> %3A60020.12918679476
> >> 65 whose highest sequenceid is 32880227 to /hbase/.oldlogs/
> 10.100.154.103%3A60020.1291867947665
> >> 2010-12-09 05:08:02,066 INFO org.apache.zookeeper.ClientCnxn: Client
> session timed out, have not heard from server in 182936ms for sessionid
> 0x12ca58c08ff0048, closing socket conn
> >> ection and attempting reconnect
> >> 2010-12-09 05:08:02,089 FATAL
> org.apache.hadoop.hbase.regionserver.HRegionServer: Aborting region server
> serverName=04.hadoop.****,60020,1291858770125, load=(requests=24, regions=
> >> 64, usedHeap=548, maxHeap=888): Unhandled exception
> >> org.apache.hadoop.hbase.YouAreDeadException:
> org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected;
> currently processing 04.hadoop.****,60020,1291858770125 as dead s
> >> erver
> >> at
> org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:217)
> >> at
> org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:271)
> >> at
> org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:744)
> >> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> >> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> at java.lang.reflect.Method.invoke(Method.java:597)
> >> at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:561)
> >> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
> >>
> >> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> >> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> >> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> >> at
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
> >> at
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
> >> at
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
> >> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.doRun(HRegionServer.java:635)
> >> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.access$000(HRegionServer.java:126)
> >> at
> org.apache.hadoop.hbase.regionserver.HRegionServer$1.run(HRegionServer.java:518)
> >> at
> org.apache.hadoop.hbase.regionserver.HRegionServer$1.run(HRegionServer.java:516)
> >> at java.security.AccessController.doPrivileged(Native Method)
> >> at javax.security.auth.Subject.doAs(Subject.java:337)
> >> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1046)
> >> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:516)
> >> at java.lang.Thread.run(Thread.java:662)
> >> 2010-12-09 05:08:02,090 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
> request=0.0, regions=64, stores=64, storefiles=136, storefileIndexSize=27,
> memsto
> >> reSize=353, compactionQueueSize=0, usedHeap=549, maxHeap=888,
> blockCacheSize=1530552, blockCacheFree=184893160, blockCacheCount=0,
> blockCacheHitRatio=0
> >> 2010-12-09 05:08:02,090 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> server on 60020
> >> 2010-12-09 05:08:02,090 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
> Server handler 0 on 60020: exiting2010-12-09 05:08:02,090 INFO
> org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server listener on 60020
> >>
> >>
> >> MASTER HBASE ->
> >>
> >> 2010-12-09 05:05:06,072 INFO org.apache.hadoop.hbase.master.BaseScanner:
> All 1 .META. region(s) scanned
> >> 2010-12-09 05:05:40,004 INFO
> org.apache.hadoop.hbase.master.ServerManager:
> 04.hadoop.jive,60020,1291858770125 znode expired
> >> 2010-12-09 05:05:40,005 DEBUG
> org.apache.hadoop.hbase.master.ServerManager:
> Added=04.hadoop.jive,60020,1291858770125 to dead servers, added shutdown
> processing operation
> >> 2010-12-09 05:05:40,005 DEBUG
> org.apache.hadoop.hbase.master.RegionServerOperationQueue: Processing todo:
> ProcessServerShutdown of 04.hadoop.jive,60020,12918587701252010-12-09
> 05:05:40,005 INFO org.apache.hadoop.hbase.master.RegionServerOperation:
> Process shutdown of server 04.hadoop.jive,60020,1291858770125: logSplit:
> false, rootRescanned: f
> >> alse, numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> >> 2010-12-09 05:05:40,008 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLog: Splitting 7 hlog(s) in
> hdfs://00.hadoop.jive:54310/hbase/.logs/04.hadoop.jive,60020,12918587701252010-12-09
> 05:05:40,008 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: Splitting
> hlog 1 of 7:
> hdfs://00.hadoop.jive:54310/hbase/.logs/04.hadoop.jive,60020,1291858770125/10.1
> >> 00.154.103%3A60020.1291867947665, length=637577092010-12-09 05:05:40,008
> INFO org.apache.hadoop.hbase.util.FSUtils: Recovering
> filehdfs://00.hadoop.jive:54310/hbase/.logs/04.hadoop.jive,60020,1291858770125/
> 10.100.154.103%3A60020
> >> .1291867947665
> >>
> >>
> >> NAMENODE ->
> >>
> >> 2010-12-09 05:08:02,471 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.addStoredBlock: blockMap updated: 10.100.159.13:50010 is added
> to blk_1531008743226086399_251615 size 63757709
> >> 2010-12-09 05:08:02,473 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 9 on 54310, call
> complete(/hbase/.logs/04.hadoop.jive,60020,1291858770125/10.100.154.103%3A60020.1291867947665,
> DFSClient_hb_m_10.194.194.79:60000_1291788452343) from 10.194.194.79:44117:
> error: org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No
> lease on /hbase/.logs/04.hadoop.jive,60020,1291858770125/10.100.154.103%3A60020.1291867947665
> File does not exist. [Lease. Holder:
> DFSClient_hb_m_10.194.194.79:60000_1291788452343, pendingcreates: 1]
> >> org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease
> on /hbase/.logs/04.hadoop.jive,60020,1291858770125/10.100.154.103%3A60020.1291867947665
> File does not exist. [Lease. Holder:
> DFSClient_hb_m_10.194.194.79:60000_1291788452343, pendingcreates: 1]
> >> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1488)
> >> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1479)
> >> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:1534)
> >> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:1522)
> >> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.complete(NameNode.java:610)
> >> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> >> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> at java.lang.reflect.Method.invoke(Method.java:597)
> >> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:528)
> >> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1319)
> >> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1315)
> >> at java.security.AccessController.doPrivileged(Native Method)
> >> at javax.security.auth.Subject.doAs(Subject.java:396)
> >> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
> >> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1313)
> >> 2010-12-09 05:08:04,206 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.addStoredBlock: blockMap updated: 10.99.97.106:50010 is added
> to blk_-734280257049179934_251614 size 7330
> >>
> >> DATANODE ->
> >>
> >> 2010-12-09 05:08:02,212 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in
> BlockReceiver.lastNodeRun: java.io.IOException: Broken pipe
> >> at sun.nio.ch.FileDispatcher.write0(Native Method)
> >> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100) at
> sun.nio.ch.IOUtil.write(IOUtil.java:71)
> >> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
> >> at
> org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
> at
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
> >> at
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
> at java.io.DataOutputStream.writeLong(DataOutputStream.java:207)
> >> at
> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.write(DataTransferProtocol.java:133)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.lastDataNodeRun(BlockReceiver.java:840)
> >> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:870)
> >> at java.lang.Thread.run(Thread.java:662)
> >> 2010-12-09 05:08:02,213 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: checkDiskError: exception:
> >> 2010-12-09 05:08:02,213 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: checkDiskError: exception:
> java.io.IOException: Broken pipe
> >> at sun.nio.ch.FileDispatcher.write0(Native Method)
> >> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
> >> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100)
> >> at sun.nio.ch.IOUtil.write(IOUtil.java:71)
> >> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
> >> at
> org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
> >> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
> >> at
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
> >> at
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
> >> at java.io.DataOutputStream.writeLong(DataOutputStream.java:207)
> >> at
> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.write(DataTransferProtocol.java:133)
> >> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.lastDataNodeRun(BlockReceiver.java:840)
> >> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:870)
> >> at java.lang.Thread.run(Thread.java:662)
> >> 2010-12-09 05:08:02,221 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder
> blk_-8817504198034990390_251613 0 Exception java.io.IOException: Broken pipe
> >> at sun.nio.ch.FileDispatcher.write0(Native Method)
> >> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
> >> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100)
> >> at sun.nio.ch.IOUtil.write(IOUtil.java:71)
> >> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
> >> at
> org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
> >> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
> >> at
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
> >> at
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
> >> at java.io.DataOutputStream.writeLong(DataOutputStream.java:207)
> >> at
> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.write(DataTransferProtocol.java:133)
> >> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.lastDataNodeRun(BlockReceiver.java:840)
> >> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:870)
> >> at java.lang.Thread.run(Thread.java:662)
> >>
> >> 2010-12-09 05:08:02,221 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block
> blk_-8817504198034990390_251613 terminating
> >> 2010-12-09 05:08:02,224 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock
> blk_-8817504198034990390_251613 received exception java.io.IOException:
> Connection reset by peer
> >> 2010-12-09 05:08:02,224 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block
> blk_4505306716668305567_250752 file
> /dist1/data/hadoop-data/current/subdir19/blk_4505306716668305567 for
> deletion
> >> 2010-12-09 05:08:02,226 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
> 10.100.154.103:50010,
> storageID=DS-866555524-10.100.154.103-50010-1291262739506, infoPort=50075,
> ipcPort=50020):DataXceiver
> >> java.io.IOException: Connection reset by peer
> >> at sun.nio.ch.FileDispatcher.read0(Native Method)
> >> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
> >> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:237)
> >> at sun.nio.ch.IOUtil.read(IOUtil.java:210)
> >> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
> >> at
> org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:55)
> >> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
> >> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
> >> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
> >> at
> java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
> >> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> >> at java.io.DataInputStream.read(DataInputStream.java:132)
> >> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:267)
> >> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:357)
> >> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:378)
> >> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:534)
> >> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:417)
> >> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:122)
> >> 2010-12-09 05:08:02,226 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block
> blk_1422641301942366074_250732 at file
> /dist1/data/hadoop-data/current/subdir19/blk_1422641301942366074
> >> 2010-12-09 05:08:02,227 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block
> blk_4505306716668305567_250752 at file
> /dist1/data/hadoop-data/current/subdir19/blk_4505306716668305567
> >>
> >>
> >>
> >>
> ************************************************************************************
> >> 05.hadoop
> >>
> ************************************************************************************
> >> REGION SERVER ->
> >> 2010-12-09 04:29:25,047 DEBUG
> org.apache.hadoop.hbase.regionserver.Store: Compaction size of data: 250.5m;
> Skipped 1 file(s), size: 206896367
> >> 2010-12-09 04:29:25,047 INFO org.apache.hadoop.hbase.regionserver.Store:
> Started compaction of 2 file(s) in data of
> article,060c4b75067378c045eead9edec2d4fb7d8f3e5e,1291812563166.95e92e97823e11e57850720a63433a62.
> into
> hdfs://00.hadoop.****:54310/hbase/article/95e92e97823e11e57850720a63433a62/.tmp,
> sequenceid=33050221
> >> 2010-12-09 04:29:26,328 INFO org.apache.hadoop.hbase.regionserver.Store:
> Completed compaction of 2 file(s) in data of
> article,060c4b75067378c045eead9edec2d4fb7d8f3e5e,1291812563166.95e92e97823e11e57850720a63433a62.;
> new storefile is
> hdfs://00.hadoop.****:54310/hbase/article/95e92e97823e11e57850720a63433a62/data/2787768957794924068;
> store size is 250.5m
> >> 2010-12-09 04:29:26,328 INFO
> org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region
> article,060c4b75067378c045eead9edec2d4fb7d8f3e5e,1291812563166.95e92e97823e11e57850720a63433a62.
> in 1sec
> >> 2010-12-09 04:29:26,328 INFO
> org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on region
> article,ed0cdc7ae6db73432e2ca47c973d545bee11ddd8,1291813400028.85a8d31d099f409dd36f3bcc4be1eba6.
> >> 2010-12-09 04:29:26,330 INFO
> org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region
> article,ed0cdc7ae6db73432e2ca47c973d545bee11ddd8,1291813400028.85a8d31d099f409dd36f3bcc4be1eba6.
> in 0sec
> >> 2010-12-09 04:32:33,081 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGIONSERVER_STOP
> >> 2010-12-09 04:32:33,081 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> server on 60020
> >> 2010-12-09 04:32:33,089 WARN org.apache.hadoop.hdfs.DFSClient:
> DFSOutputStream ResponseProcessor exception for block
> blk_2012253533858803341_250248java.io.EOFException
> >> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> >> at java.io.DataInputStream.readLong(DataInputStream.java:399)
> >> at
> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:120)
> >> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2621)
> >>
> >> 2010-12-09 04:32:33,089 INFO org.apache.zookeeper.ClientCnxn: Unable to
> read additional data from server sessionid 0x12ca58c08ff0049, likely server
> has closed socket, closing sock
> >> et connection and attempting reconnect
> >> 2010-12-09 04:32:33,089 WARN
> org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler 5 on 60020
> took 75446ms appending an edit to hlog; editcount=50505
> >> 2010-12-09 04:32:33,090 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server handler 18 on 60020: exiting
> >> 2010-12-09 04:32:33,139 INFO org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_2012253533858803341_250248 waiting for responder to
> exit.
> >> 2010-12-09 04:32:33,139 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server handler 8 on 60020: exiting
> >> 2010-12-09 04:32:33,139 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
> Server handler 1 on 60020: exiting
> >> 2010-12-09 04:32:33,139 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server handler 10 on 60020: exiting
> >> 2010-12-09 04:32:33,140 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
> Server handler 0 on 60020: exiting2010-12-09 04:32:33,140 INFO
> org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 5 on 60020:
> exiting
> >> 2010-12-09 04:32:33,139 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
> Server handler 8 on 60020: exiting2010-12-09 04:32:33,140 INFO
> org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 4 on 60020:
> exiting
> >> 2010-12-09 04:32:33,140 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
> Server handler 3 on 60020: exiting2010-12-09 04:32:33,142 INFO
> org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 7 on 60020:
> exiting
> >> 2010-12-09 04:32:33,142 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
> Server handler 9 on 60020: exiting2010-12-09 04:32:33,143 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
> >> 2010-12-09 04:32:33,143 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
> Server handler 2 on 60020: exiting
> >> 2010-12-09 04:32:33,143 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> IPC Server Responder
> >> 2010-12-09 04:32:33,143 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
> Server handler 6 on 60020: exiting
> >> 2010-12-09 04:32:33,144 INFO org.mortbay.log: Stopped
> SelectChannelConnector@0.0.0.0:60030
> >>
> >>
> >>
> >>
> >> NAMENODE ->
> >> 2010-12-09 04:32:08,408 WARN org.apache.hadoop.hdfs.StateChange: DIR*
> NameSystem.startFile: failed to create file
> /hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555
> for DFSClient_hb_m_10.194.194.79:60000_1291788452343 on client
> 10.194.194.79, because this file is already being created by NN_Recovery on
> 10.100.159.13
> >> 2010-12-09 04:32:08,408 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 6 on 54310, call
> append(/hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555,
> DFSClient_hb_m_10.194.194.79:60000_1291788452343) from 10.194.194.79:59118:
> error: org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed
> to create file /hba
> >> se/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555
> for DFSClient_hb_m_10.194.194.79:60000_1291788452343 on client
> 10.194.194.79, because this file is already being created by NN_Recovery on
> 10.100.159.13
> >> org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to
> create file /hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555
> for DFSClient_hb_m_10.194.194.79:60000_1291788452343 on client
> 10.194.194.79, because this file is already being created by NN_Recovery on
> 10.100.159.13
> >> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1194)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:1282)
> >> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.append(NameNode.java:541)
> at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
> >> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> >> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:528) at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1319)
> >> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1315)
> at java.security.AccessController.doPrivileged(Native Method)
> >> at javax.security.auth.Subject.doAs(Subject.java:396) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
> >> at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1313)2010-12-09
> 04:32:09,411 WARN org.apache.hadoop.hdfs.StateChange: DIR*
> NameSystem.startFile: failed to create file
> /hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A6
> >> 0020.1291868481555 for DFSClient_hb_m_10.194.194.79:60000_1291788452343
> on client 10.194.194.79, because this file is already being created by
> NN_Recovery on 10.100.159.132010-12-09 04:32:09,411 INFO
> org.apache.hadoop.ipc.Server: IPC Server handler 9 on 54310, call
> append(/hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13
> %3A60020.1291868
> >> 481555, DFSClient_hb_m_10.194.194.79:60000_1291788452343) from
> 10.194.194.79:59118: error:
> org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to
> create file /hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555
> for DFSClient_hb_m_10.194.194.79:60000_1291788452343 on client
> 10.194.194.79, because this file is
> >> already being created by NN_Recovery on
> 10.100.159.13org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException:
> failed to create file /hbase/.logs/05.hadoop.jive,60020,1291862294517/
> 10.100.159.13%3A60020.1291868481555 for DFSClie
> >> nt_hb_m_10.194.194.79:60000_1291788452343 on client 10.194.194.79,
> because this file is already being created by NN_Recovery on 10.100.159.13
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1194)
> >> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:1282)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.append(NameNode.java:541)
> >> at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> at java.lang.reflect.Method.invoke(Method.java:597) at
> org.apache.hadoop.ipc.RPC$Server.call(RPC.java:528)
> >> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1319)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1315)
> >>
> >>
> >>
> >> DATANODE
> >>
> >> 2010-12-09 04:32:09,534 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder
> blk_2012253533858803341_250248 1 : Thread is interrupted.
> >> 2010-12-09 04:32:09,534 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
> 10.100.159.13:50010,
> storageID=DS-145997007-10.100.159.13-50010-1291262743182, infoPort=50075,
> ipcPort=50020):DataXceiver
> >> java.io.IOException: Interrupted receiveBlock
> >> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:579)
> >> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:417)
> >> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:122)
> >> 2010-12-09 04:32:09,534 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 1 for block
> blk_2012253533858803341_250248 terminating
> >>
> >>
> >>
> ************************************************************************************
> >> 06.hadoop
> >>
> ************************************************************************************
> >> REGION SERVER ->
> >>
> >> 2010-12-09 06:17:47,215 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> NotServingRegionException;
> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
> >> 2010-12-09 06:17:47,430 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> NotServingRegionException;
> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
> >> 2010-12-09 06:17:47,549 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> NotServingRegionException;
> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
> >> 2010-12-09 06:17:47,557 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> NotServingRegionException;
> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
> >> 2010-12-09 06:17:47,563 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> NotServingRegionException;
> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
> >> 2010-12-09 06:17:48,619 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> NotServingRegionException;
> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
> >> 2010-12-09 06:17:48,619 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> NotServingRegionException;
> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
> >> 2010-12-09 06:17:48,619 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> NotServingRegionException;
> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
> >> 2010-12-09 06:17:48,619 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> NotServingRegionException;
> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
> >> 2010-12-09 06:17:48,619 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> NotServingRegionException;
> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
> >> 2010-12-09 06:17:49,703 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> NotServingRegionException;
> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
> >> 2010-12-09 06:17:49,703 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> NotServingRegionException;
> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
> >> 2010-12-09 06:17:49,703 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> NotServingRegionException;
> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
> >> 2010-12-09 06:17:49,705 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> NotServingRegionException;
> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
> >> 2010-12-09 06:17:49,729 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> NotServingRegionException;
> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
> >>
> >> A bunch of these ^
> >>
> >> Then a little later::
> >> ....
> >>
> >> 2010-12-09 06:23:23,561 DEBUG
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction
> requested for region
> article,993143fca3613014990cc6975cacdd62f4147e03,12918133247
> >> 53.19413f33fc660536d9d4f315a1283440. because:
> regionserver60020.cacheFlusher
> >> 2010-12-09 06:23:27,058 INFO
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs
> -- HDFS-200
> >> 2010-12-09 06:23:27,080 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLog: Roll
> /hbase/.logs/06.hadoop.****,60020,1291864620851/10.99.97.106%3A60020.1291875047947,
> entries=53315,
> >> filesize=63781925. New hlog
> /hbase/.logs/06.hadoop.****,60020,1291864620851/10.99.97.106
> %3A60020.1291875807006
> >> 2010-12-09 06:23:27,080 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLog: Found 1 hlogs to remove out
> of total 8; oldest outstanding sequenceid is 33526881 from region article
> >>
> ,1221b0013c61fd7446cb822f698d0bcafa487342,1291872610788.bb2aa6ae8c694a096a00e7bb91fc46c3.
> >> 2010-12-09 06:23:27,080 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLog: moving old hlog file
> /hbase/.logs/06.hadoop.****,60020,1291864620851/10.99.97.106
> %3A60020.1291872064424
> >> whose highest sequenceid is 33506639 to /hbase/.oldlogs/10.99.97.106
> %3A60020.1291872064424
> >> 2010-12-09 06:25:39,162 WARN
> org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler 12 on
> 60020 took 119034ms appending an edit to hlog; editcount=3683
> >> 2010-12-09 06:25:39,162 WARN org.apache.hadoop.hdfs.DFSClient:
> DataStreamer Exception: java.io.IOException: Broken pipe
> >> at sun.nio.ch.FileDispatcher.write0(Native Method)
> >> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
> >> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100)
> >> at sun.nio.ch.IOUtil.write(IOUtil.java:71)
> >> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
> >> at
> org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
> >> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
> >> at
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
> >> at
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
> >> at
> java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
> >> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> >> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2509)
> >> 2010-12-09 06:25:39,163 WARN org.apache.hadoop.hdfs.DFSClient:
> DFSOutputStream ResponseProcessor exception for block
> blk_1141147913164105086_258331java.io.EOFException
> >> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> >> at java.io.DataInputStream.readLong(DataInputStream.java:399)
> >> at
> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:120)
> >> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2621)
> >>
> >> 2010-12-09 06:25:39,163 INFO org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_1141147913164105086_258331 waiting for responder to
> exit.
> >> 2010-12-09 06:25:39,164 WARN org.apache.hadoop.hdfs.DFSClient:
> DFSOutputStream ResponseProcessor exception for block
> blk_-7076574401277381592_258324java.io.EOFException
> >> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> >> at java.io.DataInputStream.readLong(DataInputStream.java:399)
> >> at
> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:120)
> >> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2621)
> >>
> >> 2010-12-09 06:25:39,164 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_-7076574401277381592_258324 bad datanode[0]
> 10.99.97.106:50010
> >> 2010-12-09 06:25:39,164 INFO org.apache.zookeeper.ClientCnxn: Unable to
> read additional data from server sessionid 0x12ca58c08ff004b, likely server
> has closed socket, closing socket connection and attempting reconnect
> >> 2010-12-09 06:25:39,164 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_-7076574401277381592_258324 in pipeline
> 10.99.97.106:50010, 10.100.154.103:50010: bad datanode 10.99.97.106:50010
> >> 2010-12-09 06:25:39,165 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_1141147913164105086_258331 bad datanode[0]
> 10.99.97.106:50010
> >> 2010-12-09 06:25:39,167 INFO org.apache.zookeeper.ClientCnxn: Unable to
> read additional data from server sessionid 0x12ca58c08ff004c, likely server
> has closed socket, closing socket connection and attempting reconnect
> >> 2010-12-09 06:25:39,169 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.100.154.103:50020. Already tried 0 time(s).
> >> 2010-12-09 06:25:39,169 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_1141147913164105086_258331 in pipeline
> 10.99.97.106:50010, 10.100.159.13:50010: bad datanode 10.99.97.106:50010
> >> 2010-12-09 06:25:39,209 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGIONSERVER_STOP
> >> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> server on 60020
> >> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server handler 3 on 60020: exiting
> >> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server handler 9 on 60020: exiting
> >> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server handler 24 on 60020: exiting
> >> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
> Server handler 1 on 60020: exiting
> >> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server handler 15 on 60020: exiting
> >> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
> Server handler 0 on 60020: exiting
> >> 2010-12-09 06:25:39,214 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
> Server handler 3 on 60020: exiting
> >> 2010-12-09 06:25:39,214 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> IPC Server listener on 60020
> >> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
> Server handler 2 on 60020: exiting
> >> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
> Server handler 7 on 60020: exiting
> >> 2010-12-09 06:25:39,213 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server handler 21 on 60020: exiting
> >> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
> Server handler 4 on 60020: exiting
> >> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> IPC Server Responder
> >> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
> Server handler 5 on 60020: exiting
> >> 2010-12-09 06:25:39,216 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
> Server handler 8 on 60020: exiting
> >> 2010-12-09 06:25:39,216 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
> Server handler 9 on 60020: exiting
> >> 2010-12-09 06:25:39,216 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC
> Server handler 6 on 60020: exiting
> >> 2010-12-09 06:25:39,216 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
> >> 2010-12-09 06:25:39,223 WARN org.apache.hadoop.hdfs.DFSClient: Failed
> recovery attempt #0 from primary datanode 10.100.159.13:50010
> >>
> >>
> >>
> >>
>
>
Re: Crashing Region Servers
Posted by Lance Riedel <la...@gmail.com>.
Seems reasonable, but having trouble making sense of the GC logs I had turned on. Basically since there was a full GC a minute before this happens on that server that lasts less than a second.
Example:
So, here is what the last of the GC logs say for that Regionserver (04.hadoop on 10.100.154.10) (note the last full GC only 1 minutes before @ 5:03:29.850+0000):
2010-12-09T05:02:19.472+0000: 12170.437: [GC [PSYoungGen: 278229K->5024K(287360K)] 694269K->421693K(721856K), 0.0274840 secs] [Times: user=0.04 sys=0.00, real=0.03 secs]
2010-12-09T05:02:55.479+0000: 12206.444: [GC [PSYoungGen: 281450K->7968K(273344K)] 698120K->429658K(707840K), 0.0503000 secs] [Times: user=0.10 sys=0.00, real=0.05 secs]
2010-12-09T05:03:29.800+0000: 12240.766: [GC [PSYoungGen: 273295K->7712K(262464K)] 694986K->437306K(696960K), 0.0490770 secs] [Times: user=0.12 sys=0.01, real=0.05 secs]
2010-12-09T05:03:29.850+0000: 12240.815: [Full GC [PSYoungGen: 7712K->0K(262464K)] [PSOldGen: 429594K->353152K(429824K)] 437306K->353152K(692288K) [PSPermGen: 18016K->18016K(21248K)], 0.8960760 secs] [Times: user=0.89 sys=0.00, real=0.90 secs]
2010-12-09T05:04:03.345+0000: 12274.310: [GC [PSYoungGen: 254720K->6944K(251584K)] 607872K->360528K(681408K), 0.0331060 secs] [Times: user=0.03 sys=0.00, real=0.04 secs]
2010-12-09T05:04:33.875+0000: 12304.840: [GC [PSYoungGen: 251552K->7200K(242240K)] 605136K->367800K(672064K), 0.0336420 secs] [Times: user=0.09 sys=0.01, real=0.03 secs]
Heap
PSYoungGen total 242240K, used 223390K [0x00007fcc640b0000, 0x00007fcc77a90000, 0x00007fcc78e00000)
eden space 235008K, 91% used [0x00007fcc640b0000,0x00007fcc713cfa30,0x00007fcc72630000)
from space 7232K, 99% used [0x00007fcc769b0000,0x00007fcc770b8090,0x00007fcc770c0000)
to space 10048K, 0% used [0x00007fcc770c0000,0x00007fcc770c0000,0x00007fcc77a90000)
PSOldGen total 429824K, used 360600K [0x00007fcc3a600000, 0x00007fcc549c0000, 0x00007fcc640b0000)
object space 429824K, 83% used [0x00007fcc3a600000,0x00007fcc50626060,0x00007fcc549c0000)
PSPermGen total 21248K, used 18131K [0x00007fcc35200000, 0x00007fcc366c0000, 0x00007fcc3a600000)
object space 21248K, 85% used [0x00007fcc35200000,0x00007fcc363b4d60,0x00007fcc366c0000)
Now Dead / End of log
That said, I will continue down the path you pointed me on .
Thanks,
Lance
On Dec 9, 2010, at 11:40 AM, Jean-Daniel Cryans wrote:
> Lance,
>
> Both those lines indicate the problem:
>
> IPC Server handler 13 on 60020 took 182416ms
> Client session timed out, have not heard from server in 182936ms
>
> It's very clear that your region servers are suffering from
> pause-of-the-world garbage collection issues. Basically this one GC'ed
> for 3 minutes, which is over the 1 minute timeout to consider a region
> server dead. The rest is just the side effect of that.
>
> This subject comes often on the mailing list, for example:
> http://search-hadoop.com/m/t97q027tr7f2
>
> Make sure you don't swap, give plenty of RAM to HBase, use LZO
> compression, don't underprovision your machines, etc
>
> J-D
>
> On Thu, Dec 9, 2010 at 11:27 AM, Lance Riedel <la...@gmail.com> wrote:
>>
>> We have a 6 node cluster, 5 with region serves. 2 of the region servers have been stable for days, but 3 of them keep crashing. Here are the logs around around when the crash occurs. (btw, we are shoving approximately the twitter firehose into hbase via flume) I'm an hbase newbie, but I have been reading. Not sure what else is needed to help debug. When the problem occurs, it is concurrent with issues that appear all the way down in the data node. Also, interesting to note, all 3 servers seem to fail differently somewhat:
>>
>> Servers (6):
>> 00.hadoop is the master
>> 01.hadoop, 03.hadoop (there is no 02.hadoop - flaky machine that got taken out) - are stable Region Servers
>> 04-06.hadoop crash at different times, very different logs, Region Servers
>>
>>
>> Master Attributes:
>> HBase Version 0.89.20100924+28, r
>> Hadoop Version 0.20.2+737, r98c55c28258aa6f42250569bd7fa431ac657bdbd
>>
>> HBase Root Directory hdfs://00.hadoop.****:54310/hbase
>>
>> Load average
>> 103.0
>> Average number of regions per regionserver. Naive computation.
>> Regions On FS
>> 577
>>
>>
>> (This is after recently restarting 04-06)
>>
>>
>> Address Start Code Load
>> 01.hadoop:60030 1291268910638 requests=223, regions=118, usedHeap=519, maxHeap=987
>> 03.hadoop:60030 1291269219610 requests=23, regions=111, usedHeap=862, maxHeap=987
>> 04.hadoop:60030 1291910665912 requests=169, regions=82, usedHeap=194, maxHeap=888
>> 05.hadoop:60030 1291909584060 requests=232, regions=110, usedHeap=477, maxHeap=888
>> 06.hadoop60030 1291909723787 requests=99, regions=94, usedHeap=394, maxHeap=888
>> Total:
>> servers: 5
>> requests=746, regions=515
>>
>>
>>
>> Following are various logs around the time of the failures for 04-06
>>
>> ************************************************************************************
>> 04.hadoop
>> ************************************************************************************
>> REGION SERVER ->
>> 2010-12-09 05:01:58,640 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region article,a83858a08f2270d319f75a7b43c756c2453988e7,1291811929868.58f6d9fc80
>> c78f3ca490b0280b4f1226. in 0sec
>> 2010-12-09 05:04:30,898 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.46 MB, free=176.33 MB, max=177.79 MB, blocks=0, accesses=200583, hits=0, hitRatio=
>> 0.00%%, evictions=0, evicted=0, evictedPerRun=NaN
>> 2010-12-09 05:04:59,612 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs -- HDFS-200
>> 2010-12-09 05:04:59,618 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Roll /hbase/.logs/04.hadoop.****,60020,1291858770125/10.100.154.103%3A60020.1291870585253, entries=5318
>> 6, filesize=63776311. New hlog /hbase/.logs/04.hadoop.****,60020,1291858770125/10.100.154.103%3A60020.1291871099582
>> 2010-12-09 05:08:02,033 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: Found 1 hlogs to remove out of total 6; oldest outstanding sequenceid is 32891801 from region article
>> ,4b2039b791e894dd479b90661ca97087f61645d3,1291813201039.7dff47c63bb5648bb1ee6670c60553a5.
>> 2010-12-09 05:08:02,034 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler 13 on 60020 took 182416ms appending an edit to hlog; editcount=0
>> 2010-12-09 05:08:02,066 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: moving old hlog file /hbase/.logs/04.hadoop.****,60020,1291858770125/10.100.154.103%3A60020.12918679476
>> 65 whose highest sequenceid is 32880227 to /hbase/.oldlogs/10.100.154.103%3A60020.1291867947665
>> 2010-12-09 05:08:02,066 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 182936ms for sessionid 0x12ca58c08ff0048, closing socket conn
>> ection and attempting reconnect
>> 2010-12-09 05:08:02,089 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: Aborting region server serverName=04.hadoop.****,60020,1291858770125, load=(requests=24, regions=
>> 64, usedHeap=548, maxHeap=888): Unhandled exception
>> org.apache.hadoop.hbase.YouAreDeadException: org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing 04.hadoop.****,60020,1291858770125 as dead s
>> erver
>> at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:217)
>> at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:271)
>> at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:744)
>> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:561)
>> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>>
>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>> at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
>> at org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
>> at org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
>> at org.apache.hadoop.hbase.regionserver.HRegionServer.doRun(HRegionServer.java:635)
>> at org.apache.hadoop.hbase.regionserver.HRegionServer.access$000(HRegionServer.java:126)
>> at org.apache.hadoop.hbase.regionserver.HRegionServer$1.run(HRegionServer.java:518)
>> at org.apache.hadoop.hbase.regionserver.HRegionServer$1.run(HRegionServer.java:516)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:337)
>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1046)
>> at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:516)
>> at java.lang.Thread.run(Thread.java:662)
>> 2010-12-09 05:08:02,090 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: request=0.0, regions=64, stores=64, storefiles=136, storefileIndexSize=27, memsto
>> reSize=353, compactionQueueSize=0, usedHeap=549, maxHeap=888, blockCacheSize=1530552, blockCacheFree=184893160, blockCacheCount=0, blockCacheHitRatio=0
>> 2010-12-09 05:08:02,090 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60020
>> 2010-12-09 05:08:02,090 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 0 on 60020: exiting2010-12-09 05:08:02,090 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server listener on 60020
>>
>>
>> MASTER HBASE ->
>>
>> 2010-12-09 05:05:06,072 INFO org.apache.hadoop.hbase.master.BaseScanner: All 1 .META. region(s) scanned
>> 2010-12-09 05:05:40,004 INFO org.apache.hadoop.hbase.master.ServerManager: 04.hadoop.jive,60020,1291858770125 znode expired
>> 2010-12-09 05:05:40,005 DEBUG org.apache.hadoop.hbase.master.ServerManager: Added=04.hadoop.jive,60020,1291858770125 to dead servers, added shutdown processing operation
>> 2010-12-09 05:05:40,005 DEBUG org.apache.hadoop.hbase.master.RegionServerOperationQueue: Processing todo: ProcessServerShutdown of 04.hadoop.jive,60020,12918587701252010-12-09 05:05:40,005 INFO org.apache.hadoop.hbase.master.RegionServerOperation: Process shutdown of server 04.hadoop.jive,60020,1291858770125: logSplit: false, rootRescanned: f
>> alse, numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
>> 2010-12-09 05:05:40,008 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Splitting 7 hlog(s) in hdfs://00.hadoop.jive:54310/hbase/.logs/04.hadoop.jive,60020,12918587701252010-12-09 05:05:40,008 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: Splitting hlog 1 of 7: hdfs://00.hadoop.jive:54310/hbase/.logs/04.hadoop.jive,60020,1291858770125/10.1
>> 00.154.103%3A60020.1291867947665, length=637577092010-12-09 05:05:40,008 INFO org.apache.hadoop.hbase.util.FSUtils: Recovering filehdfs://00.hadoop.jive:54310/hbase/.logs/04.hadoop.jive,60020,1291858770125/10.100.154.103%3A60020
>> .1291867947665
>>
>>
>> NAMENODE ->
>>
>> 2010-12-09 05:08:02,471 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 10.100.159.13:50010 is added to blk_1531008743226086399_251615 size 63757709
>> 2010-12-09 05:08:02,473 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 54310, call complete(/hbase/.logs/04.hadoop.jive,60020,1291858770125/10.100.154.103%3A60020.1291867947665, DFSClient_hb_m_10.194.194.79:60000_1291788452343) from 10.194.194.79:44117: error: org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /hbase/.logs/04.hadoop.jive,60020,1291858770125/10.100.154.103%3A60020.1291867947665 File does not exist. [Lease. Holder: DFSClient_hb_m_10.194.194.79:60000_1291788452343, pendingcreates: 1]
>> org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /hbase/.logs/04.hadoop.jive,60020,1291858770125/10.100.154.103%3A60020.1291867947665 File does not exist. [Lease. Holder: DFSClient_hb_m_10.194.194.79:60000_1291788452343, pendingcreates: 1]
>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1488)
>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1479)
>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:1534)
>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:1522)
>> at org.apache.hadoop.hdfs.server.namenode.NameNode.complete(NameNode.java:610)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:528)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1319)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1315)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1313)
>> 2010-12-09 05:08:04,206 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 10.99.97.106:50010 is added to blk_-734280257049179934_251614 size 7330
>>
>> DATANODE ->
>>
>> 2010-12-09 05:08:02,212 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in BlockReceiver.lastNodeRun: java.io.IOException: Broken pipe
>> at sun.nio.ch.FileDispatcher.write0(Native Method)
>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100) at sun.nio.ch.IOUtil.write(IOUtil.java:71)
>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
>> at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142) at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
>> at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107) at java.io.DataOutputStream.writeLong(DataOutputStream.java:207)
>> at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.write(DataTransferProtocol.java:133) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.lastDataNodeRun(BlockReceiver.java:840)
>> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:870)
>> at java.lang.Thread.run(Thread.java:662)
>> 2010-12-09 05:08:02,213 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: checkDiskError: exception:
>> 2010-12-09 05:08:02,213 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: checkDiskError: exception: java.io.IOException: Broken pipe
>> at sun.nio.ch.FileDispatcher.write0(Native Method)
>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100)
>> at sun.nio.ch.IOUtil.write(IOUtil.java:71)
>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
>> at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
>> at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
>> at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
>> at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
>> at java.io.DataOutputStream.writeLong(DataOutputStream.java:207)
>> at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.write(DataTransferProtocol.java:133)
>> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.lastDataNodeRun(BlockReceiver.java:840)
>> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:870)
>> at java.lang.Thread.run(Thread.java:662)
>> 2010-12-09 05:08:02,221 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder blk_-8817504198034990390_251613 0 Exception java.io.IOException: Broken pipe
>> at sun.nio.ch.FileDispatcher.write0(Native Method)
>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100)
>> at sun.nio.ch.IOUtil.write(IOUtil.java:71)
>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
>> at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
>> at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
>> at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
>> at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
>> at java.io.DataOutputStream.writeLong(DataOutputStream.java:207)
>> at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.write(DataTransferProtocol.java:133)
>> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.lastDataNodeRun(BlockReceiver.java:840)
>> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:870)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> 2010-12-09 05:08:02,221 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block blk_-8817504198034990390_251613 terminating
>> 2010-12-09 05:08:02,224 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock blk_-8817504198034990390_251613 received exception java.io.IOException: Connection reset by peer
>> 2010-12-09 05:08:02,224 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_4505306716668305567_250752 file /dist1/data/hadoop-data/current/subdir19/blk_4505306716668305567 for deletion
>> 2010-12-09 05:08:02,226 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(10.100.154.103:50010, storageID=DS-866555524-10.100.154.103-50010-1291262739506, infoPort=50075, ipcPort=50020):DataXceiver
>> java.io.IOException: Connection reset by peer
>> at sun.nio.ch.FileDispatcher.read0(Native Method)
>> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
>> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:237)
>> at sun.nio.ch.IOUtil.read(IOUtil.java:210)
>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
>> at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:55)
>> at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
>> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
>> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
>> at java.io.DataInputStream.read(DataInputStream.java:132)
>> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:267)
>> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:357)
>> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:378)
>> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:534)
>> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:417)
>> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:122)
>> 2010-12-09 05:08:02,226 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block blk_1422641301942366074_250732 at file /dist1/data/hadoop-data/current/subdir19/blk_1422641301942366074
>> 2010-12-09 05:08:02,227 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block blk_4505306716668305567_250752 at file /dist1/data/hadoop-data/current/subdir19/blk_4505306716668305567
>>
>>
>>
>> ************************************************************************************
>> 05.hadoop
>> ************************************************************************************
>> REGION SERVER ->
>> 2010-12-09 04:29:25,047 DEBUG org.apache.hadoop.hbase.regionserver.Store: Compaction size of data: 250.5m; Skipped 1 file(s), size: 206896367
>> 2010-12-09 04:29:25,047 INFO org.apache.hadoop.hbase.regionserver.Store: Started compaction of 2 file(s) in data of article,060c4b75067378c045eead9edec2d4fb7d8f3e5e,1291812563166.95e92e97823e11e57850720a63433a62. into hdfs://00.hadoop.****:54310/hbase/article/95e92e97823e11e57850720a63433a62/.tmp, sequenceid=33050221
>> 2010-12-09 04:29:26,328 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 2 file(s) in data of article,060c4b75067378c045eead9edec2d4fb7d8f3e5e,1291812563166.95e92e97823e11e57850720a63433a62.; new storefile is hdfs://00.hadoop.****:54310/hbase/article/95e92e97823e11e57850720a63433a62/data/2787768957794924068; store size is 250.5m
>> 2010-12-09 04:29:26,328 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region article,060c4b75067378c045eead9edec2d4fb7d8f3e5e,1291812563166.95e92e97823e11e57850720a63433a62. in 1sec
>> 2010-12-09 04:29:26,328 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on region article,ed0cdc7ae6db73432e2ca47c973d545bee11ddd8,1291813400028.85a8d31d099f409dd36f3bcc4be1eba6.
>> 2010-12-09 04:29:26,330 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region article,ed0cdc7ae6db73432e2ca47c973d545bee11ddd8,1291813400028.85a8d31d099f409dd36f3bcc4be1eba6. in 0sec
>> 2010-12-09 04:32:33,081 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGIONSERVER_STOP
>> 2010-12-09 04:32:33,081 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60020
>> 2010-12-09 04:32:33,089 WARN org.apache.hadoop.hdfs.DFSClient: DFSOutputStream ResponseProcessor exception for block blk_2012253533858803341_250248java.io.EOFException
>> at java.io.DataInputStream.readFully(DataInputStream.java:180)
>> at java.io.DataInputStream.readLong(DataInputStream.java:399)
>> at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:120)
>> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2621)
>>
>> 2010-12-09 04:32:33,089 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x12ca58c08ff0049, likely server has closed socket, closing sock
>> et connection and attempting reconnect
>> 2010-12-09 04:32:33,089 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler 5 on 60020 took 75446ms appending an edit to hlog; editcount=50505
>> 2010-12-09 04:32:33,090 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 18 on 60020: exiting
>> 2010-12-09 04:32:33,139 INFO org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_2012253533858803341_250248 waiting for responder to exit.
>> 2010-12-09 04:32:33,139 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 8 on 60020: exiting
>> 2010-12-09 04:32:33,139 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 1 on 60020: exiting
>> 2010-12-09 04:32:33,139 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 10 on 60020: exiting
>> 2010-12-09 04:32:33,140 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 0 on 60020: exiting2010-12-09 04:32:33,140 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 5 on 60020: exiting
>> 2010-12-09 04:32:33,139 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 8 on 60020: exiting2010-12-09 04:32:33,140 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 4 on 60020: exiting
>> 2010-12-09 04:32:33,140 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 3 on 60020: exiting2010-12-09 04:32:33,142 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 7 on 60020: exiting
>> 2010-12-09 04:32:33,142 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 9 on 60020: exiting2010-12-09 04:32:33,143 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
>> 2010-12-09 04:32:33,143 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 2 on 60020: exiting
>> 2010-12-09 04:32:33,143 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server Responder
>> 2010-12-09 04:32:33,143 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 6 on 60020: exiting
>> 2010-12-09 04:32:33,144 INFO org.mortbay.log: Stopped SelectChannelConnector@0.0.0.0:60030
>>
>>
>>
>>
>> NAMENODE ->
>> 2010-12-09 04:32:08,408 WARN org.apache.hadoop.hdfs.StateChange: DIR* NameSystem.startFile: failed to create file /hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555 for DFSClient_hb_m_10.194.194.79:60000_1291788452343 on client 10.194.194.79, because this file is already being created by NN_Recovery on 10.100.159.13
>> 2010-12-09 04:32:08,408 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 54310, call append(/hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555, DFSClient_hb_m_10.194.194.79:60000_1291788452343) from 10.194.194.79:59118: error: org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to create file /hba
>> se/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555 for DFSClient_hb_m_10.194.194.79:60000_1291788452343 on client 10.194.194.79, because this file is already being created by NN_Recovery on 10.100.159.13
>> org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to create file /hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555 for DFSClient_hb_m_10.194.194.79:60000_1291788452343 on client 10.194.194.79, because this file is already being created by NN_Recovery on 10.100.159.13
>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1194) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:1282)
>> at org.apache.hadoop.hdfs.server.namenode.NameNode.append(NameNode.java:541) at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:528) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1319)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1315) at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1313)2010-12-09 04:32:09,411 WARN org.apache.hadoop.hdfs.StateChange: DIR* NameSystem.startFile: failed to create file /hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A6
>> 0020.1291868481555 for DFSClient_hb_m_10.194.194.79:60000_1291788452343 on client 10.194.194.79, because this file is already being created by NN_Recovery on 10.100.159.132010-12-09 04:32:09,411 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 54310, call append(/hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868
>> 481555, DFSClient_hb_m_10.194.194.79:60000_1291788452343) from 10.194.194.79:59118: error: org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to create file /hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555 for DFSClient_hb_m_10.194.194.79:60000_1291788452343 on client 10.194.194.79, because this file is
>> already being created by NN_Recovery on 10.100.159.13org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to create file /hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555 for DFSClie
>> nt_hb_m_10.194.194.79:60000_1291788452343 on client 10.194.194.79, because this file is already being created by NN_Recovery on 10.100.159.13 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1194)
>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:1282) at org.apache.hadoop.hdfs.server.namenode.NameNode.append(NameNode.java:541)
>> at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:528)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1319) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1315)
>>
>>
>>
>> DATANODE
>>
>> 2010-12-09 04:32:09,534 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder blk_2012253533858803341_250248 1 : Thread is interrupted.
>> 2010-12-09 04:32:09,534 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(10.100.159.13:50010, storageID=DS-145997007-10.100.159.13-50010-1291262743182, infoPort=50075, ipcPort=50020):DataXceiver
>> java.io.IOException: Interrupted receiveBlock
>> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:579)
>> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:417)
>> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:122)
>> 2010-12-09 04:32:09,534 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 1 for block blk_2012253533858803341_250248 terminating
>>
>>
>> ************************************************************************************
>> 06.hadoop
>> ************************************************************************************
>> REGION SERVER ->
>>
>> 2010-12-09 06:17:47,215 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>> 2010-12-09 06:17:47,430 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>> 2010-12-09 06:17:47,549 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>> 2010-12-09 06:17:47,557 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>> 2010-12-09 06:17:47,563 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>> 2010-12-09 06:17:48,619 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>> 2010-12-09 06:17:48,619 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>> 2010-12-09 06:17:48,619 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>> 2010-12-09 06:17:48,619 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>> 2010-12-09 06:17:48,619 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>> 2010-12-09 06:17:49,703 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>> 2010-12-09 06:17:49,703 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>> 2010-12-09 06:17:49,703 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>> 2010-12-09 06:17:49,705 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>> 2010-12-09 06:17:49,729 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>>
>> A bunch of these ^
>>
>> Then a little later::
>> ....
>>
>> 2010-12-09 06:23:23,561 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction requested for region article,993143fca3613014990cc6975cacdd62f4147e03,12918133247
>> 53.19413f33fc660536d9d4f315a1283440. because: regionserver60020.cacheFlusher
>> 2010-12-09 06:23:27,058 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs -- HDFS-200
>> 2010-12-09 06:23:27,080 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Roll /hbase/.logs/06.hadoop.****,60020,1291864620851/10.99.97.106%3A60020.1291875047947, entries=53315,
>> filesize=63781925. New hlog /hbase/.logs/06.hadoop.****,60020,1291864620851/10.99.97.106%3A60020.1291875807006
>> 2010-12-09 06:23:27,080 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: Found 1 hlogs to remove out of total 8; oldest outstanding sequenceid is 33526881 from region article
>> ,1221b0013c61fd7446cb822f698d0bcafa487342,1291872610788.bb2aa6ae8c694a096a00e7bb91fc46c3.
>> 2010-12-09 06:23:27,080 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: moving old hlog file /hbase/.logs/06.hadoop.****,60020,1291864620851/10.99.97.106%3A60020.1291872064424
>> whose highest sequenceid is 33506639 to /hbase/.oldlogs/10.99.97.106%3A60020.1291872064424
>> 2010-12-09 06:25:39,162 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler 12 on 60020 took 119034ms appending an edit to hlog; editcount=3683
>> 2010-12-09 06:25:39,162 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception: java.io.IOException: Broken pipe
>> at sun.nio.ch.FileDispatcher.write0(Native Method)
>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100)
>> at sun.nio.ch.IOUtil.write(IOUtil.java:71)
>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
>> at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
>> at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
>> at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
>> at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
>> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
>> at java.io.DataOutputStream.write(DataOutputStream.java:90)
>> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2509)
>> 2010-12-09 06:25:39,163 WARN org.apache.hadoop.hdfs.DFSClient: DFSOutputStream ResponseProcessor exception for block blk_1141147913164105086_258331java.io.EOFException
>> at java.io.DataInputStream.readFully(DataInputStream.java:180)
>> at java.io.DataInputStream.readLong(DataInputStream.java:399)
>> at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:120)
>> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2621)
>>
>> 2010-12-09 06:25:39,163 INFO org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_1141147913164105086_258331 waiting for responder to exit.
>> 2010-12-09 06:25:39,164 WARN org.apache.hadoop.hdfs.DFSClient: DFSOutputStream ResponseProcessor exception for block blk_-7076574401277381592_258324java.io.EOFException
>> at java.io.DataInputStream.readFully(DataInputStream.java:180)
>> at java.io.DataInputStream.readLong(DataInputStream.java:399)
>> at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:120)
>> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2621)
>>
>> 2010-12-09 06:25:39,164 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_-7076574401277381592_258324 bad datanode[0] 10.99.97.106:50010
>> 2010-12-09 06:25:39,164 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x12ca58c08ff004b, likely server has closed socket, closing socket connection and attempting reconnect
>> 2010-12-09 06:25:39,164 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_-7076574401277381592_258324 in pipeline 10.99.97.106:50010, 10.100.154.103:50010: bad datanode 10.99.97.106:50010
>> 2010-12-09 06:25:39,165 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_1141147913164105086_258331 bad datanode[0] 10.99.97.106:50010
>> 2010-12-09 06:25:39,167 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x12ca58c08ff004c, likely server has closed socket, closing socket connection and attempting reconnect
>> 2010-12-09 06:25:39,169 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.100.154.103:50020. Already tried 0 time(s).
>> 2010-12-09 06:25:39,169 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_1141147913164105086_258331 in pipeline 10.99.97.106:50010, 10.100.159.13:50010: bad datanode 10.99.97.106:50010
>> 2010-12-09 06:25:39,209 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGIONSERVER_STOP
>> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60020
>> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 3 on 60020: exiting
>> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 9 on 60020: exiting
>> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 24 on 60020: exiting
>> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 1 on 60020: exiting
>> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 15 on 60020: exiting
>> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 0 on 60020: exiting
>> 2010-12-09 06:25:39,214 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 3 on 60020: exiting
>> 2010-12-09 06:25:39,214 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server listener on 60020
>> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 2 on 60020: exiting
>> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 7 on 60020: exiting
>> 2010-12-09 06:25:39,213 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 21 on 60020: exiting
>> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 4 on 60020: exiting
>> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server Responder
>> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 5 on 60020: exiting
>> 2010-12-09 06:25:39,216 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 8 on 60020: exiting
>> 2010-12-09 06:25:39,216 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 9 on 60020: exiting
>> 2010-12-09 06:25:39,216 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 6 on 60020: exiting
>> 2010-12-09 06:25:39,216 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
>> 2010-12-09 06:25:39,223 WARN org.apache.hadoop.hdfs.DFSClient: Failed recovery attempt #0 from primary datanode 10.100.159.13:50010
>>
>>
>>
>>
Re: Crashing Region Servers
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Lance,
Both those lines indicate the problem:
IPC Server handler 13 on 60020 took 182416ms
Client session timed out, have not heard from server in 182936ms
It's very clear that your region servers are suffering from
pause-of-the-world garbage collection issues. Basically this one GC'ed
for 3 minutes, which is over the 1 minute timeout to consider a region
server dead. The rest is just the side effect of that.
This subject comes often on the mailing list, for example:
http://search-hadoop.com/m/t97q027tr7f2
Make sure you don't swap, give plenty of RAM to HBase, use LZO
compression, don't underprovision your machines, etc
J-D
On Thu, Dec 9, 2010 at 11:27 AM, Lance Riedel <la...@gmail.com> wrote:
>
> We have a 6 node cluster, 5 with region serves. 2 of the region servers have been stable for days, but 3 of them keep crashing. Here are the logs around around when the crash occurs. (btw, we are shoving approximately the twitter firehose into hbase via flume) I'm an hbase newbie, but I have been reading. Not sure what else is needed to help debug. When the problem occurs, it is concurrent with issues that appear all the way down in the data node. Also, interesting to note, all 3 servers seem to fail differently somewhat:
>
> Servers (6):
> 00.hadoop is the master
> 01.hadoop, 03.hadoop (there is no 02.hadoop - flaky machine that got taken out) - are stable Region Servers
> 04-06.hadoop crash at different times, very different logs, Region Servers
>
>
> Master Attributes:
> HBase Version 0.89.20100924+28, r
> Hadoop Version 0.20.2+737, r98c55c28258aa6f42250569bd7fa431ac657bdbd
>
> HBase Root Directory hdfs://00.hadoop.****:54310/hbase
>
> Load average
> 103.0
> Average number of regions per regionserver. Naive computation.
> Regions On FS
> 577
>
>
> (This is after recently restarting 04-06)
>
>
> Address Start Code Load
> 01.hadoop:60030 1291268910638 requests=223, regions=118, usedHeap=519, maxHeap=987
> 03.hadoop:60030 1291269219610 requests=23, regions=111, usedHeap=862, maxHeap=987
> 04.hadoop:60030 1291910665912 requests=169, regions=82, usedHeap=194, maxHeap=888
> 05.hadoop:60030 1291909584060 requests=232, regions=110, usedHeap=477, maxHeap=888
> 06.hadoop60030 1291909723787 requests=99, regions=94, usedHeap=394, maxHeap=888
> Total:
> servers: 5
> requests=746, regions=515
>
>
>
> Following are various logs around the time of the failures for 04-06
>
> ************************************************************************************
> 04.hadoop
> ************************************************************************************
> REGION SERVER ->
> 2010-12-09 05:01:58,640 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region article,a83858a08f2270d319f75a7b43c756c2453988e7,1291811929868.58f6d9fc80
> c78f3ca490b0280b4f1226. in 0sec
> 2010-12-09 05:04:30,898 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.46 MB, free=176.33 MB, max=177.79 MB, blocks=0, accesses=200583, hits=0, hitRatio=
> 0.00%%, evictions=0, evicted=0, evictedPerRun=NaN
> 2010-12-09 05:04:59,612 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs -- HDFS-200
> 2010-12-09 05:04:59,618 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Roll /hbase/.logs/04.hadoop.****,60020,1291858770125/10.100.154.103%3A60020.1291870585253, entries=5318
> 6, filesize=63776311. New hlog /hbase/.logs/04.hadoop.****,60020,1291858770125/10.100.154.103%3A60020.1291871099582
> 2010-12-09 05:08:02,033 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: Found 1 hlogs to remove out of total 6; oldest outstanding sequenceid is 32891801 from region article
> ,4b2039b791e894dd479b90661ca97087f61645d3,1291813201039.7dff47c63bb5648bb1ee6670c60553a5.
> 2010-12-09 05:08:02,034 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler 13 on 60020 took 182416ms appending an edit to hlog; editcount=0
> 2010-12-09 05:08:02,066 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: moving old hlog file /hbase/.logs/04.hadoop.****,60020,1291858770125/10.100.154.103%3A60020.12918679476
> 65 whose highest sequenceid is 32880227 to /hbase/.oldlogs/10.100.154.103%3A60020.1291867947665
> 2010-12-09 05:08:02,066 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 182936ms for sessionid 0x12ca58c08ff0048, closing socket conn
> ection and attempting reconnect
> 2010-12-09 05:08:02,089 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: Aborting region server serverName=04.hadoop.****,60020,1291858770125, load=(requests=24, regions=
> 64, usedHeap=548, maxHeap=888): Unhandled exception
> org.apache.hadoop.hbase.YouAreDeadException: org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing 04.hadoop.****,60020,1291858770125 as dead s
> erver
> at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:217)
> at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:271)
> at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:744)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:561)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
> at org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
> at org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.doRun(HRegionServer.java:635)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.access$000(HRegionServer.java:126)
> at org.apache.hadoop.hbase.regionserver.HRegionServer$1.run(HRegionServer.java:518)
> at org.apache.hadoop.hbase.regionserver.HRegionServer$1.run(HRegionServer.java:516)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:337)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1046)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:516)
> at java.lang.Thread.run(Thread.java:662)
> 2010-12-09 05:08:02,090 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: request=0.0, regions=64, stores=64, storefiles=136, storefileIndexSize=27, memsto
> reSize=353, compactionQueueSize=0, usedHeap=549, maxHeap=888, blockCacheSize=1530552, blockCacheFree=184893160, blockCacheCount=0, blockCacheHitRatio=0
> 2010-12-09 05:08:02,090 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60020
> 2010-12-09 05:08:02,090 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 0 on 60020: exiting2010-12-09 05:08:02,090 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server listener on 60020
>
>
> MASTER HBASE ->
>
> 2010-12-09 05:05:06,072 INFO org.apache.hadoop.hbase.master.BaseScanner: All 1 .META. region(s) scanned
> 2010-12-09 05:05:40,004 INFO org.apache.hadoop.hbase.master.ServerManager: 04.hadoop.jive,60020,1291858770125 znode expired
> 2010-12-09 05:05:40,005 DEBUG org.apache.hadoop.hbase.master.ServerManager: Added=04.hadoop.jive,60020,1291858770125 to dead servers, added shutdown processing operation
> 2010-12-09 05:05:40,005 DEBUG org.apache.hadoop.hbase.master.RegionServerOperationQueue: Processing todo: ProcessServerShutdown of 04.hadoop.jive,60020,12918587701252010-12-09 05:05:40,005 INFO org.apache.hadoop.hbase.master.RegionServerOperation: Process shutdown of server 04.hadoop.jive,60020,1291858770125: logSplit: false, rootRescanned: f
> alse, numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
> 2010-12-09 05:05:40,008 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Splitting 7 hlog(s) in hdfs://00.hadoop.jive:54310/hbase/.logs/04.hadoop.jive,60020,12918587701252010-12-09 05:05:40,008 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: Splitting hlog 1 of 7: hdfs://00.hadoop.jive:54310/hbase/.logs/04.hadoop.jive,60020,1291858770125/10.1
> 00.154.103%3A60020.1291867947665, length=637577092010-12-09 05:05:40,008 INFO org.apache.hadoop.hbase.util.FSUtils: Recovering filehdfs://00.hadoop.jive:54310/hbase/.logs/04.hadoop.jive,60020,1291858770125/10.100.154.103%3A60020
> .1291867947665
>
>
> NAMENODE ->
>
> 2010-12-09 05:08:02,471 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 10.100.159.13:50010 is added to blk_1531008743226086399_251615 size 63757709
> 2010-12-09 05:08:02,473 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 54310, call complete(/hbase/.logs/04.hadoop.jive,60020,1291858770125/10.100.154.103%3A60020.1291867947665, DFSClient_hb_m_10.194.194.79:60000_1291788452343) from 10.194.194.79:44117: error: org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /hbase/.logs/04.hadoop.jive,60020,1291858770125/10.100.154.103%3A60020.1291867947665 File does not exist. [Lease. Holder: DFSClient_hb_m_10.194.194.79:60000_1291788452343, pendingcreates: 1]
> org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /hbase/.logs/04.hadoop.jive,60020,1291858770125/10.100.154.103%3A60020.1291867947665 File does not exist. [Lease. Holder: DFSClient_hb_m_10.194.194.79:60000_1291788452343, pendingcreates: 1]
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1488)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1479)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:1534)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:1522)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.complete(NameNode.java:610)
> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:528)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1319)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1315)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1313)
> 2010-12-09 05:08:04,206 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 10.99.97.106:50010 is added to blk_-734280257049179934_251614 size 7330
>
> DATANODE ->
>
> 2010-12-09 05:08:02,212 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in BlockReceiver.lastNodeRun: java.io.IOException: Broken pipe
> at sun.nio.ch.FileDispatcher.write0(Native Method)
> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100) at sun.nio.ch.IOUtil.write(IOUtil.java:71)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
> at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142) at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
> at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107) at java.io.DataOutputStream.writeLong(DataOutputStream.java:207)
> at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.write(DataTransferProtocol.java:133) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.lastDataNodeRun(BlockReceiver.java:840)
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:870)
> at java.lang.Thread.run(Thread.java:662)
> 2010-12-09 05:08:02,213 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: checkDiskError: exception:
> 2010-12-09 05:08:02,213 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: checkDiskError: exception: java.io.IOException: Broken pipe
> at sun.nio.ch.FileDispatcher.write0(Native Method)
> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100)
> at sun.nio.ch.IOUtil.write(IOUtil.java:71)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
> at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
> at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
> at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
> at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
> at java.io.DataOutputStream.writeLong(DataOutputStream.java:207)
> at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.write(DataTransferProtocol.java:133)
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.lastDataNodeRun(BlockReceiver.java:840)
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:870)
> at java.lang.Thread.run(Thread.java:662)
> 2010-12-09 05:08:02,221 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder blk_-8817504198034990390_251613 0 Exception java.io.IOException: Broken pipe
> at sun.nio.ch.FileDispatcher.write0(Native Method)
> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100)
> at sun.nio.ch.IOUtil.write(IOUtil.java:71)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
> at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
> at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
> at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
> at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
> at java.io.DataOutputStream.writeLong(DataOutputStream.java:207)
> at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.write(DataTransferProtocol.java:133)
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.lastDataNodeRun(BlockReceiver.java:840)
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:870)
> at java.lang.Thread.run(Thread.java:662)
>
> 2010-12-09 05:08:02,221 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block blk_-8817504198034990390_251613 terminating
> 2010-12-09 05:08:02,224 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock blk_-8817504198034990390_251613 received exception java.io.IOException: Connection reset by peer
> 2010-12-09 05:08:02,224 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block blk_4505306716668305567_250752 file /dist1/data/hadoop-data/current/subdir19/blk_4505306716668305567 for deletion
> 2010-12-09 05:08:02,226 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(10.100.154.103:50010, storageID=DS-866555524-10.100.154.103-50010-1291262739506, infoPort=50075, ipcPort=50020):DataXceiver
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcher.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:237)
> at sun.nio.ch.IOUtil.read(IOUtil.java:210)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
> at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:55)
> at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> at java.io.DataInputStream.read(DataInputStream.java:132)
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:267)
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:357)
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:378)
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:534)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:417)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:122)
> 2010-12-09 05:08:02,226 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block blk_1422641301942366074_250732 at file /dist1/data/hadoop-data/current/subdir19/blk_1422641301942366074
> 2010-12-09 05:08:02,227 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block blk_4505306716668305567_250752 at file /dist1/data/hadoop-data/current/subdir19/blk_4505306716668305567
>
>
>
> ************************************************************************************
> 05.hadoop
> ************************************************************************************
> REGION SERVER ->
> 2010-12-09 04:29:25,047 DEBUG org.apache.hadoop.hbase.regionserver.Store: Compaction size of data: 250.5m; Skipped 1 file(s), size: 206896367
> 2010-12-09 04:29:25,047 INFO org.apache.hadoop.hbase.regionserver.Store: Started compaction of 2 file(s) in data of article,060c4b75067378c045eead9edec2d4fb7d8f3e5e,1291812563166.95e92e97823e11e57850720a63433a62. into hdfs://00.hadoop.****:54310/hbase/article/95e92e97823e11e57850720a63433a62/.tmp, sequenceid=33050221
> 2010-12-09 04:29:26,328 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 2 file(s) in data of article,060c4b75067378c045eead9edec2d4fb7d8f3e5e,1291812563166.95e92e97823e11e57850720a63433a62.; new storefile is hdfs://00.hadoop.****:54310/hbase/article/95e92e97823e11e57850720a63433a62/data/2787768957794924068; store size is 250.5m
> 2010-12-09 04:29:26,328 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region article,060c4b75067378c045eead9edec2d4fb7d8f3e5e,1291812563166.95e92e97823e11e57850720a63433a62. in 1sec
> 2010-12-09 04:29:26,328 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on region article,ed0cdc7ae6db73432e2ca47c973d545bee11ddd8,1291813400028.85a8d31d099f409dd36f3bcc4be1eba6.
> 2010-12-09 04:29:26,330 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region article,ed0cdc7ae6db73432e2ca47c973d545bee11ddd8,1291813400028.85a8d31d099f409dd36f3bcc4be1eba6. in 0sec
> 2010-12-09 04:32:33,081 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGIONSERVER_STOP
> 2010-12-09 04:32:33,081 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60020
> 2010-12-09 04:32:33,089 WARN org.apache.hadoop.hdfs.DFSClient: DFSOutputStream ResponseProcessor exception for block blk_2012253533858803341_250248java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at java.io.DataInputStream.readLong(DataInputStream.java:399)
> at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:120)
> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2621)
>
> 2010-12-09 04:32:33,089 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x12ca58c08ff0049, likely server has closed socket, closing sock
> et connection and attempting reconnect
> 2010-12-09 04:32:33,089 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler 5 on 60020 took 75446ms appending an edit to hlog; editcount=50505
> 2010-12-09 04:32:33,090 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 18 on 60020: exiting
> 2010-12-09 04:32:33,139 INFO org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_2012253533858803341_250248 waiting for responder to exit.
> 2010-12-09 04:32:33,139 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 8 on 60020: exiting
> 2010-12-09 04:32:33,139 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 1 on 60020: exiting
> 2010-12-09 04:32:33,139 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 10 on 60020: exiting
> 2010-12-09 04:32:33,140 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 0 on 60020: exiting2010-12-09 04:32:33,140 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 5 on 60020: exiting
> 2010-12-09 04:32:33,139 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 8 on 60020: exiting2010-12-09 04:32:33,140 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 4 on 60020: exiting
> 2010-12-09 04:32:33,140 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 3 on 60020: exiting2010-12-09 04:32:33,142 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 7 on 60020: exiting
> 2010-12-09 04:32:33,142 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 9 on 60020: exiting2010-12-09 04:32:33,143 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
> 2010-12-09 04:32:33,143 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 2 on 60020: exiting
> 2010-12-09 04:32:33,143 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server Responder
> 2010-12-09 04:32:33,143 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 6 on 60020: exiting
> 2010-12-09 04:32:33,144 INFO org.mortbay.log: Stopped SelectChannelConnector@0.0.0.0:60030
>
>
>
>
> NAMENODE ->
> 2010-12-09 04:32:08,408 WARN org.apache.hadoop.hdfs.StateChange: DIR* NameSystem.startFile: failed to create file /hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555 for DFSClient_hb_m_10.194.194.79:60000_1291788452343 on client 10.194.194.79, because this file is already being created by NN_Recovery on 10.100.159.13
> 2010-12-09 04:32:08,408 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 54310, call append(/hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555, DFSClient_hb_m_10.194.194.79:60000_1291788452343) from 10.194.194.79:59118: error: org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to create file /hba
> se/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555 for DFSClient_hb_m_10.194.194.79:60000_1291788452343 on client 10.194.194.79, because this file is already being created by NN_Recovery on 10.100.159.13
> org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to create file /hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555 for DFSClient_hb_m_10.194.194.79:60000_1291788452343 on client 10.194.194.79, because this file is already being created by NN_Recovery on 10.100.159.13
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1194) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:1282)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.append(NameNode.java:541) at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:528) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1319)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1315) at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1313)2010-12-09 04:32:09,411 WARN org.apache.hadoop.hdfs.StateChange: DIR* NameSystem.startFile: failed to create file /hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A6
> 0020.1291868481555 for DFSClient_hb_m_10.194.194.79:60000_1291788452343 on client 10.194.194.79, because this file is already being created by NN_Recovery on 10.100.159.132010-12-09 04:32:09,411 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 54310, call append(/hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868
> 481555, DFSClient_hb_m_10.194.194.79:60000_1291788452343) from 10.194.194.79:59118: error: org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to create file /hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555 for DFSClient_hb_m_10.194.194.79:60000_1291788452343 on client 10.194.194.79, because this file is
> already being created by NN_Recovery on 10.100.159.13org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to create file /hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555 for DFSClie
> nt_hb_m_10.194.194.79:60000_1291788452343 on client 10.194.194.79, because this file is already being created by NN_Recovery on 10.100.159.13 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1194)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:1282) at org.apache.hadoop.hdfs.server.namenode.NameNode.append(NameNode.java:541)
> at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:528)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1319) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1315)
>
>
>
> DATANODE
>
> 2010-12-09 04:32:09,534 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder blk_2012253533858803341_250248 1 : Thread is interrupted.
> 2010-12-09 04:32:09,534 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(10.100.159.13:50010, storageID=DS-145997007-10.100.159.13-50010-1291262743182, infoPort=50075, ipcPort=50020):DataXceiver
> java.io.IOException: Interrupted receiveBlock
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:579)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:417)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:122)
> 2010-12-09 04:32:09,534 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 1 for block blk_2012253533858803341_250248 terminating
>
>
> ************************************************************************************
> 06.hadoop
> ************************************************************************************
> REGION SERVER ->
>
> 2010-12-09 06:17:47,215 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
> 2010-12-09 06:17:47,430 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
> 2010-12-09 06:17:47,549 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
> 2010-12-09 06:17:47,557 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
> 2010-12-09 06:17:47,563 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
> 2010-12-09 06:17:48,619 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
> 2010-12-09 06:17:48,619 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
> 2010-12-09 06:17:48,619 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
> 2010-12-09 06:17:48,619 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
> 2010-12-09 06:17:48,619 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
> 2010-12-09 06:17:49,703 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
> 2010-12-09 06:17:49,703 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
> 2010-12-09 06:17:49,703 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
> 2010-12-09 06:17:49,705 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
> 2010-12-09 06:17:49,729 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: NotServingRegionException; article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f.
>
> A bunch of these ^
>
> Then a little later::
> ....
>
> 2010-12-09 06:23:23,561 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction requested for region article,993143fca3613014990cc6975cacdd62f4147e03,12918133247
> 53.19413f33fc660536d9d4f315a1283440. because: regionserver60020.cacheFlusher
> 2010-12-09 06:23:27,058 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs -- HDFS-200
> 2010-12-09 06:23:27,080 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Roll /hbase/.logs/06.hadoop.****,60020,1291864620851/10.99.97.106%3A60020.1291875047947, entries=53315,
> filesize=63781925. New hlog /hbase/.logs/06.hadoop.****,60020,1291864620851/10.99.97.106%3A60020.1291875807006
> 2010-12-09 06:23:27,080 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: Found 1 hlogs to remove out of total 8; oldest outstanding sequenceid is 33526881 from region article
> ,1221b0013c61fd7446cb822f698d0bcafa487342,1291872610788.bb2aa6ae8c694a096a00e7bb91fc46c3.
> 2010-12-09 06:23:27,080 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: moving old hlog file /hbase/.logs/06.hadoop.****,60020,1291864620851/10.99.97.106%3A60020.1291872064424
> whose highest sequenceid is 33506639 to /hbase/.oldlogs/10.99.97.106%3A60020.1291872064424
> 2010-12-09 06:25:39,162 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler 12 on 60020 took 119034ms appending an edit to hlog; editcount=3683
> 2010-12-09 06:25:39,162 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception: java.io.IOException: Broken pipe
> at sun.nio.ch.FileDispatcher.write0(Native Method)
> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100)
> at sun.nio.ch.IOUtil.write(IOUtil.java:71)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
> at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
> at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
> at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
> at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2509)
> 2010-12-09 06:25:39,163 WARN org.apache.hadoop.hdfs.DFSClient: DFSOutputStream ResponseProcessor exception for block blk_1141147913164105086_258331java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at java.io.DataInputStream.readLong(DataInputStream.java:399)
> at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:120)
> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2621)
>
> 2010-12-09 06:25:39,163 INFO org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_1141147913164105086_258331 waiting for responder to exit.
> 2010-12-09 06:25:39,164 WARN org.apache.hadoop.hdfs.DFSClient: DFSOutputStream ResponseProcessor exception for block blk_-7076574401277381592_258324java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at java.io.DataInputStream.readLong(DataInputStream.java:399)
> at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:120)
> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2621)
>
> 2010-12-09 06:25:39,164 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_-7076574401277381592_258324 bad datanode[0] 10.99.97.106:50010
> 2010-12-09 06:25:39,164 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x12ca58c08ff004b, likely server has closed socket, closing socket connection and attempting reconnect
> 2010-12-09 06:25:39,164 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_-7076574401277381592_258324 in pipeline 10.99.97.106:50010, 10.100.154.103:50010: bad datanode 10.99.97.106:50010
> 2010-12-09 06:25:39,165 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_1141147913164105086_258331 bad datanode[0] 10.99.97.106:50010
> 2010-12-09 06:25:39,167 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x12ca58c08ff004c, likely server has closed socket, closing socket connection and attempting reconnect
> 2010-12-09 06:25:39,169 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /10.100.154.103:50020. Already tried 0 time(s).
> 2010-12-09 06:25:39,169 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_1141147913164105086_258331 in pipeline 10.99.97.106:50010, 10.100.159.13:50010: bad datanode 10.99.97.106:50010
> 2010-12-09 06:25:39,209 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGIONSERVER_STOP
> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60020
> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 3 on 60020: exiting
> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 9 on 60020: exiting
> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 24 on 60020: exiting
> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 1 on 60020: exiting
> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 15 on 60020: exiting
> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 0 on 60020: exiting
> 2010-12-09 06:25:39,214 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 3 on 60020: exiting
> 2010-12-09 06:25:39,214 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server listener on 60020
> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 2 on 60020: exiting
> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 7 on 60020: exiting
> 2010-12-09 06:25:39,213 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 21 on 60020: exiting
> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 4 on 60020: exiting
> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server Responder
> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 5 on 60020: exiting
> 2010-12-09 06:25:39,216 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 8 on 60020: exiting
> 2010-12-09 06:25:39,216 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 9 on 60020: exiting
> 2010-12-09 06:25:39,216 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 6 on 60020: exiting
> 2010-12-09 06:25:39,216 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
> 2010-12-09 06:25:39,223 WARN org.apache.hadoop.hdfs.DFSClient: Failed recovery attempt #0 from primary datanode 10.100.159.13:50010
>
>
>
>