You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Zheng Lv <lv...@gmail.com> on 2009/09/27 11:45:27 UTC
is there any problem with our environment?
Hello Everyone,
About one or two months ago, we made a test program which inserts one
web page per 10ms into hbase, but after running it we got many exceptions
and many regionservers shutdown very soon. Many people suggested me to make
zks exclusive. And Now we have built a cluster including one master, three
regionservers, and three zookeepers, exclusively, but after we ran the same
test program, regionservers shutdown too, even sooner.
Now I'm confused. Is there any problem with the environment? Or some
other errors? Someone give some suggestions?
BTW, I noticed that when hbase starting up, there are some exceptions in
logging files, both master, regionservers, zookeepers.
I will reply this mail to post up all the logs.
Thank you,
LvZheng
Re: is there any problem with our environment?
Posted by Zheng Lv <lv...@gmail.com>.
These are the second shut down server's log:
2009-10-09 15:23:12,158 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/hbase/.logs/ubuntu3,60020,1255059681726/hlog.dat.1255072983694 whose
highest sequence/edit id is 8079718
2009-10-09 15:24:28,717 WARN org.apache.hadoop.hdfs.DFSClient:
DFSOutputStream ResponseProcessor exception for block
blk_2196913463479121687_23657java.net.SocketTimeoutException: 69000 millis
timeout while waiting for channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/192.168.33.8:33876 remote=/
192.168.33.8:50010]
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.DataInputStream.readFully(DataInputStream.java:178)
at java.io.DataInputStream.readLong(DataInputStream.java:399)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2369)
2009-10-09 15:24:34,936 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_2196913463479121687_23657 bad datanode[0]
192.168.33.8:50010
2009-10-09 15:24:34,936 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_2196913463479121687_23657 in pipeline
192.168.33.8:50010, 192.168.33.5:50010, 192.168.33.6:50010: bad datanode
192.168.33.8:50010
2009-10-09 15:24:34,960 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC
Server handler 5 on 60020 took 75039ms appending an edit to hlog;
editcount=6325
2009-10-09 15:24:34,962 WARN org.apache.hadoop.hbase.regionserver.HLog:
regionserver/192.168.33.8:60020.logFlusher took 69944ms optional sync'ing
hlog; editcount=6331
2009-10-09 15:24:35,076 INFO org.apache.hadoop.hbase.regionserver.HLog: Roll
/hbase/.logs/ubuntu3,60020,1255059681726/hlog.dat.1255072992146,
entries=6415, calcsize=63770126, filesize=63085397. New hlog
/hbase/.logs/ubuntu3,60020,1255059681726/hlog.dat.1255073075063
2009-10-09 15:24:38,543 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072536796_732973,1255072671531
in 1mins, 44sec
2009-10-09 15:24:38,544 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting split of region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072536796_732973,1255072671531
2009-10-09 15:24:39,821 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072536796_732973,1255072671531
2009-10-09 15:24:41,531 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072536796_732973,1255073078552/447724835
available; sequence id is 8090065
2009-10-09 15:24:41,531 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072536796_732973,1255073078552
2009-10-09 15:24:41,690 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072554416_736287,1255073078552/1620807431
available; sequence id is 8090066
2009-10-09 15:24:41,691 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072554416_736287,1255073078552
2009-10-09 15:24:41,714 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META
updated, and report to master all successful. Old region=REGION => {NAME =>
'webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072536796_732973,1255072671531',
STARTKEY => 'http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072536796_732973',
ENDKEY => '', ENCODED => 99294965, OFFLINE => true, SPLIT => true, TABLE =>
{{NAME => 'webpage', FAMILIES => [{NAME => 'CF_CONTENT', COMPRESSION =>
'NONE', VERSIONS => '2', TTL => '2147483647', BLOCKSIZE => '65536',
IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'CF_INFORMATION',
COMPRESSION => 'NONE', VERSIONS => '1', TTL => '2147483647', BLOCKSIZE =>
'65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, new regions:
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072536796_732973,1255073078552,
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072554416_736287,1255073078552.
Split took 3sec
2009-10-09 15:24:41,714 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072362203_728621,1255072671531
2009-10-09 15:24:43,119 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072536796_732973,1255073078552
2009-10-09 15:24:43,119 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072554416_736287,1255073078552
2009-10-09 15:24:43,120 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072536796_732973,1255073078552
2009-10-09 15:24:43,245 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072536796_732973,1255073078552/447724835
available; sequence id is 8090065
2009-10-09 15:24:43,245 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072554416_736287,1255073078552
2009-10-09 15:24:43,397 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072554416_736287,1255073078552/1620807431
available; sequence id is 8090066
2009-10-09 15:24:51,275 INFO org.apache.hadoop.hbase.regionserver.HLog: Roll
/hbase/.logs/ubuntu3,60020,1255059681726/hlog.dat.1255073075063,
entries=6410, calcsize=63768309, filesize=63084111. New hlog
/hbase/.logs/ubuntu3,60020,1255059681726/hlog.dat.1255073091269
2009-10-09 15:24:51,276 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/hbase/.logs/ubuntu3,60020,1255059681726/hlog.dat.1255072992146 whose
highest sequence/edit id is 8086133
2009-10-09 15:24:56,814 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC
Server handler 9 on 60020 took 1663ms appending an edit to hlog;
editcount=5165
2009-10-09 15:24:57,227 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072362203_728621,1255072671531
in 15sec
2009-10-09 15:24:57,228 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072536796_732973,1255073078552
2009-10-09 15:24:58,968 INFO org.apache.hadoop.hbase.regionserver.HLog: Roll
/hbase/.logs/ubuntu3,60020,1255059681726/hlog.dat.1255073091269,
entries=6420, calcsize=63771943, filesize=63086683. New hlog
/hbase/.logs/ubuntu3,60020,1255059681726/hlog.dat.1255073098961
2009-10-09 15:24:58,968 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/hbase/.logs/ubuntu3,60020,1255059681726/hlog.dat.1255073075063 whose
highest sequence/edit id is 8092549
2009-10-09 15:24:58,974 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC
Server handler 3 on 60020 took 1149ms appending an edit to hlog; editcount=0
2009-10-09 15:25:03,892 INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Too many store files
for region webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072554416_736287,1255073078552:
5, requesting compaction and waiting
2009-10-09 15:25:06,189 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC
Server handler 8 on 60020 took 1077ms appending an edit to hlog;
editcount=6342
2009-10-09 15:25:06,265 INFO org.apache.hadoop.hbase.regionserver.HLog: Roll
/hbase/.logs/ubuntu3,60020,1255059681726/hlog.dat.1255073098961,
entries=6414, calcsize=63770126, filesize=63085212. New hlog
/hbase/.logs/ubuntu3,60020,1255059681726/hlog.dat.1255073106258
2009-10-09 15:25:08,937 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072536796_732973,1255073078552
in 11sec
2009-10-09 15:25:08,937 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072554416_736287,1255073078552
2009-10-09 15:25:09,897 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Blocking updates for 'IPC Server handler 5 on 60020' on region
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072554416_736287,1255073078552:
memstore size 128.0m is >= than blocking 128.0m size
2009-10-09 15:25:25,819 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072554416_736287,1255073078552
in 16sec
2009-10-09 15:25:25,819 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting split of region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072554416_736287,1255073078552
2009-10-09 15:25:25,900 INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Compaction has
completed, we waited 22500ms, finishing flush of region webpage,http:\x2F\
x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072554416_736287,1255073078552
2009-10-09 15:25:29,550 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Unblocking updates for region
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072554416_736287,1255073078552
'IPC Server handler 5 on 60020'
2009-10-09 15:25:29,550 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072554416_736287,1255073078552
2009-10-09 15:25:30,065 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072554416_736287,1255073125827/2031468408
available; sequence id is 8110565
2009-10-09 15:25:30,065 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072554416_736287,1255073125827
2009-10-09 15:25:30,708 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072716434_740161,1255073125827/562635413
available; sequence id is 8110566
2009-10-09 15:25:30,708 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072716434_740161,1255073125827
2009-10-09 15:25:30,731 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META
updated, and report to master all successful. Old region=REGION => {NAME =>
'webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072554416_736287,1255073078552',
STARTKEY => 'http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072554416_736287',
ENDKEY => '', ENCODED => 1620807431, OFFLINE => true, SPLIT => true, TABLE
=> {{NAME => 'webpage', FAMILIES => [{NAME => 'CF_CONTENT', COMPRESSION =>
'NONE', VERSIONS => '2', TTL => '2147483647', BLOCKSIZE => '65536',
IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'CF_INFORMATION',
COMPRESSION => 'NONE', VERSIONS => '1', TTL => '2147483647', BLOCKSIZE =>
'65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, new regions:
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072554416_736287,1255073125827,
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255072716434_740161,1255073125827.
Split took 4sec
2009-10-09 15:28:05,326 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x24375d149b0001 to sun.nio.ch.SelectionKeyImpl@55229f61
java.io.IOException: TIMED OUT
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-10-09 15:28:05,406 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x24375d149b0003 to sun.nio.ch.SelectionKeyImpl@540dc4f1
java.io.IOException: TIMED OUT
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-10-09 15:28:05,426 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: Disconnected, type: None, path: null
2009-10-09 15:28:05,484 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu2/192.168.33.9:2222
2009-10-09 15:28:06,195 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu2/192.168.33.9:2222
2009-10-09 15:28:08,486 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x24375d149b0001 to sun.nio.ch.SelectionKeyImpl@3b4378b0
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
2009-10-09 15:28:08,486 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown input
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-10-09 15:28:08,487 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-10-09 15:28:08,487 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x24375d149b0003 to sun.nio.ch.SelectionKeyImpl@4aa88935
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
2009-10-09 15:28:08,487 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown input
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-10-09 15:28:08,488 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-10-09 15:28:08,718 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu12/192.168.33.12:2222
2009-10-09 15:28:09,274 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu10/192.168.33.10:2222
2009-10-09 15:28:20,128 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: ubuntu6/192.168.33.7:9000. Already tried 0 time(s).
2009-10-09 15:28:22,058 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x24375d149b0003 to sun.nio.ch.SelectionKeyImpl@7ec81cdd
java.io.IOException: TIMED OUT
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-10-09 15:28:22,059 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.net.SocketException: Transport endpoint is not connected
at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-10-09 15:28:22,606 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x24375d149b0001 to sun.nio.ch.SelectionKeyImpl@3df3232
java.io.IOException: TIMED OUT
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-10-09 15:28:22,606 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.net.SocketException: Transport endpoint is not connected
at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-10-09 15:28:23,432 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu10/192.168.33.10:2222
2009-10-09 15:28:24,563 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu12/192.168.33.12:2222
2009-10-09 15:28:34,827 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.33.8:32946 remote=ubuntu10/192.168.33.10:2222]
2009-10-09 15:28:34,827 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-10-09 15:28:34,833 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x24375d149b0003 to sun.nio.ch.SelectionKeyImpl@4ccbf648
java.io.IOException: Session Expired
at
org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:548)
at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:661)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
2009-10-09 15:28:34,833 INFO org.apache.zookeeper.ZooKeeper: Closing
session: 0x24375d149b0003
2009-10-09 15:28:34,834 INFO org.apache.zookeeper.ClientCnxn: Closing
ClientCnxn for session: 0x24375d149b0003
2009-10-09 15:28:34,834 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.33.8:33682 remote=ubuntu12/192.168.33.12:2222]
2009-10-09 15:28:34,834 INFO org.apache.zookeeper.ClientCnxn: Disconnecting
ClientCnxn for session: 0x24375d149b0003
2009-10-09 15:28:34,834 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-10-09 15:28:34,835 INFO org.apache.zookeeper.ZooKeeper: Session:
0x24375d149b0003 closed
2009-10-09 15:28:34,835 INFO org.apache.zookeeper.ClientCnxn: EventThread
shut down
2009-10-09 15:28:34,836 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: Expired, type: None, path: null
2009-10-09 15:28:34,836 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x24375d149b0001 to sun.nio.ch.SelectionKeyImpl@6296185b
java.io.IOException: Session Expired
at
org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:548)
at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:661)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
2009-10-09 15:28:34,836 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session
expired
2009-10-09 15:28:34,837 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
java.io.IOException: Server not running, aborting
at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1914)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1899)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-09 15:28:34,844 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
(Retry: 0)
org.apache.hadoop.hbase.Leases$LeaseStillHeldException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
at java.lang.Thread.run(Thread.java:619)
2009-10-09 15:28:34,895 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
request=0.0, regions=283, stores=564, storefiles=564, storefileIndexSize=49,
memstoreSize=0, usedHeap=306, maxHeap=2993, blockCacheSize=86974704,
blockCacheFree=540873296, blockCacheCount=693, blockCacheHitRatio=0
2009-10-09 15:28:34,897 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 7 on 60020, call next(-3749056290061789970) from 192.168.33.7:47319:
error: java.io.IOException: Server not running, aborting
java.io.IOException: Server not running, aborting
at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1914)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1899)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-09 15:28:34,901 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
(Retry: 1)
org.apache.hadoop.hbase.Leases$LeaseStillHeldException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
at java.lang.Thread.run(Thread.java:619)
2009-10-09 15:28:34,901 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Stop was requested,
clearing the toDo despite of the exception
2009-10-09 15:28:34,902 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
server on 60020
2009-10-09 15:28:34,902 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
java.io.IOException: Server not running, aborting
at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.close(HRegionServer.java:1946)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-09 15:28:34,902 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 7 on 60020: exiting
2009-10-09 15:28:34,903 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 2 on 60020: exiting
2009-10-09 15:28:34,903 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 8 on 60020: exiting
2009-10-09 15:28:34,903 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC
Server listener on 60020
2009-10-09 15:28:34,904 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 1 on 60020: exiting
2009-10-09 15:28:34,904 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 5 on 60020: exiting
2009-10-09 15:28:34,903 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 9 on 60020: exiting
2009-10-09 15:28:34,905 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 3 on 60020: exiting
2009-10-09 15:28:34,905 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 4 on 60020: exiting
2009-10-09 15:28:34,904 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 0 on 60020: exiting
2009-10-09 15:28:34,920 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
2009-10-09 15:28:34,920 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC
Server Responder
2009-10-09 15:28:34,967 INFO
org.apache.hadoop.hbase.regionserver.LogFlusher:
regionserver/192.168.33.8:60020.logFlusher exiting
2009-10-09 15:28:35,106 INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher:
regionserver/192.168.33.8:60020.cacheFlusher exiting
2009-10-09 15:28:35,110 INFO org.apache.hadoop.hbase.regionserver.LogRoller:
LogRoller exiting.
2009-10-09 15:28:35,110 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread:
regionserver/192.168.33.8:60020.compactor exiting
2009-10-09 15:28:35,111 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer$MajorCompactionChecker:
regionserver/192.168.33.8:60020.majorCompactionChecker exiting
2009-10-09 15:28:43,136 INFO org.apache.hadoop.hbase.Leases:
regionserver/192.168.33.8:60020.leaseChecker closing leases
2009-10-09 15:28:43,136 INFO org.apache.hadoop.hbase.Leases:
regionserver/192.168.33.8:60020.leaseChecker closed leases
2009-10-09 15:28:43,402 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: worker thread exiting
2009-10-09 15:29:34,897 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: ubuntu6/192.168.33.7:9000. Already tried 0 time(s).
2009-10-09 15:29:34,904 WARN org.apache.hadoop.hdfs.DFSClient: Problem
renewing lease for DFSClient_1152927602
java.io.IOException: Call to ubuntu6/192.168.33.7:9000 failed on local
exception: java.io.IOException
at org.apache.hadoop.ipc.Client.wrapException(Client.java:774)
at org.apache.hadoop.ipc.Client.call(Client.java:742)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy1.renewLease(Unknown Source)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy1.renewLease(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.renew(DFSClient.java:1016)
at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1028)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException
at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:416)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:445)
Caused by: java.lang.InterruptedException
... 2 more
... ...
2009-10-09 15:29:35,245 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: aborting server at:
192.168.33.8:60020
2009-10-09 15:29:35,246 INFO org.apache.zookeeper.ZooKeeper: Closing
session: 0x24375d149b0001
2009-10-09 15:29:35,246 INFO org.apache.zookeeper.ClientCnxn: Closing
ClientCnxn for session: 0x24375d149b0001
2009-10-09 15:29:35,246 INFO org.apache.zookeeper.ClientCnxn: Disconnecting
ClientCnxn for session: 0x24375d149b0001
2009-10-09 15:29:35,246 INFO org.apache.zookeeper.ClientCnxn: EventThread
shut down
2009-10-09 15:29:35,246 INFO org.apache.zookeeper.ZooKeeper: Session:
0x24375d149b0001 closed
2009-10-09 15:29:35,247 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver/
192.168.33.8:60020 exiting
2009-10-09 15:29:35,252 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Starting shutdown
thread.
2009-10-09 15:29:35,252 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Shutdown thread complete
Re: is there any problem with our environment?
Posted by Zheng Lv <lv...@gmail.com>.
Hello Stack,
Can you see my mail?It seems you don't see it.
2009/11/21 stack <st...@duboce.net>
> On Fri, Nov 20, 2009 at 12:28 AM, Zheng Lv <lvzheng19800619@gmail.com
> >wrote:
>
> > Hello Stack,
> > Remember the "no route to host" exceptions last time? Now there isn't any
> > more, and the test program can be running for several days.
>
>
> How did you fix it?
>
>
>
> > Thank you.
> > Recently we started running our crawling program, which crawls webpages
> and
> > then insert them to hbase.
> > But we got so many "org.apache.hadoop.hbase.NotServingRegionException"
> like
> > that:
> >
> > 2009-11-20 12:36:41,898 ERROR
> > org.apache.hadoop.hbase.regionserver.HRegionServer:
> > org.apache.hadoop.hbase.NotServingRegionException: webpage,
> >
> >
> http://bbs.city.tianya.cn/tianyacity/Content/178/1/536629.shtml,1258691377544
> >
>
> So figure out whats happening to that region by grepping its name in the
> master log. Why is it offline so long? Are machines loaded? Swapping?
>
> Are the crawlers running on same machines as hbase?
>
> What crawler are you using?
>
> Andrew Purtell has written up some notes on getting a nice balance between
> crawl process and hbase such that all runs smoothly in private
> correspondence. Let me ask him if its ok to forward the list.
>
>
> ....
>
> 2009-11-20 12:36:25,259 INFO org.apache.hadoop.hbase.master.ServerManager:
> > Processing MSG_REPORT_SPLIT:
> > webpage,http:\x2F\x2Fbbs.city.tianya.cn <http://x2fbbs.city.tianya.cn/>
> > \x2Ftianyacity\x2FContent\x2F178\x2F1\x2F536629.shtml,1258691377544:
> > Daughters; webpage,http:\x2F\x2Fbbs.city.tianya.cn<http://x2fbbs.city.tianya.cn/>
> > \x2Ftianyacity\x2FContent\x2F178\x2F1\x2F536629.shtml,1258691779496,
> > webpage,http:\x2F\x2Fbbs.city.tianya.cn <http://x2fbbs.city.tianya.cn/>
> > \x2Ftianyacity\x2FContent\x2F329\x2F1\x2F164370.shtml,1258691779496
> > from ubuntu12,60020,1258687326554;
> >
> > Yeah, its split. Thats normal. Whats not normal is the client not
> finding
> the daughter split in its new location. Did the daughters get deployed
> promptly?
>
>
>
> > And a few hours later, some rs shutdown.
> >
> > I read the mail
> >
> >
> http://mail-archives.apache.org/mod_mbox/hadoop-hbase-user/200907.mbox/%3C9b27a8a60907272122y1bfa6254n95948942d5ca7f88@mail.gmail.com%3E
> > ,
> > which was sent by my partner Angus. In the mail you told us it was a case
> > of
> > "HBASE-1671", Fix Version of which is 0.20.0, but the hbase version we
> are
> > using is just 0.20.0.
> >
>
> Can you update to hbase 0.20.2? It has a bunch of fixes that could be
> related to the above.
> Yours,
> St.Ack
>
>
>
> > Any idea?
> > Best Regards,
> > LvZheng
> >
> >
> >
> >
> >
> > 2009/10/13 stack <st...@duboce.net>
> >
> > > Thanks for posting. Its much easier reading the logs from there.
> > >
> > > Looking in nohup.out I see it can't find region 'webpage,http:\x2F\
> > > x2Fnews.163.com <http://x2fnews.163.com/> <http://x2fnews.163.com/>
> > >
> >
> \x2F09\x2F080\x2F0\x2F5FOO155J0001124J.html1255072992000_751685,1255316061169'.
> > > It never finds it. It looks like it was assigned successfully to
> > > 192.168.33.5 going by the master log. Once you've figured out the
> > > hardware/networking issues, lets work at getting that region back on
> > line.
> > >
> > > The master timed out its session against zk because of 'no route to
> > host'.
> > >
> > > St.Ack
> > >
> > > On Mon, Oct 12, 2009 at 12:23 AM, Zheng Lv <lvzheng19800619@gmail.com
> > > >wrote:
> > >
> > > > Hello Stack,
> > > > I have enabled DEBUG and restarted the test program. This time the
> > > > master shut down, and I have put the logs on skydrive.
> > > >
> > > >
> > >
> >
> http://cid-a331bb289a14fbef.skydrive.live.com/browse.aspx/.Public?uc=2&isFromRichUpload=1
> > > > .
> > > > "nohup.out" is our test program log,
> "hbase-cyd-master-ubuntu6.log"
> > is
> > > > master log.
> > > >
> > > > On the other hand, today we found that when we run "dmesg", there
> > were
> > > > many logs like "[3641697.122769] r8169: eth0: link down". And I think
> > > this
> > > > might be the reason of so many "no route to host" and "Time Out". Now
> > our
> > > > system manager is checking, if we have a result we will let you
> know.:)
> > > > Thanks,
> > > > LvZheng.
> > > >
> > > > 2009/10/11 stack <st...@duboce.net>
> > > >
> > > > > On Fri, Oct 9, 2009 at 3:18 AM, Zheng Lv <
> lvzheng19800619@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > ...
> > > > > > so,
> > > > > > > please remove the delay so hbase fails faster so it doesn't
> > take
> > > > so
> > > > > > long to
> > > > > > > figure the issue.
> > > > > > > Are you inserting every 10ms because hbase is falling over
> on
> > > you?
> > > > > If
> > > > > > Yes I inserted every 10ms because I'm afraid hbase would fall
> > > over.
> > > > > Now
> > > > > > I have removed the delay.
> > > > > >
> > > > > > After doing these, We have run the test program again, and one
> > > > region
> > > > > > server shut down after about 2 hours, another one 3.
> > > > > > I will post the logs on these two servers in following reply
> > > mails.
> > > > > >
> > > > > >
> > > > > Thanks for doing the above.
> > > > >
> > > > > For the future, debugging, please enable DEBUG and put your logs
> > > > somewhere
> > > > > where I can pull them or put them up in pastebin. Logs in email
> > > messages
> > > > > are hard to follow. Thanks.
> > > > >
> > > > >
> > > > > > > Ok. So this is hbase 0.20.0? Tell us about your hardware.
> > > What
> > > > > kind
> > > > > > is
> > > > > > > it? CPU/RAM/Disks.
> > > > > > Yes we are using hbase 0.20.0. And the following is our
> > > hardware:
> > > > > >
> > > > > > CPU:amd x3 710
> > > > > > RAM:8g ddr2 800
> > > > > > Disk:270g(raid0)
> > > > > >
> > > > > >
> > > > > Thats an interesting chip -- 3 cores! The above should be fine as
> > long
> > > > as
> > > > > you coral your mapreduce jobs running on same cluster.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > > We have 7 servers with above hardware, one for master, three
> for
> > > > > > namenodes / regionservers, and the other 3 for zks.
> > > > > > By the way, what kind of hardware and environment do you
> suggest
> > > we
> > > > > > have?
> > > > > >
> > > > >
> > > > >
> > > > > This configuration seems fine to start with. Later we might
> > experiment
> > > > > running zk on same machines as regionservers and then up number of
> > > > > regionservers to 6 and up the quorum members to 5.
> > > > >
> > > > > St.Ack
> > > > >
> > > > >
> > > > > >
> > > > > > Thank you, very much.
> > > > > > LvZheng.
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: is there any problem with our environment?
Posted by Andrew Purtell <ap...@apache.org>.
hbase.hregion.memstore.block.multiplier=4 increases the amount of data that will be buffered after a flush is started before a writer will be blocked (up to 4 times the flush limit).
hbase.hstore.blockingStoreFiles=20 increases the number of flush files which may exist on disk before writers are blocked for a compaction.
Both trade off against heap space and compaction/scan/query time to service a high write load environment.
- Andy
________________________________
From: Zheng Lv <lv...@gmail.com>
To: hbase-user@hadoop.apache.org
Sent: Thu, November 26, 2009 1:19:24 AM
Subject: Re: is there any problem with our environment?
Hello Andy,
Thank you very much for your suggestions, that will be helpful.
I have one question, according to what you adjust the first 2
properties(hbase.hregion.memstore.block.multiplier,
hbase.hstore.blockingStoreFiles)?
Regards,
LvZheng.
2009/11/26 Andrew Purtell <ap...@apache.org>
> Try this: Do not cache on the crawlers, just write through.
>
> Run each region server with plenty of heap (4 GB to start). So it seems you
> need more RAM on your systems, or you should move your crawlers off to
> separate servers to free up RAM and CPU.
>
> Adjust your HBase site config as follows:
>
> <property>
> <name>hbase.hregion.memstore.block.multiplier</name>
> <value>4</value>
> </property>
>
> <property>
> <name>hbase.hstore.blockingStoreFiles</name>
> <value>20</value>
> </property>
>
> <property>
> <name>dfs.replication</name>
> <value>2</value>
> </property>
>
> <property>
> <name>hfile.block.cache.size</name>
> <!-- disable -->
> <value>0</value>
> </property>
>
> Be advised that the probability of OOM under high write load is high if you
> do not provide adequate heap. This may manifest either as OOM exceptions or
> as GC taking so much time as to drop the ZK session. Concurrent garbage
> collector will keep GC latency low but it needs a larger available heap
> space in return. You should also tune your HBASE_OPTS in conf/hbase-env.sh
> according to this advice:
> http://osdir.com/ml/hbase-user-hadoop-apache/2009-10/msg00276.html
>
> With your monitoring tools watch the amount of disk wait time on the OS
> level. If disk wait times exceed 40%, you need more spindles (disk) for your
> datanodes or more datanodes (servers) to spread the I/O load.
>
> - Andy
>
>
>
>
> ________________________________
> From: Zheng Lv <lv...@gmail.com>
> To: hbase-user@hadoop.apache.org
> Sent: Tue, November 24, 2009 8:22:04 PM
> Subject: Re: is there any problem with our environment?
>
> Hello Stack,
> >Sorry for taking time getting back to you Lv.
> Never mind:).
>
> >I posted Andrew's crawling notes yesterday. Did you see them? I thought
> >they might be of use to you.
> You mean you'v sent mail to me? I didn't get it.
>
> >This is interesting. You were just looking at message logs and saw the
> >above?
> We checked /var/log/messages on some servers, and we got many "link down"
> and "link up".
>
> >You don't need 6 zk nodes. Make it 3 (In an earlier mail you said you had
> 3
> >only but above would seem to say you have 6 zk nodes). You could up the
> >ticktime from 2 to 3 seconds (We learned that maximum session time is
> >20*ticktime no matter what session timeout is set to).
> >
> We will change our cluster as you told me.
> >(In an earlier mail you said you had 3 only but above would seem to say
> you
> have 6 zk nodes)
> We added some, we thought 3 is insufficient.
>
> >For 8Gs of RAM and a single disk might not be sufficient to carry all of
> the
> >above daemons on a single node. You have cluster-wide monitoring setup?
> >Are these nodes swapping? From the logs I've seen, HDFS is for sure
> >struggling.
> Our system manager has just deployed mrtg+snmp on those servers, and we
> will
> share the data with you.
>
> >The crawler writes straight into hbase or does it write the
> >local disk?
> Each crawler has a cache in memory, when it is full, the crawler will flush
> webpages in cache to hbase.
> >What is the tasktracker doing? Are mapreduce jobs running
> >concurrently on these nodes?
> Now the tasktracker process is running, but there is not any job on it.
> But,
> when updater, which is another part of our product, is on there will be.
> We will restart hbase0.20.2 with DEBUG enabled and monitor it, I'll be back
> when we get useful data.
> Regards,
> Lv Zheng.
>
>
>
>
>
>
>
> 2009/11/25 stack <st...@duboce.net>
>
> > I took a look at your logs. Please run with DEBUG enabled going forward
> > (if
> > you upgraded to 0.20.2, then it'll be on by default).
> >
> > HDFS is struggling.
> >
> > Looking at this region:
> >
> >
> >
> http://blog.blog.tianya.cn/blogger/post_Date.asp?blogID=1818563&CategoryID=1256944&idWriter=0&Key=0&NextPostID=888888888&PageNo=2
> >
> > ... there seems to be an issue w/ accounting. It looks like it got
> > assigned
> > to 13 before the logs you gave me but then the regionserver its on, while
> > it
> > thinks its carrying it in one regard -- its trying to flush it so it can
> > let
> > go of old WAL files -- it then is telling clients that ask for it that
> its
> > not serving it. Seeing an earlier master and regionserver 13 log would
> > help
> > figure what happened. Try restart on 0.20.2. Tell me more about the
> > loading on these machines, how its done. Do you have monitoring software
> > running?
> >
> > Thanks,
> > St.Ack
> >
> >
> >
> >
> >
> > On Tue, Nov 24, 2009 at 10:30 AM, stack <st...@duboce.net> wrote:
> >
> > > Sorry for taking time getting back to you Lv.
> > >
> > > I posted Andrew's crawling notes yesterday. Did you see them? I
> thought
> > > they might be of use to you.
> > >
> > >
> > > On Sun, Nov 22, 2009 at 10:30 PM, Zheng Lv <lvzheng19800619@gmail.com
> > >wrote:
> > >
> > >> >How did you fix it?
> > >> We found some "link down" and "link up" in some server's syslog, our
> > >> system
> > >> manager suggested to change the network cards on those servers, so we
> > >> changed them, and the errors disappeared.
> > >>
> > >
> > >
> > > This is interesting. You were just looking at message logs and saw the
> > > above?
> > >
> > >
> > >
> > >>
> > >>
> > >> >So figure out whats happening to that region by grepping its name in
> > the
> > >> >master log. Why is it offline so long? Are machines loaded?
> > Swapping?
> > >>
> > >> >Yeah, its split. Thats normal. Whats not normal is the client not
> > >> finding
> > >> >the daughter split in its new location. Did the daughters get
> deployed
> > >> >promptly?
> > >> I'm sorry, I don't know exactly how I can get these infomation from
> the
> > >> log.
> > >> Can you tell me? And I have upload the logs to sky drives
> > >>
> > >>
> >
> http://cid-a331bb289a14fbef.skydrive.live.com/browse.aspx/.Public/1123?uc=1&isFromRichUpload=1&lc=2052
> > >> .
> > >> There are two master logs, which are from different days, and two
> > >> regionserver logs, which are from the first 2 shutdown servers.
> > >>
> > >>
> > > I'm looking at these logs now.
> > >
> > >
> > >
> > >> >
> > >> >Are the crawlers running on same machines as hbase?
> > >> Yes, they are. And the cluster we are using is like that:
> > >> 1 master/namenode/jobtracker/crawler server.
> > >> 6 rs/zk/datanode/tasktracker/crawler client.
> > >>
> > >>
> > > You don't need 6 zk nodes. Make it 3 (In an earlier mail you said you
> > had
> > > 3 only but above would seem to say you have 6 zk nodes). You could up
> > the
> > > ticktime from 2 to 3 seconds (We learned that maximum session time is
> > > 20*ticktime no matter what session timeout is set to).
> > >
> > > For 8Gs of RAM and a single disk might not be sufficient to carry all
> of
> > > the above daemons on a single node. You have cluster-wide monitoring
> > > setup? Are these nodes swapping? From the logs I've seen, HDFS is for
> > sure
> > > struggling. The crawler writes straight into hbase or does it write
> the
> > > local disk? What is the tasktracker doing? Are mapreduce jobs running
> > > concurrently on these nodes?
> > >
> > >
> > >> >Can you update to hbase 0.20.2? It has a bunch of fixes that could
> be
> > >> >related to the above.
> > >> We have updated to hbase 0.20.2 this morning, and we will re-run the
> > >> crawler
> > >> in a minute, and we will share the result with you.
> > >>
> > >> Good stuff.
> > >
> > > I'll be back after some study of your logs.
> > > St.Ack
> > >
> > >
> > >
> > >> Regards,
> > >> LvZheng.
> > >>
> > >>
> > >> 2009/11/21 stack <st...@duboce.net>
> > >>
> > >> > On Fri, Nov 20, 2009 at 12:28 AM, Zheng Lv <
> lvzheng19800619@gmail.com
> > >> > >wrote:
> > >> >
> > >> > > Hello Stack,
> > >> > > Remember the "no route to host" exceptions last time? Now there
> > isn't
> > >> any
> > >> > > more, and the test program can be running for several days.
> > >> >
> > >> >
> > >> > How did you fix it?
> > >> >
> > >> >
> > >> >
> > >> > > Thank you.
> > >> > > Recently we started running our crawling program, which crawls
> > >> webpages
> > >> > and
> > >> > > then insert them to hbase.
> > >> > > But we got so many
> > "org.apache.hadoop.hbase.NotServingRegionException"
> > >> > like
> > >> > > that:
> > >> > >
> > >> > > 2009-11-20 12:36:41,898 ERROR
> > >> > > org.apache.hadoop.hbase.regionserver.HRegionServer:
> > >> > > org.apache.hadoop.hbase.NotServingRegionException: webpage,
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> http://bbs.city.tianya.cn/tianyacity/Content/178/1/536629.shtml,1258691377544
> > >> > >
> > >> >
> > >> > So figure out whats happening to that region by grepping its name in
> > the
> > >> > master log. Why is it offline so long? Are machines loaded?
> > Swapping?
> > >> >
> > >> > Are the crawlers running on same machines as hbase?
> > >> >
> > >> > What crawler are you using?
> > >> >
> > >> > Andrew Purtell has written up some notes on getting a nice balance
> > >> between
> > >> > crawl process and hbase such that all runs smoothly in private
> > >> > correspondence. Let me ask him if its ok to forward the list.
> > >> >
> > >> >
> > >> > ....
> > >> >
> > >> > 2009-11-20 12:36:25,259 INFO
> > >> org.apache.hadoop.hbase.master.ServerManager:
> > >> > > Processing MSG_REPORT_SPLIT:
> > >> > > webpage,http:\x2F\x2Fbbs.city.tianya.cn<http://x2fbbs.city.tianya.cn/>
> <http://x2fbbs.city.tianya.cn/><
> > >> http://x2fbbs.city.tianya.cn/>
> > >> > >
> \x2Ftianyacity\x2FContent\x2F178\x2F1\x2F536629.shtml,1258691377544:
> > >> > > Daughters; webpage,http:\x2F\x2Fbbs.city.tianya.cn<http://x2fbbs.city.tianya.cn/>
> <http://x2fbbs.city.tianya.cn/>
> > <
> > >> http://x2fbbs.city.tianya.cn/>
> > >> > >
> \x2Ftianyacity\x2FContent\x2F178\x2F1\x2F536629.shtml,1258691779496,
> > >> > > webpage,http:\x2F\x2Fbbs.city.tianya.cn<http://x2fbbs.city.tianya.cn/>
> <http://x2fbbs.city.tianya.cn/><
> > >> http://x2fbbs.city.tianya.cn/>
> > >> > >
> \x2Ftianyacity\x2FContent\x2F329\x2F1\x2F164370.shtml,1258691779496
> > >> > > from ubuntu12,60020,1258687326554;
> > >> > >
> > >> > > Yeah, its split. Thats normal. Whats not normal is the client
> not
> > >> > finding
> > >> > the daughter split in its new location. Did the daughters get
> > deployed
> > >> > promptly?
> > >> >
> > >> >
> > >> >
> > >> > > And a few hours later, some rs shutdown.
> > >> > >
> > >> > > I read the mail
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> http://mail-archives.apache.org/mod_mbox/hadoop-hbase-user/200907.mbox/%3C9b27a8a60907272122y1bfa6254n95948942d5ca7f88@mail.gmail.com%3E
> > >> > > ,
> > >> > > which was sent by my partner Angus. In the mail you told us it was
> a
> > >> case
> > >> > > of
> > >> > > "HBASE-1671", Fix Version of which is 0.20.0, but the hbase
> version
> > we
> > >> > are
> > >> > > using is just 0.20.0.
> > >> > >
> > >> >
> > >> > Can you update to hbase 0.20.2? It has a bunch of fixes that could
> be
> > >> > related to the above.
> > >> > Yours,
> > >> > St.Ack
> > >> >
> > >> >
> > >> >
> > >> > > Any idea?
> > >> > > Best Regards,
> > >> > > LvZheng
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > 2009/10/13 stack <st...@duboce.net>
> > >> > >
> > >> > > > Thanks for posting. Its much easier reading the logs from
> there.
> > >> > > >
> > >> > > > Looking in nohup.out I see it can't find region
> > 'webpage,http:\x2F\
> > >> > > > x2Fnews.163.com <http://x2fnews.163.com/> <
> http://x2fnews.163.com/> <
> > http://x2fnews.163.com/> <http://x2fnews.163.com/>
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> \x2F09\x2F080\x2F0\x2F5FOO155J0001124J.html1255072992000_751685,1255316061169'.
> > >> > > > It never finds it. It looks like it was assigned successfully
> to
> > >> > > > 192.168.33.5 going by the master log. Once you've figured out
> the
> > >> > > > hardware/networking issues, lets work at getting that region
> back
> > on
> > >> > > line.
> > >> > > >
> > >> > > > The master timed out its session against zk because of 'no route
> > to
> > >> > > host'.
> > >> > > >
> > >> > > > St.Ack
> > >> > > >
> > >> > > > On Mon, Oct 12, 2009 at 12:23 AM, Zheng Lv <
> > >> lvzheng19800619@gmail.com
> > >> > > > >wrote:
> > >> > > >
> > >> > > > > Hello Stack,
> > >> > > > > I have enabled DEBUG and restarted the test program. This
> > time
> > >> the
> > >> > > > > master shut down, and I have put the logs on skydrive.
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> http://cid-a331bb289a14fbef.skydrive.live.com/browse.aspx/.Public?uc=2&isFromRichUpload=1
> > >> > > > > .
> > >> > > > > "nohup.out" is our test program log,
> > >> > "hbase-cyd-master-ubuntu6.log"
> > >> > > is
> > >> > > > > master log.
> > >> > > > >
> > >> > > > > On the other hand, today we found that when we run "dmesg",
> > >> there
> > >> > > were
> > >> > > > > many logs like "[3641697.122769] r8169: eth0: link down". And
> I
> > >> think
> > >> > > > this
> > >> > > > > might be the reason of so many "no route to host" and "Time
> > Out".
> > >> Now
> > >> > > our
> > >> > > > > system manager is checking, if we have a result we will let
> you
> > >> > know.:)
> > >> > > > > Thanks,
> > >> > > > > LvZheng.
> > >> > > > >
> > >> > > > > 2009/10/11 stack <st...@duboce.net>
> > >> > > > >
> > >> > > > > > On Fri, Oct 9, 2009 at 3:18 AM, Zheng Lv <
> > >> > lvzheng19800619@gmail.com>
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > ...
> > >> > > > > > > so,
> > >> > > > > > > > please remove the delay so hbase fails faster so it
> > >> doesn't
> > >> > > take
> > >> > > > > so
> > >> > > > > > > long to
> > >> > > > > > > > figure the issue.
> > >> > > > > > > > Are you inserting every 10ms because hbase is falling
> > >> over
> > >> > on
> > >> > > > you?
> > >> > > > > > If
> > >> > > > > > > Yes I inserted every 10ms because I'm afraid hbase
> would
> > >> fall
> > >> > > > over.
> > >> > > > > > Now
> > >> > > > > > > I have removed the delay.
> > >> > > > > > >
> > >> > > > > > > After doing these, We have run the test program again,
> > and
> > >> one
> > >> > > > > region
> > >> > > > > > > server shut down after about 2 hours, another one 3.
> > >> > > > > > > I will post the logs on these two servers in following
> > >> reply
> > >> > > > mails.
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > Thanks for doing the above.
> > >> > > > > >
> > >> > > > > > For the future, debugging, please enable DEBUG and put your
> > logs
> > >> > > > > somewhere
> > >> > > > > > where I can pull them or put them up in pastebin. Logs in
> > email
> > >> > > > messages
> > >> > > > > > are hard to follow. Thanks.
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > > > Ok. So this is hbase 0.20.0? Tell us about your
> > >> hardware.
> > >> > > > What
> > >> > > > > > kind
> > >> > > > > > > is
> > >> > > > > > > > it? CPU/RAM/Disks.
> > >> > > > > > > Yes we are using hbase 0.20.0. And the following is
> our
> > >> > > > hardware:
> > >> > > > > > >
> > >> > > > > > > CPU:amd x3 710
> > >> > > > > > > RAM:8g ddr2 800
> > >> > > > > > > Disk:270g(raid0)
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > Thats an interesting chip -- 3 cores! The above should be
> > fine
> > >> as
> > >> > > long
> > >> > > > > as
> > >> > > > > > you coral your mapreduce jobs running on same cluster.
> > >> > > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > > We have 7 servers with above hardware, one for master,
> > >> three
> > >> > for
> > >> > > > > > > namenodes / regionservers, and the other 3 for zks.
> > >> > > > > > > By the way, what kind of hardware and environment do
> you
> > >> > suggest
> > >> > > > we
> > >> > > > > > > have?
> > >> > > > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > This configuration seems fine to start with. Later we might
> > >> > > experiment
> > >> > > > > > running zk on same machines as regionservers and then up
> > number
> > >> of
> > >> > > > > > regionservers to 6 and up the quorum members to 5.
> > >> > > > > >
> > >> > > > > > St.Ack
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > >
> > >> > > > > > > Thank you, very much.
> > >> > > > > > > LvZheng.
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>
>
>
>
>
Re: is there any problem with our environment?
Posted by Zheng Lv <lv...@gmail.com>.
Hello Andy,
Thank you very much for your suggestions, that will be helpful.
I have one question, according to what you adjust the first 2
properties(hbase.hregion.memstore.block.multiplier,
hbase.hstore.blockingStoreFiles)?
Regards,
LvZheng.
2009/11/26 Andrew Purtell <ap...@apache.org>
> Try this: Do not cache on the crawlers, just write through.
>
> Run each region server with plenty of heap (4 GB to start). So it seems you
> need more RAM on your systems, or you should move your crawlers off to
> separate servers to free up RAM and CPU.
>
> Adjust your HBase site config as follows:
>
> <property>
> <name>hbase.hregion.memstore.block.multiplier</name>
> <value>4</value>
> </property>
>
> <property>
> <name>hbase.hstore.blockingStoreFiles</name>
> <value>20</value>
> </property>
>
> <property>
> <name>dfs.replication</name>
> <value>2</value>
> </property>
>
> <property>
> <name>hfile.block.cache.size</name>
> <!-- disable -->
> <value>0</value>
> </property>
>
> Be advised that the probability of OOM under high write load is high if you
> do not provide adequate heap. This may manifest either as OOM exceptions or
> as GC taking so much time as to drop the ZK session. Concurrent garbage
> collector will keep GC latency low but it needs a larger available heap
> space in return. You should also tune your HBASE_OPTS in conf/hbase-env.sh
> according to this advice:
> http://osdir.com/ml/hbase-user-hadoop-apache/2009-10/msg00276.html
>
> With your monitoring tools watch the amount of disk wait time on the OS
> level. If disk wait times exceed 40%, you need more spindles (disk) for your
> datanodes or more datanodes (servers) to spread the I/O load.
>
> - Andy
>
>
>
>
> ________________________________
> From: Zheng Lv <lv...@gmail.com>
> To: hbase-user@hadoop.apache.org
> Sent: Tue, November 24, 2009 8:22:04 PM
> Subject: Re: is there any problem with our environment?
>
> Hello Stack,
> >Sorry for taking time getting back to you Lv.
> Never mind:).
>
> >I posted Andrew's crawling notes yesterday. Did you see them? I thought
> >they might be of use to you.
> You mean you'v sent mail to me? I didn't get it.
>
> >This is interesting. You were just looking at message logs and saw the
> >above?
> We checked /var/log/messages on some servers, and we got many "link down"
> and "link up".
>
> >You don't need 6 zk nodes. Make it 3 (In an earlier mail you said you had
> 3
> >only but above would seem to say you have 6 zk nodes). You could up the
> >ticktime from 2 to 3 seconds (We learned that maximum session time is
> >20*ticktime no matter what session timeout is set to).
> >
> We will change our cluster as you told me.
> >(In an earlier mail you said you had 3 only but above would seem to say
> you
> have 6 zk nodes)
> We added some, we thought 3 is insufficient.
>
> >For 8Gs of RAM and a single disk might not be sufficient to carry all of
> the
> >above daemons on a single node. You have cluster-wide monitoring setup?
> >Are these nodes swapping? From the logs I've seen, HDFS is for sure
> >struggling.
> Our system manager has just deployed mrtg+snmp on those servers, and we
> will
> share the data with you.
>
> >The crawler writes straight into hbase or does it write the
> >local disk?
> Each crawler has a cache in memory, when it is full, the crawler will flush
> webpages in cache to hbase.
> >What is the tasktracker doing? Are mapreduce jobs running
> >concurrently on these nodes?
> Now the tasktracker process is running, but there is not any job on it.
> But,
> when updater, which is another part of our product, is on there will be.
> We will restart hbase0.20.2 with DEBUG enabled and monitor it, I'll be back
> when we get useful data.
> Regards,
> Lv Zheng.
>
>
>
>
>
>
>
> 2009/11/25 stack <st...@duboce.net>
>
> > I took a look at your logs. Please run with DEBUG enabled going forward
> > (if
> > you upgraded to 0.20.2, then it'll be on by default).
> >
> > HDFS is struggling.
> >
> > Looking at this region:
> >
> >
> >
> http://blog.blog.tianya.cn/blogger/post_Date.asp?blogID=1818563&CategoryID=1256944&idWriter=0&Key=0&NextPostID=888888888&PageNo=2
> >
> > ... there seems to be an issue w/ accounting. It looks like it got
> > assigned
> > to 13 before the logs you gave me but then the regionserver its on, while
> > it
> > thinks its carrying it in one regard -- its trying to flush it so it can
> > let
> > go of old WAL files -- it then is telling clients that ask for it that
> its
> > not serving it. Seeing an earlier master and regionserver 13 log would
> > help
> > figure what happened. Try restart on 0.20.2. Tell me more about the
> > loading on these machines, how its done. Do you have monitoring software
> > running?
> >
> > Thanks,
> > St.Ack
> >
> >
> >
> >
> >
> > On Tue, Nov 24, 2009 at 10:30 AM, stack <st...@duboce.net> wrote:
> >
> > > Sorry for taking time getting back to you Lv.
> > >
> > > I posted Andrew's crawling notes yesterday. Did you see them? I
> thought
> > > they might be of use to you.
> > >
> > >
> > > On Sun, Nov 22, 2009 at 10:30 PM, Zheng Lv <lvzheng19800619@gmail.com
> > >wrote:
> > >
> > >> >How did you fix it?
> > >> We found some "link down" and "link up" in some server's syslog, our
> > >> system
> > >> manager suggested to change the network cards on those servers, so we
> > >> changed them, and the errors disappeared.
> > >>
> > >
> > >
> > > This is interesting. You were just looking at message logs and saw the
> > > above?
> > >
> > >
> > >
> > >>
> > >>
> > >> >So figure out whats happening to that region by grepping its name in
> > the
> > >> >master log. Why is it offline so long? Are machines loaded?
> > Swapping?
> > >>
> > >> >Yeah, its split. Thats normal. Whats not normal is the client not
> > >> finding
> > >> >the daughter split in its new location. Did the daughters get
> deployed
> > >> >promptly?
> > >> I'm sorry, I don't know exactly how I can get these infomation from
> the
> > >> log.
> > >> Can you tell me? And I have upload the logs to sky drives
> > >>
> > >>
> >
> http://cid-a331bb289a14fbef.skydrive.live.com/browse.aspx/.Public/1123?uc=1&isFromRichUpload=1&lc=2052
> > >> .
> > >> There are two master logs, which are from different days, and two
> > >> regionserver logs, which are from the first 2 shutdown servers.
> > >>
> > >>
> > > I'm looking at these logs now.
> > >
> > >
> > >
> > >> >
> > >> >Are the crawlers running on same machines as hbase?
> > >> Yes, they are. And the cluster we are using is like that:
> > >> 1 master/namenode/jobtracker/crawler server.
> > >> 6 rs/zk/datanode/tasktracker/crawler client.
> > >>
> > >>
> > > You don't need 6 zk nodes. Make it 3 (In an earlier mail you said you
> > had
> > > 3 only but above would seem to say you have 6 zk nodes). You could up
> > the
> > > ticktime from 2 to 3 seconds (We learned that maximum session time is
> > > 20*ticktime no matter what session timeout is set to).
> > >
> > > For 8Gs of RAM and a single disk might not be sufficient to carry all
> of
> > > the above daemons on a single node. You have cluster-wide monitoring
> > > setup? Are these nodes swapping? From the logs I've seen, HDFS is for
> > sure
> > > struggling. The crawler writes straight into hbase or does it write
> the
> > > local disk? What is the tasktracker doing? Are mapreduce jobs running
> > > concurrently on these nodes?
> > >
> > >
> > >> >Can you update to hbase 0.20.2? It has a bunch of fixes that could
> be
> > >> >related to the above.
> > >> We have updated to hbase 0.20.2 this morning, and we will re-run the
> > >> crawler
> > >> in a minute, and we will share the result with you.
> > >>
> > >> Good stuff.
> > >
> > > I'll be back after some study of your logs.
> > > St.Ack
> > >
> > >
> > >
> > >> Regards,
> > >> LvZheng.
> > >>
> > >>
> > >> 2009/11/21 stack <st...@duboce.net>
> > >>
> > >> > On Fri, Nov 20, 2009 at 12:28 AM, Zheng Lv <
> lvzheng19800619@gmail.com
> > >> > >wrote:
> > >> >
> > >> > > Hello Stack,
> > >> > > Remember the "no route to host" exceptions last time? Now there
> > isn't
> > >> any
> > >> > > more, and the test program can be running for several days.
> > >> >
> > >> >
> > >> > How did you fix it?
> > >> >
> > >> >
> > >> >
> > >> > > Thank you.
> > >> > > Recently we started running our crawling program, which crawls
> > >> webpages
> > >> > and
> > >> > > then insert them to hbase.
> > >> > > But we got so many
> > "org.apache.hadoop.hbase.NotServingRegionException"
> > >> > like
> > >> > > that:
> > >> > >
> > >> > > 2009-11-20 12:36:41,898 ERROR
> > >> > > org.apache.hadoop.hbase.regionserver.HRegionServer:
> > >> > > org.apache.hadoop.hbase.NotServingRegionException: webpage,
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> http://bbs.city.tianya.cn/tianyacity/Content/178/1/536629.shtml,1258691377544
> > >> > >
> > >> >
> > >> > So figure out whats happening to that region by grepping its name in
> > the
> > >> > master log. Why is it offline so long? Are machines loaded?
> > Swapping?
> > >> >
> > >> > Are the crawlers running on same machines as hbase?
> > >> >
> > >> > What crawler are you using?
> > >> >
> > >> > Andrew Purtell has written up some notes on getting a nice balance
> > >> between
> > >> > crawl process and hbase such that all runs smoothly in private
> > >> > correspondence. Let me ask him if its ok to forward the list.
> > >> >
> > >> >
> > >> > ....
> > >> >
> > >> > 2009-11-20 12:36:25,259 INFO
> > >> org.apache.hadoop.hbase.master.ServerManager:
> > >> > > Processing MSG_REPORT_SPLIT:
> > >> > > webpage,http:\x2F\x2Fbbs.city.tianya.cn<http://x2fbbs.city.tianya.cn/>
> <http://x2fbbs.city.tianya.cn/><
> > >> http://x2fbbs.city.tianya.cn/>
> > >> > >
> \x2Ftianyacity\x2FContent\x2F178\x2F1\x2F536629.shtml,1258691377544:
> > >> > > Daughters; webpage,http:\x2F\x2Fbbs.city.tianya.cn<http://x2fbbs.city.tianya.cn/>
> <http://x2fbbs.city.tianya.cn/>
> > <
> > >> http://x2fbbs.city.tianya.cn/>
> > >> > >
> \x2Ftianyacity\x2FContent\x2F178\x2F1\x2F536629.shtml,1258691779496,
> > >> > > webpage,http:\x2F\x2Fbbs.city.tianya.cn<http://x2fbbs.city.tianya.cn/>
> <http://x2fbbs.city.tianya.cn/><
> > >> http://x2fbbs.city.tianya.cn/>
> > >> > >
> \x2Ftianyacity\x2FContent\x2F329\x2F1\x2F164370.shtml,1258691779496
> > >> > > from ubuntu12,60020,1258687326554;
> > >> > >
> > >> > > Yeah, its split. Thats normal. Whats not normal is the client
> not
> > >> > finding
> > >> > the daughter split in its new location. Did the daughters get
> > deployed
> > >> > promptly?
> > >> >
> > >> >
> > >> >
> > >> > > And a few hours later, some rs shutdown.
> > >> > >
> > >> > > I read the mail
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> http://mail-archives.apache.org/mod_mbox/hadoop-hbase-user/200907.mbox/%3C9b27a8a60907272122y1bfa6254n95948942d5ca7f88@mail.gmail.com%3E
> > >> > > ,
> > >> > > which was sent by my partner Angus. In the mail you told us it was
> a
> > >> case
> > >> > > of
> > >> > > "HBASE-1671", Fix Version of which is 0.20.0, but the hbase
> version
> > we
> > >> > are
> > >> > > using is just 0.20.0.
> > >> > >
> > >> >
> > >> > Can you update to hbase 0.20.2? It has a bunch of fixes that could
> be
> > >> > related to the above.
> > >> > Yours,
> > >> > St.Ack
> > >> >
> > >> >
> > >> >
> > >> > > Any idea?
> > >> > > Best Regards,
> > >> > > LvZheng
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > 2009/10/13 stack <st...@duboce.net>
> > >> > >
> > >> > > > Thanks for posting. Its much easier reading the logs from
> there.
> > >> > > >
> > >> > > > Looking in nohup.out I see it can't find region
> > 'webpage,http:\x2F\
> > >> > > > x2Fnews.163.com <http://x2fnews.163.com/> <
> http://x2fnews.163.com/> <
> > http://x2fnews.163.com/> <http://x2fnews.163.com/>
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> \x2F09\x2F080\x2F0\x2F5FOO155J0001124J.html1255072992000_751685,1255316061169'.
> > >> > > > It never finds it. It looks like it was assigned successfully
> to
> > >> > > > 192.168.33.5 going by the master log. Once you've figured out
> the
> > >> > > > hardware/networking issues, lets work at getting that region
> back
> > on
> > >> > > line.
> > >> > > >
> > >> > > > The master timed out its session against zk because of 'no route
> > to
> > >> > > host'.
> > >> > > >
> > >> > > > St.Ack
> > >> > > >
> > >> > > > On Mon, Oct 12, 2009 at 12:23 AM, Zheng Lv <
> > >> lvzheng19800619@gmail.com
> > >> > > > >wrote:
> > >> > > >
> > >> > > > > Hello Stack,
> > >> > > > > I have enabled DEBUG and restarted the test program. This
> > time
> > >> the
> > >> > > > > master shut down, and I have put the logs on skydrive.
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> http://cid-a331bb289a14fbef.skydrive.live.com/browse.aspx/.Public?uc=2&isFromRichUpload=1
> > >> > > > > .
> > >> > > > > "nohup.out" is our test program log,
> > >> > "hbase-cyd-master-ubuntu6.log"
> > >> > > is
> > >> > > > > master log.
> > >> > > > >
> > >> > > > > On the other hand, today we found that when we run "dmesg",
> > >> there
> > >> > > were
> > >> > > > > many logs like "[3641697.122769] r8169: eth0: link down". And
> I
> > >> think
> > >> > > > this
> > >> > > > > might be the reason of so many "no route to host" and "Time
> > Out".
> > >> Now
> > >> > > our
> > >> > > > > system manager is checking, if we have a result we will let
> you
> > >> > know.:)
> > >> > > > > Thanks,
> > >> > > > > LvZheng.
> > >> > > > >
> > >> > > > > 2009/10/11 stack <st...@duboce.net>
> > >> > > > >
> > >> > > > > > On Fri, Oct 9, 2009 at 3:18 AM, Zheng Lv <
> > >> > lvzheng19800619@gmail.com>
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > ...
> > >> > > > > > > so,
> > >> > > > > > > > please remove the delay so hbase fails faster so it
> > >> doesn't
> > >> > > take
> > >> > > > > so
> > >> > > > > > > long to
> > >> > > > > > > > figure the issue.
> > >> > > > > > > > Are you inserting every 10ms because hbase is falling
> > >> over
> > >> > on
> > >> > > > you?
> > >> > > > > > If
> > >> > > > > > > Yes I inserted every 10ms because I'm afraid hbase
> would
> > >> fall
> > >> > > > over.
> > >> > > > > > Now
> > >> > > > > > > I have removed the delay.
> > >> > > > > > >
> > >> > > > > > > After doing these, We have run the test program again,
> > and
> > >> one
> > >> > > > > region
> > >> > > > > > > server shut down after about 2 hours, another one 3.
> > >> > > > > > > I will post the logs on these two servers in following
> > >> reply
> > >> > > > mails.
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > Thanks for doing the above.
> > >> > > > > >
> > >> > > > > > For the future, debugging, please enable DEBUG and put your
> > logs
> > >> > > > > somewhere
> > >> > > > > > where I can pull them or put them up in pastebin. Logs in
> > email
> > >> > > > messages
> > >> > > > > > are hard to follow. Thanks.
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > > > Ok. So this is hbase 0.20.0? Tell us about your
> > >> hardware.
> > >> > > > What
> > >> > > > > > kind
> > >> > > > > > > is
> > >> > > > > > > > it? CPU/RAM/Disks.
> > >> > > > > > > Yes we are using hbase 0.20.0. And the following is
> our
> > >> > > > hardware:
> > >> > > > > > >
> > >> > > > > > > CPU:amd x3 710
> > >> > > > > > > RAM:8g ddr2 800
> > >> > > > > > > Disk:270g(raid0)
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > Thats an interesting chip -- 3 cores! The above should be
> > fine
> > >> as
> > >> > > long
> > >> > > > > as
> > >> > > > > > you coral your mapreduce jobs running on same cluster.
> > >> > > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > > We have 7 servers with above hardware, one for master,
> > >> three
> > >> > for
> > >> > > > > > > namenodes / regionservers, and the other 3 for zks.
> > >> > > > > > > By the way, what kind of hardware and environment do
> you
> > >> > suggest
> > >> > > > we
> > >> > > > > > > have?
> > >> > > > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > This configuration seems fine to start with. Later we might
> > >> > > experiment
> > >> > > > > > running zk on same machines as regionservers and then up
> > number
> > >> of
> > >> > > > > > regionservers to 6 and up the quorum members to 5.
> > >> > > > > >
> > >> > > > > > St.Ack
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > >
> > >> > > > > > > Thank you, very much.
> > >> > > > > > > LvZheng.
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>
>
>
>
>
Re: is there any problem with our environment?
Posted by Andrew Purtell <ap...@apache.org>.
Yes.
But, you should consider using Ganglia to monitor your servers: http://ganglia.sourceforge.net/
You can track memory, CPU, disk, and network use over time and correlate events in your Hadoop and HBase logs with metrics readings.
- Andy
________________________________
From: Zheng Lv <lv...@gmail.com>
To: hbase-user@hadoop.apache.org
Sent: Thu, November 26, 2009 7:23:30 PM
Subject: Re: is there any problem with our environment?
Hello Andy,
I have a question.
>If disk wait times exceed 40%, you need more spindles (disk) for your
datanodes or more datanodes (servers) to spread the I/O >load.
Is the "disk wait times exceed 40%" you mentioned means the "wa" value when
running "vmstat"? If not, what does it mean? and how do we monitor it?
Thank you.
Regards,
LvZheng.
2009/11/26 Andrew Purtell <ap...@apache.org>
> Try this: Do not cache on the crawlers, just write through.
>
> Run each region server with plenty of heap (4 GB to start). So it seems you
> need more RAM on your systems, or you should move your crawlers off to
> separate servers to free up RAM and CPU.
>
> Adjust your HBase site config as follows:
>
> <property>
> <name>hbase.hregion.memstore.block.multiplier</name>
> <value>4</value>
> </property>
>
> <property>
> <name>hbase.hstore.blockingStoreFiles</name>
> <value>20</value>
> </property>
>
> <property>
> <name>dfs.replication</name>
> <value>2</value>
> </property>
>
> <property>
> <name>hfile.block.cache.size</name>
> <!-- disable -->
> <value>0</value>
> </property>
>
> Be advised that the probability of OOM under high write load is high if you
> do not provide adequate heap. This may manifest either as OOM exceptions or
> as GC taking so much time as to drop the ZK session. Concurrent garbage
> collector will keep GC latency low but it needs a larger available heap
> space in return. You should also tune your HBASE_OPTS in conf/hbase-env.sh
> according to this advice:
> http://osdir.com/ml/hbase-user-hadoop-apache/2009-10/msg00276.html
>
> With your monitoring tools watch the amount of disk wait time on the OS
> level. If disk wait times exceed 40%, you need more spindles (disk) for your
> datanodes or more datanodes (servers) to spread the I/O load.
>
> - Andy
>
>
>
>
> ________________________________
> From: Zheng Lv <lv...@gmail.com>
> To: hbase-user@hadoop.apache.org
> Sent: Tue, November 24, 2009 8:22:04 PM
> Subject: Re: is there any problem with our environment?
>
> Hello Stack,
> >Sorry for taking time getting back to you Lv.
> Never mind:).
>
> >I posted Andrew's crawling notes yesterday. Did you see them? I thought
> >they might be of use to you.
> You mean you'v sent mail to me? I didn't get it.
>
> >This is interesting. You were just looking at message logs and saw the
> >above?
> We checked /var/log/messages on some servers, and we got many "link down"
> and "link up".
>
> >You don't need 6 zk nodes. Make it 3 (In an earlier mail you said you had
> 3
> >only but above would seem to say you have 6 zk nodes). You could up the
> >ticktime from 2 to 3 seconds (We learned that maximum session time is
> >20*ticktime no matter what session timeout is set to).
> >
> We will change our cluster as you told me.
> >(In an earlier mail you said you had 3 only but above would seem to say
> you
> have 6 zk nodes)
> We added some, we thought 3 is insufficient.
>
> >For 8Gs of RAM and a single disk might not be sufficient to carry all of
> the
> >above daemons on a single node. You have cluster-wide monitoring setup?
> >Are these nodes swapping? From the logs I've seen, HDFS is for sure
> >struggling.
> Our system manager has just deployed mrtg+snmp on those servers, and we
> will
> share the data with you.
>
> >The crawler writes straight into hbase or does it write the
> >local disk?
> Each crawler has a cache in memory, when it is full, the crawler will flush
> webpages in cache to hbase.
> >What is the tasktracker doing? Are mapreduce jobs running
> >concurrently on these nodes?
> Now the tasktracker process is running, but there is not any job on it.
> But,
> when updater, which is another part of our product, is on there will be.
> We will restart hbase0.20.2 with DEBUG enabled and monitor it, I'll be back
> when we get useful data.
> Regards,
> Lv Zheng.
>
>
>
>
>
>
>
> 2009/11/25 stack <st...@duboce.net>
>
> > I took a look at your logs. Please run with DEBUG enabled going forward
> > (if
> > you upgraded to 0.20.2, then it'll be on by default).
> >
> > HDFS is struggling.
> >
> > Looking at this region:
> >
> >
> >
> http://blog.blog.tianya.cn/blogger/post_Date.asp?blogID=1818563&CategoryID=1256944&idWriter=0&Key=0&NextPostID=888888888&PageNo=2
> >
> > ... there seems to be an issue w/ accounting. It looks like it got
> > assigned
> > to 13 before the logs you gave me but then the regionserver its on, while
> > it
> > thinks its carrying it in one regard -- its trying to flush it so it can
> > let
> > go of old WAL files -- it then is telling clients that ask for it that
> its
> > not serving it. Seeing an earlier master and regionserver 13 log would
> > help
> > figure what happened. Try restart on 0.20.2. Tell me more about the
> > loading on these machines, how its done. Do you have monitoring software
> > running?
> >
> > Thanks,
> > St.Ack
> >
> >
> >
> >
> >
> > On Tue, Nov 24, 2009 at 10:30 AM, stack <st...@duboce.net> wrote:
> >
> > > Sorry for taking time getting back to you Lv.
> > >
> > > I posted Andrew's crawling notes yesterday. Did you see them? I
> thought
> > > they might be of use to you.
> > >
> > >
> > > On Sun, Nov 22, 2009 at 10:30 PM, Zheng Lv <lvzheng19800619@gmail.com
> > >wrote:
> > >
> > >> >How did you fix it?
> > >> We found some "link down" and "link up" in some server's syslog, our
> > >> system
> > >> manager suggested to change the network cards on those servers, so we
> > >> changed them, and the errors disappeared.
> > >>
> > >
> > >
> > > This is interesting. You were just looking at message logs and saw the
> > > above?
> > >
> > >
> > >
> > >>
> > >>
> > >> >So figure out whats happening to that region by grepping its name in
> > the
> > >> >master log. Why is it offline so long? Are machines loaded?
> > Swapping?
> > >>
> > >> >Yeah, its split. Thats normal. Whats not normal is the client not
> > >> finding
> > >> >the daughter split in its new location. Did the daughters get
> deployed
> > >> >promptly?
> > >> I'm sorry, I don't know exactly how I can get these infomation from
> the
> > >> log.
> > >> Can you tell me? And I have upload the logs to sky drives
> > >>
> > >>
> >
> http://cid-a331bb289a14fbef.skydrive.live.com/browse.aspx/.Public/1123?uc=1&isFromRichUpload=1&lc=2052
> > >> .
> > >> There are two master logs, which are from different days, and two
> > >> regionserver logs, which are from the first 2 shutdown servers.
> > >>
> > >>
> > > I'm looking at these logs now.
> > >
> > >
> > >
> > >> >
> > >> >Are the crawlers running on same machines as hbase?
> > >> Yes, they are. And the cluster we are using is like that:
> > >> 1 master/namenode/jobtracker/crawler server.
> > >> 6 rs/zk/datanode/tasktracker/crawler client.
> > >>
> > >>
> > > You don't need 6 zk nodes. Make it 3 (In an earlier mail you said you
> > had
> > > 3 only but above would seem to say you have 6 zk nodes). You could up
> > the
> > > ticktime from 2 to 3 seconds (We learned that maximum session time is
> > > 20*ticktime no matter what session timeout is set to).
> > >
> > > For 8Gs of RAM and a single disk might not be sufficient to carry all
> of
> > > the above daemons on a single node. You have cluster-wide monitoring
> > > setup? Are these nodes swapping? From the logs I've seen, HDFS is for
> > sure
> > > struggling. The crawler writes straight into hbase or does it write
> the
> > > local disk? What is the tasktracker doing? Are mapreduce jobs running
> > > concurrently on these nodes?
> > >
> > >
> > >> >Can you update to hbase 0.20.2? It has a bunch of fixes that could
> be
> > >> >related to the above.
> > >> We have updated to hbase 0.20.2 this morning, and we will re-run the
> > >> crawler
> > >> in a minute, and we will share the result with you.
> > >>
> > >> Good stuff.
> > >
> > > I'll be back after some study of your logs.
> > > St.Ack
> > >
> > >
> > >
> > >> Regards,
> > >> LvZheng.
> > >>
> > >>
> > >> 2009/11/21 stack <st...@duboce.net>
> > >>
> > >> > On Fri, Nov 20, 2009 at 12:28 AM, Zheng Lv <
> lvzheng19800619@gmail.com
> > >> > >wrote:
> > >> >
> > >> > > Hello Stack,
> > >> > > Remember the "no route to host" exceptions last time? Now there
> > isn't
> > >> any
> > >> > > more, and the test program can be running for several days.
> > >> >
> > >> >
> > >> > How did you fix it?
> > >> >
> > >> >
> > >> >
> > >> > > Thank you.
> > >> > > Recently we started running our crawling program, which crawls
> > >> webpages
> > >> > and
> > >> > > then insert them to hbase.
> > >> > > But we got so many
> > "org.apache.hadoop.hbase.NotServingRegionException"
> > >> > like
> > >> > > that:
> > >> > >
> > >> > > 2009-11-20 12:36:41,898 ERROR
> > >> > > org.apache.hadoop.hbase.regionserver.HRegionServer:
> > >> > > org.apache.hadoop.hbase.NotServingRegionException: webpage,
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> http://bbs.city.tianya.cn/tianyacity/Content/178/1/536629.shtml,1258691377544
> > >> > >
> > >> >
> > >> > So figure out whats happening to that region by grepping its name in
> > the
> > >> > master log. Why is it offline so long? Are machines loaded?
> > Swapping?
> > >> >
> > >> > Are the crawlers running on same machines as hbase?
> > >> >
> > >> > What crawler are you using?
> > >> >
> > >> > Andrew Purtell has written up some notes on getting a nice balance
> > >> between
> > >> > crawl process and hbase such that all runs smoothly in private
> > >> > correspondence. Let me ask him if its ok to forward the list.
> > >> >
> > >> >
> > >> > ....
> > >> >
> > >> > 2009-11-20 12:36:25,259 INFO
> > >> org.apache.hadoop.hbase.master.ServerManager:
> > >> > > Processing MSG_REPORT_SPLIT:
> > >> > > webpage,http:\x2F\x2Fbbs.city.tianya.cn<http://x2fbbs.city.tianya.cn/>
> <http://x2fbbs.city.tianya.cn/><
> > >> http://x2fbbs.city.tianya.cn/>
> > >> > >
> \x2Ftianyacity\x2FContent\x2F178\x2F1\x2F536629.shtml,1258691377544:
> > >> > > Daughters; webpage,http:\x2F\x2Fbbs.city.tianya.cn<http://x2fbbs.city.tianya.cn/>
> <http://x2fbbs.city.tianya.cn/>
> > <
> > >> http://x2fbbs.city.tianya.cn/>
> > >> > >
> \x2Ftianyacity\x2FContent\x2F178\x2F1\x2F536629.shtml,1258691779496,
> > >> > > webpage,http:\x2F\x2Fbbs.city.tianya.cn<http://x2fbbs.city.tianya.cn/>
> <http://x2fbbs.city.tianya.cn/><
> > >> http://x2fbbs.city.tianya.cn/>
> > >> > >
> \x2Ftianyacity\x2FContent\x2F329\x2F1\x2F164370.shtml,1258691779496
> > >> > > from ubuntu12,60020,1258687326554;
> > >> > >
> > >> > > Yeah, its split. Thats normal. Whats not normal is the client
> not
> > >> > finding
> > >> > the daughter split in its new location. Did the daughters get
> > deployed
> > >> > promptly?
> > >> >
> > >> >
> > >> >
> > >> > > And a few hours later, some rs shutdown.
> > >> > >
> > >> > > I read the mail
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> http://mail-archives.apache.org/mod_mbox/hadoop-hbase-user/200907.mbox/%3C9b27a8a60907272122y1bfa6254n95948942d5ca7f88@mail.gmail.com%3E
> > >> > > ,
> > >> > > which was sent by my partner Angus. In the mail you told us it was
> a
> > >> case
> > >> > > of
> > >> > > "HBASE-1671", Fix Version of which is 0.20.0, but the hbase
> version
> > we
> > >> > are
> > >> > > using is just 0.20.0.
> > >> > >
> > >> >
> > >> > Can you update to hbase 0.20.2? It has a bunch of fixes that could
> be
> > >> > related to the above.
> > >> > Yours,
> > >> > St.Ack
> > >> >
> > >> >
> > >> >
> > >> > > Any idea?
> > >> > > Best Regards,
> > >> > > LvZheng
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > 2009/10/13 stack <st...@duboce.net>
> > >> > >
> > >> > > > Thanks for posting. Its much easier reading the logs from
> there.
> > >> > > >
> > >> > > > Looking in nohup.out I see it can't find region
> > 'webpage,http:\x2F\
> > >> > > > x2Fnews.163.com <http://x2fnews.163.com/> <
> http://x2fnews.163.com/> <
> > http://x2fnews.163.com/> <http://x2fnews.163.com/>
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> \x2F09\x2F080\x2F0\x2F5FOO155J0001124J.html1255072992000_751685,1255316061169'.
> > >> > > > It never finds it. It looks like it was assigned successfully
> to
> > >> > > > 192.168.33.5 going by the master log. Once you've figured out
> the
> > >> > > > hardware/networking issues, lets work at getting that region
> back
> > on
> > >> > > line.
> > >> > > >
> > >> > > > The master timed out its session against zk because of 'no route
> > to
> > >> > > host'.
> > >> > > >
> > >> > > > St.Ack
> > >> > > >
> > >> > > > On Mon, Oct 12, 2009 at 12:23 AM, Zheng Lv <
> > >> lvzheng19800619@gmail.com
> > >> > > > >wrote:
> > >> > > >
> > >> > > > > Hello Stack,
> > >> > > > > I have enabled DEBUG and restarted the test program. This
> > time
> > >> the
> > >> > > > > master shut down, and I have put the logs on skydrive.
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> http://cid-a331bb289a14fbef.skydrive.live.com/browse.aspx/.Public?uc=2&isFromRichUpload=1
> > >> > > > > .
> > >> > > > > "nohup.out" is our test program log,
> > >> > "hbase-cyd-master-ubuntu6.log"
> > >> > > is
> > >> > > > > master log.
> > >> > > > >
> > >> > > > > On the other hand, today we found that when we run "dmesg",
> > >> there
> > >> > > were
> > >> > > > > many logs like "[3641697.122769] r8169: eth0: link down". And
> I
> > >> think
> > >> > > > this
> > >> > > > > might be the reason of so many "no route to host" and "Time
> > Out".
> > >> Now
> > >> > > our
> > >> > > > > system manager is checking, if we have a result we will let
> you
> > >> > know.:)
> > >> > > > > Thanks,
> > >> > > > > LvZheng.
> > >> > > > >
> > >> > > > > 2009/10/11 stack <st...@duboce.net>
> > >> > > > >
> > >> > > > > > On Fri, Oct 9, 2009 at 3:18 AM, Zheng Lv <
> > >> > lvzheng19800619@gmail.com>
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > ...
> > >> > > > > > > so,
> > >> > > > > > > > please remove the delay so hbase fails faster so it
> > >> doesn't
> > >> > > take
> > >> > > > > so
> > >> > > > > > > long to
> > >> > > > > > > > figure the issue.
> > >> > > > > > > > Are you inserting every 10ms because hbase is falling
> > >> over
> > >> > on
> > >> > > > you?
> > >> > > > > > If
> > >> > > > > > > Yes I inserted every 10ms because I'm afraid hbase
> would
> > >> fall
> > >> > > > over.
> > >> > > > > > Now
> > >> > > > > > > I have removed the delay.
> > >> > > > > > >
> > >> > > > > > > After doing these, We have run the test program again,
> > and
> > >> one
> > >> > > > > region
> > >> > > > > > > server shut down after about 2 hours, another one 3.
> > >> > > > > > > I will post the logs on these two servers in following
> > >> reply
> > >> > > > mails.
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > Thanks for doing the above.
> > >> > > > > >
> > >> > > > > > For the future, debugging, please enable DEBUG and put your
> > logs
> > >> > > > > somewhere
> > >> > > > > > where I can pull them or put them up in pastebin. Logs in
> > email
> > >> > > > messages
> > >> > > > > > are hard to follow. Thanks.
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > > > Ok. So this is hbase 0.20.0? Tell us about your
> > >> hardware.
> > >> > > > What
> > >> > > > > > kind
> > >> > > > > > > is
> > >> > > > > > > > it? CPU/RAM/Disks.
> > >> > > > > > > Yes we are using hbase 0.20.0. And the following is
> our
> > >> > > > hardware:
> > >> > > > > > >
> > >> > > > > > > CPU:amd x3 710
> > >> > > > > > > RAM:8g ddr2 800
> > >> > > > > > > Disk:270g(raid0)
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > Thats an interesting chip -- 3 cores! The above should be
> > fine
> > >> as
> > >> > > long
> > >> > > > > as
> > >> > > > > > you coral your mapreduce jobs running on same cluster.
> > >> > > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > > We have 7 servers with above hardware, one for master,
> > >> three
> > >> > for
> > >> > > > > > > namenodes / regionservers, and the other 3 for zks.
> > >> > > > > > > By the way, what kind of hardware and environment do
> you
> > >> > suggest
> > >> > > > we
> > >> > > > > > > have?
> > >> > > > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > This configuration seems fine to start with. Later we might
> > >> > > experiment
> > >> > > > > > running zk on same machines as regionservers and then up
> > number
> > >> of
> > >> > > > > > regionservers to 6 and up the quorum members to 5.
> > >> > > > > >
> > >> > > > > > St.Ack
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > >
> > >> > > > > > > Thank you, very much.
> > >> > > > > > > LvZheng.
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>
>
>
>
>
Re: is there any problem with our environment?
Posted by Zheng Lv <lv...@gmail.com>.
Hello Andy,
I have a question.
>If disk wait times exceed 40%, you need more spindles (disk) for your
datanodes or more datanodes (servers) to spread the I/O >load.
Is the "disk wait times exceed 40%" you mentioned means the "wa" value when
running "vmstat"? If not, what does it mean? and how do we monitor it?
Thank you.
Regards,
LvZheng.
2009/11/26 Andrew Purtell <ap...@apache.org>
> Try this: Do not cache on the crawlers, just write through.
>
> Run each region server with plenty of heap (4 GB to start). So it seems you
> need more RAM on your systems, or you should move your crawlers off to
> separate servers to free up RAM and CPU.
>
> Adjust your HBase site config as follows:
>
> <property>
> <name>hbase.hregion.memstore.block.multiplier</name>
> <value>4</value>
> </property>
>
> <property>
> <name>hbase.hstore.blockingStoreFiles</name>
> <value>20</value>
> </property>
>
> <property>
> <name>dfs.replication</name>
> <value>2</value>
> </property>
>
> <property>
> <name>hfile.block.cache.size</name>
> <!-- disable -->
> <value>0</value>
> </property>
>
> Be advised that the probability of OOM under high write load is high if you
> do not provide adequate heap. This may manifest either as OOM exceptions or
> as GC taking so much time as to drop the ZK session. Concurrent garbage
> collector will keep GC latency low but it needs a larger available heap
> space in return. You should also tune your HBASE_OPTS in conf/hbase-env.sh
> according to this advice:
> http://osdir.com/ml/hbase-user-hadoop-apache/2009-10/msg00276.html
>
> With your monitoring tools watch the amount of disk wait time on the OS
> level. If disk wait times exceed 40%, you need more spindles (disk) for your
> datanodes or more datanodes (servers) to spread the I/O load.
>
> - Andy
>
>
>
>
> ________________________________
> From: Zheng Lv <lv...@gmail.com>
> To: hbase-user@hadoop.apache.org
> Sent: Tue, November 24, 2009 8:22:04 PM
> Subject: Re: is there any problem with our environment?
>
> Hello Stack,
> >Sorry for taking time getting back to you Lv.
> Never mind:).
>
> >I posted Andrew's crawling notes yesterday. Did you see them? I thought
> >they might be of use to you.
> You mean you'v sent mail to me? I didn't get it.
>
> >This is interesting. You were just looking at message logs and saw the
> >above?
> We checked /var/log/messages on some servers, and we got many "link down"
> and "link up".
>
> >You don't need 6 zk nodes. Make it 3 (In an earlier mail you said you had
> 3
> >only but above would seem to say you have 6 zk nodes). You could up the
> >ticktime from 2 to 3 seconds (We learned that maximum session time is
> >20*ticktime no matter what session timeout is set to).
> >
> We will change our cluster as you told me.
> >(In an earlier mail you said you had 3 only but above would seem to say
> you
> have 6 zk nodes)
> We added some, we thought 3 is insufficient.
>
> >For 8Gs of RAM and a single disk might not be sufficient to carry all of
> the
> >above daemons on a single node. You have cluster-wide monitoring setup?
> >Are these nodes swapping? From the logs I've seen, HDFS is for sure
> >struggling.
> Our system manager has just deployed mrtg+snmp on those servers, and we
> will
> share the data with you.
>
> >The crawler writes straight into hbase or does it write the
> >local disk?
> Each crawler has a cache in memory, when it is full, the crawler will flush
> webpages in cache to hbase.
> >What is the tasktracker doing? Are mapreduce jobs running
> >concurrently on these nodes?
> Now the tasktracker process is running, but there is not any job on it.
> But,
> when updater, which is another part of our product, is on there will be.
> We will restart hbase0.20.2 with DEBUG enabled and monitor it, I'll be back
> when we get useful data.
> Regards,
> Lv Zheng.
>
>
>
>
>
>
>
> 2009/11/25 stack <st...@duboce.net>
>
> > I took a look at your logs. Please run with DEBUG enabled going forward
> > (if
> > you upgraded to 0.20.2, then it'll be on by default).
> >
> > HDFS is struggling.
> >
> > Looking at this region:
> >
> >
> >
> http://blog.blog.tianya.cn/blogger/post_Date.asp?blogID=1818563&CategoryID=1256944&idWriter=0&Key=0&NextPostID=888888888&PageNo=2
> >
> > ... there seems to be an issue w/ accounting. It looks like it got
> > assigned
> > to 13 before the logs you gave me but then the regionserver its on, while
> > it
> > thinks its carrying it in one regard -- its trying to flush it so it can
> > let
> > go of old WAL files -- it then is telling clients that ask for it that
> its
> > not serving it. Seeing an earlier master and regionserver 13 log would
> > help
> > figure what happened. Try restart on 0.20.2. Tell me more about the
> > loading on these machines, how its done. Do you have monitoring software
> > running?
> >
> > Thanks,
> > St.Ack
> >
> >
> >
> >
> >
> > On Tue, Nov 24, 2009 at 10:30 AM, stack <st...@duboce.net> wrote:
> >
> > > Sorry for taking time getting back to you Lv.
> > >
> > > I posted Andrew's crawling notes yesterday. Did you see them? I
> thought
> > > they might be of use to you.
> > >
> > >
> > > On Sun, Nov 22, 2009 at 10:30 PM, Zheng Lv <lvzheng19800619@gmail.com
> > >wrote:
> > >
> > >> >How did you fix it?
> > >> We found some "link down" and "link up" in some server's syslog, our
> > >> system
> > >> manager suggested to change the network cards on those servers, so we
> > >> changed them, and the errors disappeared.
> > >>
> > >
> > >
> > > This is interesting. You were just looking at message logs and saw the
> > > above?
> > >
> > >
> > >
> > >>
> > >>
> > >> >So figure out whats happening to that region by grepping its name in
> > the
> > >> >master log. Why is it offline so long? Are machines loaded?
> > Swapping?
> > >>
> > >> >Yeah, its split. Thats normal. Whats not normal is the client not
> > >> finding
> > >> >the daughter split in its new location. Did the daughters get
> deployed
> > >> >promptly?
> > >> I'm sorry, I don't know exactly how I can get these infomation from
> the
> > >> log.
> > >> Can you tell me? And I have upload the logs to sky drives
> > >>
> > >>
> >
> http://cid-a331bb289a14fbef.skydrive.live.com/browse.aspx/.Public/1123?uc=1&isFromRichUpload=1&lc=2052
> > >> .
> > >> There are two master logs, which are from different days, and two
> > >> regionserver logs, which are from the first 2 shutdown servers.
> > >>
> > >>
> > > I'm looking at these logs now.
> > >
> > >
> > >
> > >> >
> > >> >Are the crawlers running on same machines as hbase?
> > >> Yes, they are. And the cluster we are using is like that:
> > >> 1 master/namenode/jobtracker/crawler server.
> > >> 6 rs/zk/datanode/tasktracker/crawler client.
> > >>
> > >>
> > > You don't need 6 zk nodes. Make it 3 (In an earlier mail you said you
> > had
> > > 3 only but above would seem to say you have 6 zk nodes). You could up
> > the
> > > ticktime from 2 to 3 seconds (We learned that maximum session time is
> > > 20*ticktime no matter what session timeout is set to).
> > >
> > > For 8Gs of RAM and a single disk might not be sufficient to carry all
> of
> > > the above daemons on a single node. You have cluster-wide monitoring
> > > setup? Are these nodes swapping? From the logs I've seen, HDFS is for
> > sure
> > > struggling. The crawler writes straight into hbase or does it write
> the
> > > local disk? What is the tasktracker doing? Are mapreduce jobs running
> > > concurrently on these nodes?
> > >
> > >
> > >> >Can you update to hbase 0.20.2? It has a bunch of fixes that could
> be
> > >> >related to the above.
> > >> We have updated to hbase 0.20.2 this morning, and we will re-run the
> > >> crawler
> > >> in a minute, and we will share the result with you.
> > >>
> > >> Good stuff.
> > >
> > > I'll be back after some study of your logs.
> > > St.Ack
> > >
> > >
> > >
> > >> Regards,
> > >> LvZheng.
> > >>
> > >>
> > >> 2009/11/21 stack <st...@duboce.net>
> > >>
> > >> > On Fri, Nov 20, 2009 at 12:28 AM, Zheng Lv <
> lvzheng19800619@gmail.com
> > >> > >wrote:
> > >> >
> > >> > > Hello Stack,
> > >> > > Remember the "no route to host" exceptions last time? Now there
> > isn't
> > >> any
> > >> > > more, and the test program can be running for several days.
> > >> >
> > >> >
> > >> > How did you fix it?
> > >> >
> > >> >
> > >> >
> > >> > > Thank you.
> > >> > > Recently we started running our crawling program, which crawls
> > >> webpages
> > >> > and
> > >> > > then insert them to hbase.
> > >> > > But we got so many
> > "org.apache.hadoop.hbase.NotServingRegionException"
> > >> > like
> > >> > > that:
> > >> > >
> > >> > > 2009-11-20 12:36:41,898 ERROR
> > >> > > org.apache.hadoop.hbase.regionserver.HRegionServer:
> > >> > > org.apache.hadoop.hbase.NotServingRegionException: webpage,
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> http://bbs.city.tianya.cn/tianyacity/Content/178/1/536629.shtml,1258691377544
> > >> > >
> > >> >
> > >> > So figure out whats happening to that region by grepping its name in
> > the
> > >> > master log. Why is it offline so long? Are machines loaded?
> > Swapping?
> > >> >
> > >> > Are the crawlers running on same machines as hbase?
> > >> >
> > >> > What crawler are you using?
> > >> >
> > >> > Andrew Purtell has written up some notes on getting a nice balance
> > >> between
> > >> > crawl process and hbase such that all runs smoothly in private
> > >> > correspondence. Let me ask him if its ok to forward the list.
> > >> >
> > >> >
> > >> > ....
> > >> >
> > >> > 2009-11-20 12:36:25,259 INFO
> > >> org.apache.hadoop.hbase.master.ServerManager:
> > >> > > Processing MSG_REPORT_SPLIT:
> > >> > > webpage,http:\x2F\x2Fbbs.city.tianya.cn<http://x2fbbs.city.tianya.cn/>
> <http://x2fbbs.city.tianya.cn/><
> > >> http://x2fbbs.city.tianya.cn/>
> > >> > >
> \x2Ftianyacity\x2FContent\x2F178\x2F1\x2F536629.shtml,1258691377544:
> > >> > > Daughters; webpage,http:\x2F\x2Fbbs.city.tianya.cn<http://x2fbbs.city.tianya.cn/>
> <http://x2fbbs.city.tianya.cn/>
> > <
> > >> http://x2fbbs.city.tianya.cn/>
> > >> > >
> \x2Ftianyacity\x2FContent\x2F178\x2F1\x2F536629.shtml,1258691779496,
> > >> > > webpage,http:\x2F\x2Fbbs.city.tianya.cn<http://x2fbbs.city.tianya.cn/>
> <http://x2fbbs.city.tianya.cn/><
> > >> http://x2fbbs.city.tianya.cn/>
> > >> > >
> \x2Ftianyacity\x2FContent\x2F329\x2F1\x2F164370.shtml,1258691779496
> > >> > > from ubuntu12,60020,1258687326554;
> > >> > >
> > >> > > Yeah, its split. Thats normal. Whats not normal is the client
> not
> > >> > finding
> > >> > the daughter split in its new location. Did the daughters get
> > deployed
> > >> > promptly?
> > >> >
> > >> >
> > >> >
> > >> > > And a few hours later, some rs shutdown.
> > >> > >
> > >> > > I read the mail
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> http://mail-archives.apache.org/mod_mbox/hadoop-hbase-user/200907.mbox/%3C9b27a8a60907272122y1bfa6254n95948942d5ca7f88@mail.gmail.com%3E
> > >> > > ,
> > >> > > which was sent by my partner Angus. In the mail you told us it was
> a
> > >> case
> > >> > > of
> > >> > > "HBASE-1671", Fix Version of which is 0.20.0, but the hbase
> version
> > we
> > >> > are
> > >> > > using is just 0.20.0.
> > >> > >
> > >> >
> > >> > Can you update to hbase 0.20.2? It has a bunch of fixes that could
> be
> > >> > related to the above.
> > >> > Yours,
> > >> > St.Ack
> > >> >
> > >> >
> > >> >
> > >> > > Any idea?
> > >> > > Best Regards,
> > >> > > LvZheng
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > 2009/10/13 stack <st...@duboce.net>
> > >> > >
> > >> > > > Thanks for posting. Its much easier reading the logs from
> there.
> > >> > > >
> > >> > > > Looking in nohup.out I see it can't find region
> > 'webpage,http:\x2F\
> > >> > > > x2Fnews.163.com <http://x2fnews.163.com/> <
> http://x2fnews.163.com/> <
> > http://x2fnews.163.com/> <http://x2fnews.163.com/>
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> \x2F09\x2F080\x2F0\x2F5FOO155J0001124J.html1255072992000_751685,1255316061169'.
> > >> > > > It never finds it. It looks like it was assigned successfully
> to
> > >> > > > 192.168.33.5 going by the master log. Once you've figured out
> the
> > >> > > > hardware/networking issues, lets work at getting that region
> back
> > on
> > >> > > line.
> > >> > > >
> > >> > > > The master timed out its session against zk because of 'no route
> > to
> > >> > > host'.
> > >> > > >
> > >> > > > St.Ack
> > >> > > >
> > >> > > > On Mon, Oct 12, 2009 at 12:23 AM, Zheng Lv <
> > >> lvzheng19800619@gmail.com
> > >> > > > >wrote:
> > >> > > >
> > >> > > > > Hello Stack,
> > >> > > > > I have enabled DEBUG and restarted the test program. This
> > time
> > >> the
> > >> > > > > master shut down, and I have put the logs on skydrive.
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> http://cid-a331bb289a14fbef.skydrive.live.com/browse.aspx/.Public?uc=2&isFromRichUpload=1
> > >> > > > > .
> > >> > > > > "nohup.out" is our test program log,
> > >> > "hbase-cyd-master-ubuntu6.log"
> > >> > > is
> > >> > > > > master log.
> > >> > > > >
> > >> > > > > On the other hand, today we found that when we run "dmesg",
> > >> there
> > >> > > were
> > >> > > > > many logs like "[3641697.122769] r8169: eth0: link down". And
> I
> > >> think
> > >> > > > this
> > >> > > > > might be the reason of so many "no route to host" and "Time
> > Out".
> > >> Now
> > >> > > our
> > >> > > > > system manager is checking, if we have a result we will let
> you
> > >> > know.:)
> > >> > > > > Thanks,
> > >> > > > > LvZheng.
> > >> > > > >
> > >> > > > > 2009/10/11 stack <st...@duboce.net>
> > >> > > > >
> > >> > > > > > On Fri, Oct 9, 2009 at 3:18 AM, Zheng Lv <
> > >> > lvzheng19800619@gmail.com>
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > ...
> > >> > > > > > > so,
> > >> > > > > > > > please remove the delay so hbase fails faster so it
> > >> doesn't
> > >> > > take
> > >> > > > > so
> > >> > > > > > > long to
> > >> > > > > > > > figure the issue.
> > >> > > > > > > > Are you inserting every 10ms because hbase is falling
> > >> over
> > >> > on
> > >> > > > you?
> > >> > > > > > If
> > >> > > > > > > Yes I inserted every 10ms because I'm afraid hbase
> would
> > >> fall
> > >> > > > over.
> > >> > > > > > Now
> > >> > > > > > > I have removed the delay.
> > >> > > > > > >
> > >> > > > > > > After doing these, We have run the test program again,
> > and
> > >> one
> > >> > > > > region
> > >> > > > > > > server shut down after about 2 hours, another one 3.
> > >> > > > > > > I will post the logs on these two servers in following
> > >> reply
> > >> > > > mails.
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > Thanks for doing the above.
> > >> > > > > >
> > >> > > > > > For the future, debugging, please enable DEBUG and put your
> > logs
> > >> > > > > somewhere
> > >> > > > > > where I can pull them or put them up in pastebin. Logs in
> > email
> > >> > > > messages
> > >> > > > > > are hard to follow. Thanks.
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > > > Ok. So this is hbase 0.20.0? Tell us about your
> > >> hardware.
> > >> > > > What
> > >> > > > > > kind
> > >> > > > > > > is
> > >> > > > > > > > it? CPU/RAM/Disks.
> > >> > > > > > > Yes we are using hbase 0.20.0. And the following is
> our
> > >> > > > hardware:
> > >> > > > > > >
> > >> > > > > > > CPU:amd x3 710
> > >> > > > > > > RAM:8g ddr2 800
> > >> > > > > > > Disk:270g(raid0)
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > Thats an interesting chip -- 3 cores! The above should be
> > fine
> > >> as
> > >> > > long
> > >> > > > > as
> > >> > > > > > you coral your mapreduce jobs running on same cluster.
> > >> > > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > > We have 7 servers with above hardware, one for master,
> > >> three
> > >> > for
> > >> > > > > > > namenodes / regionservers, and the other 3 for zks.
> > >> > > > > > > By the way, what kind of hardware and environment do
> you
> > >> > suggest
> > >> > > > we
> > >> > > > > > > have?
> > >> > > > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > This configuration seems fine to start with. Later we might
> > >> > > experiment
> > >> > > > > > running zk on same machines as regionservers and then up
> > number
> > >> of
> > >> > > > > > regionservers to 6 and up the quorum members to 5.
> > >> > > > > >
> > >> > > > > > St.Ack
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > >
> > >> > > > > > > Thank you, very much.
> > >> > > > > > > LvZheng.
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>
>
>
>
>
Re: is there any problem with our environment?
Posted by Andrew Purtell <ap...@apache.org>.
Try this: Do not cache on the crawlers, just write through.
Run each region server with plenty of heap (4 GB to start). So it seems you need more RAM on your systems, or you should move your crawlers off to separate servers to free up RAM and CPU.
Adjust your HBase site config as follows:
<property>
<name>hbase.hregion.memstore.block.multiplier</name>
<value>4</value>
</property>
<property>
<name>hbase.hstore.blockingStoreFiles</name>
<value>20</value>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>hfile.block.cache.size</name>
<!-- disable -->
<value>0</value>
</property>
Be advised that the probability of OOM under high write load is high if you do not provide adequate heap. This may manifest either as OOM exceptions or as GC taking so much time as to drop the ZK session. Concurrent garbage collector will keep GC latency low but it needs a larger available heap space in return. You should also tune your HBASE_OPTS in conf/hbase-env.sh according to this advice: http://osdir.com/ml/hbase-user-hadoop-apache/2009-10/msg00276.html
With your monitoring tools watch the amount of disk wait time on the OS level. If disk wait times exceed 40%, you need more spindles (disk) for your datanodes or more datanodes (servers) to spread the I/O load.
- Andy
________________________________
From: Zheng Lv <lv...@gmail.com>
To: hbase-user@hadoop.apache.org
Sent: Tue, November 24, 2009 8:22:04 PM
Subject: Re: is there any problem with our environment?
Hello Stack,
>Sorry for taking time getting back to you Lv.
Never mind:).
>I posted Andrew's crawling notes yesterday. Did you see them? I thought
>they might be of use to you.
You mean you'v sent mail to me? I didn't get it.
>This is interesting. You were just looking at message logs and saw the
>above?
We checked /var/log/messages on some servers, and we got many "link down"
and "link up".
>You don't need 6 zk nodes. Make it 3 (In an earlier mail you said you had
3
>only but above would seem to say you have 6 zk nodes). You could up the
>ticktime from 2 to 3 seconds (We learned that maximum session time is
>20*ticktime no matter what session timeout is set to).
>
We will change our cluster as you told me.
>(In an earlier mail you said you had 3 only but above would seem to say you
have 6 zk nodes)
We added some, we thought 3 is insufficient.
>For 8Gs of RAM and a single disk might not be sufficient to carry all of
the
>above daemons on a single node. You have cluster-wide monitoring setup?
>Are these nodes swapping? From the logs I've seen, HDFS is for sure
>struggling.
Our system manager has just deployed mrtg+snmp on those servers, and we will
share the data with you.
>The crawler writes straight into hbase or does it write the
>local disk?
Each crawler has a cache in memory, when it is full, the crawler will flush
webpages in cache to hbase.
>What is the tasktracker doing? Are mapreduce jobs running
>concurrently on these nodes?
Now the tasktracker process is running, but there is not any job on it. But,
when updater, which is another part of our product, is on there will be.
We will restart hbase0.20.2 with DEBUG enabled and monitor it, I'll be back
when we get useful data.
Regards,
Lv Zheng.
2009/11/25 stack <st...@duboce.net>
> I took a look at your logs. Please run with DEBUG enabled going forward
> (if
> you upgraded to 0.20.2, then it'll be on by default).
>
> HDFS is struggling.
>
> Looking at this region:
>
>
> http://blog.blog.tianya.cn/blogger/post_Date.asp?blogID=1818563&CategoryID=1256944&idWriter=0&Key=0&NextPostID=888888888&PageNo=2
>
> ... there seems to be an issue w/ accounting. It looks like it got
> assigned
> to 13 before the logs you gave me but then the regionserver its on, while
> it
> thinks its carrying it in one regard -- its trying to flush it so it can
> let
> go of old WAL files -- it then is telling clients that ask for it that its
> not serving it. Seeing an earlier master and regionserver 13 log would
> help
> figure what happened. Try restart on 0.20.2. Tell me more about the
> loading on these machines, how its done. Do you have monitoring software
> running?
>
> Thanks,
> St.Ack
>
>
>
>
>
> On Tue, Nov 24, 2009 at 10:30 AM, stack <st...@duboce.net> wrote:
>
> > Sorry for taking time getting back to you Lv.
> >
> > I posted Andrew's crawling notes yesterday. Did you see them? I thought
> > they might be of use to you.
> >
> >
> > On Sun, Nov 22, 2009 at 10:30 PM, Zheng Lv <lvzheng19800619@gmail.com
> >wrote:
> >
> >> >How did you fix it?
> >> We found some "link down" and "link up" in some server's syslog, our
> >> system
> >> manager suggested to change the network cards on those servers, so we
> >> changed them, and the errors disappeared.
> >>
> >
> >
> > This is interesting. You were just looking at message logs and saw the
> > above?
> >
> >
> >
> >>
> >>
> >> >So figure out whats happening to that region by grepping its name in
> the
> >> >master log. Why is it offline so long? Are machines loaded?
> Swapping?
> >>
> >> >Yeah, its split. Thats normal. Whats not normal is the client not
> >> finding
> >> >the daughter split in its new location. Did the daughters get deployed
> >> >promptly?
> >> I'm sorry, I don't know exactly how I can get these infomation from the
> >> log.
> >> Can you tell me? And I have upload the logs to sky drives
> >>
> >>
> http://cid-a331bb289a14fbef.skydrive.live.com/browse.aspx/.Public/1123?uc=1&isFromRichUpload=1&lc=2052
> >> .
> >> There are two master logs, which are from different days, and two
> >> regionserver logs, which are from the first 2 shutdown servers.
> >>
> >>
> > I'm looking at these logs now.
> >
> >
> >
> >> >
> >> >Are the crawlers running on same machines as hbase?
> >> Yes, they are. And the cluster we are using is like that:
> >> 1 master/namenode/jobtracker/crawler server.
> >> 6 rs/zk/datanode/tasktracker/crawler client.
> >>
> >>
> > You don't need 6 zk nodes. Make it 3 (In an earlier mail you said you
> had
> > 3 only but above would seem to say you have 6 zk nodes). You could up
> the
> > ticktime from 2 to 3 seconds (We learned that maximum session time is
> > 20*ticktime no matter what session timeout is set to).
> >
> > For 8Gs of RAM and a single disk might not be sufficient to carry all of
> > the above daemons on a single node. You have cluster-wide monitoring
> > setup? Are these nodes swapping? From the logs I've seen, HDFS is for
> sure
> > struggling. The crawler writes straight into hbase or does it write the
> > local disk? What is the tasktracker doing? Are mapreduce jobs running
> > concurrently on these nodes?
> >
> >
> >> >Can you update to hbase 0.20.2? It has a bunch of fixes that could be
> >> >related to the above.
> >> We have updated to hbase 0.20.2 this morning, and we will re-run the
> >> crawler
> >> in a minute, and we will share the result with you.
> >>
> >> Good stuff.
> >
> > I'll be back after some study of your logs.
> > St.Ack
> >
> >
> >
> >> Regards,
> >> LvZheng.
> >>
> >>
> >> 2009/11/21 stack <st...@duboce.net>
> >>
> >> > On Fri, Nov 20, 2009 at 12:28 AM, Zheng Lv <lvzheng19800619@gmail.com
> >> > >wrote:
> >> >
> >> > > Hello Stack,
> >> > > Remember the "no route to host" exceptions last time? Now there
> isn't
> >> any
> >> > > more, and the test program can be running for several days.
> >> >
> >> >
> >> > How did you fix it?
> >> >
> >> >
> >> >
> >> > > Thank you.
> >> > > Recently we started running our crawling program, which crawls
> >> webpages
> >> > and
> >> > > then insert them to hbase.
> >> > > But we got so many
> "org.apache.hadoop.hbase.NotServingRegionException"
> >> > like
> >> > > that:
> >> > >
> >> > > 2009-11-20 12:36:41,898 ERROR
> >> > > org.apache.hadoop.hbase.regionserver.HRegionServer:
> >> > > org.apache.hadoop.hbase.NotServingRegionException: webpage,
> >> > >
> >> > >
> >> >
> >>
> http://bbs.city.tianya.cn/tianyacity/Content/178/1/536629.shtml,1258691377544
> >> > >
> >> >
> >> > So figure out whats happening to that region by grepping its name in
> the
> >> > master log. Why is it offline so long? Are machines loaded?
> Swapping?
> >> >
> >> > Are the crawlers running on same machines as hbase?
> >> >
> >> > What crawler are you using?
> >> >
> >> > Andrew Purtell has written up some notes on getting a nice balance
> >> between
> >> > crawl process and hbase such that all runs smoothly in private
> >> > correspondence. Let me ask him if its ok to forward the list.
> >> >
> >> >
> >> > ....
> >> >
> >> > 2009-11-20 12:36:25,259 INFO
> >> org.apache.hadoop.hbase.master.ServerManager:
> >> > > Processing MSG_REPORT_SPLIT:
> >> > > webpage,http:\x2F\x2Fbbs.city.tianya.cn<http://x2fbbs.city.tianya.cn/><
> >> http://x2fbbs.city.tianya.cn/>
> >> > > \x2Ftianyacity\x2FContent\x2F178\x2F1\x2F536629.shtml,1258691377544:
> >> > > Daughters; webpage,http:\x2F\x2Fbbs.city.tianya.cn<http://x2fbbs.city.tianya.cn/>
> <
> >> http://x2fbbs.city.tianya.cn/>
> >> > > \x2Ftianyacity\x2FContent\x2F178\x2F1\x2F536629.shtml,1258691779496,
> >> > > webpage,http:\x2F\x2Fbbs.city.tianya.cn<http://x2fbbs.city.tianya.cn/><
> >> http://x2fbbs.city.tianya.cn/>
> >> > > \x2Ftianyacity\x2FContent\x2F329\x2F1\x2F164370.shtml,1258691779496
> >> > > from ubuntu12,60020,1258687326554;
> >> > >
> >> > > Yeah, its split. Thats normal. Whats not normal is the client not
> >> > finding
> >> > the daughter split in its new location. Did the daughters get
> deployed
> >> > promptly?
> >> >
> >> >
> >> >
> >> > > And a few hours later, some rs shutdown.
> >> > >
> >> > > I read the mail
> >> > >
> >> > >
> >> >
> >>
> http://mail-archives.apache.org/mod_mbox/hadoop-hbase-user/200907.mbox/%3C9b27a8a60907272122y1bfa6254n95948942d5ca7f88@mail.gmail.com%3E
> >> > > ,
> >> > > which was sent by my partner Angus. In the mail you told us it was a
> >> case
> >> > > of
> >> > > "HBASE-1671", Fix Version of which is 0.20.0, but the hbase version
> we
> >> > are
> >> > > using is just 0.20.0.
> >> > >
> >> >
> >> > Can you update to hbase 0.20.2? It has a bunch of fixes that could be
> >> > related to the above.
> >> > Yours,
> >> > St.Ack
> >> >
> >> >
> >> >
> >> > > Any idea?
> >> > > Best Regards,
> >> > > LvZheng
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > 2009/10/13 stack <st...@duboce.net>
> >> > >
> >> > > > Thanks for posting. Its much easier reading the logs from there.
> >> > > >
> >> > > > Looking in nohup.out I see it can't find region
> 'webpage,http:\x2F\
> >> > > > x2Fnews.163.com <http://x2fnews.163.com/> <
> http://x2fnews.163.com/> <http://x2fnews.163.com/>
> >> > > >
> >> > >
> >> >
> >>
> \x2F09\x2F080\x2F0\x2F5FOO155J0001124J.html1255072992000_751685,1255316061169'.
> >> > > > It never finds it. It looks like it was assigned successfully to
> >> > > > 192.168.33.5 going by the master log. Once you've figured out the
> >> > > > hardware/networking issues, lets work at getting that region back
> on
> >> > > line.
> >> > > >
> >> > > > The master timed out its session against zk because of 'no route
> to
> >> > > host'.
> >> > > >
> >> > > > St.Ack
> >> > > >
> >> > > > On Mon, Oct 12, 2009 at 12:23 AM, Zheng Lv <
> >> lvzheng19800619@gmail.com
> >> > > > >wrote:
> >> > > >
> >> > > > > Hello Stack,
> >> > > > > I have enabled DEBUG and restarted the test program. This
> time
> >> the
> >> > > > > master shut down, and I have put the logs on skydrive.
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> http://cid-a331bb289a14fbef.skydrive.live.com/browse.aspx/.Public?uc=2&isFromRichUpload=1
> >> > > > > .
> >> > > > > "nohup.out" is our test program log,
> >> > "hbase-cyd-master-ubuntu6.log"
> >> > > is
> >> > > > > master log.
> >> > > > >
> >> > > > > On the other hand, today we found that when we run "dmesg",
> >> there
> >> > > were
> >> > > > > many logs like "[3641697.122769] r8169: eth0: link down". And I
> >> think
> >> > > > this
> >> > > > > might be the reason of so many "no route to host" and "Time
> Out".
> >> Now
> >> > > our
> >> > > > > system manager is checking, if we have a result we will let you
> >> > know.:)
> >> > > > > Thanks,
> >> > > > > LvZheng.
> >> > > > >
> >> > > > > 2009/10/11 stack <st...@duboce.net>
> >> > > > >
> >> > > > > > On Fri, Oct 9, 2009 at 3:18 AM, Zheng Lv <
> >> > lvzheng19800619@gmail.com>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > ...
> >> > > > > > > so,
> >> > > > > > > > please remove the delay so hbase fails faster so it
> >> doesn't
> >> > > take
> >> > > > > so
> >> > > > > > > long to
> >> > > > > > > > figure the issue.
> >> > > > > > > > Are you inserting every 10ms because hbase is falling
> >> over
> >> > on
> >> > > > you?
> >> > > > > > If
> >> > > > > > > Yes I inserted every 10ms because I'm afraid hbase would
> >> fall
> >> > > > over.
> >> > > > > > Now
> >> > > > > > > I have removed the delay.
> >> > > > > > >
> >> > > > > > > After doing these, We have run the test program again,
> and
> >> one
> >> > > > > region
> >> > > > > > > server shut down after about 2 hours, another one 3.
> >> > > > > > > I will post the logs on these two servers in following
> >> reply
> >> > > > mails.
> >> > > > > > >
> >> > > > > > >
> >> > > > > > Thanks for doing the above.
> >> > > > > >
> >> > > > > > For the future, debugging, please enable DEBUG and put your
> logs
> >> > > > > somewhere
> >> > > > > > where I can pull them or put them up in pastebin. Logs in
> email
> >> > > > messages
> >> > > > > > are hard to follow. Thanks.
> >> > > > > >
> >> > > > > >
> >> > > > > > > > Ok. So this is hbase 0.20.0? Tell us about your
> >> hardware.
> >> > > > What
> >> > > > > > kind
> >> > > > > > > is
> >> > > > > > > > it? CPU/RAM/Disks.
> >> > > > > > > Yes we are using hbase 0.20.0. And the following is our
> >> > > > hardware:
> >> > > > > > >
> >> > > > > > > CPU:amd x3 710
> >> > > > > > > RAM:8g ddr2 800
> >> > > > > > > Disk:270g(raid0)
> >> > > > > > >
> >> > > > > > >
> >> > > > > > Thats an interesting chip -- 3 cores! The above should be
> fine
> >> as
> >> > > long
> >> > > > > as
> >> > > > > > you coral your mapreduce jobs running on same cluster.
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > > > We have 7 servers with above hardware, one for master,
> >> three
> >> > for
> >> > > > > > > namenodes / regionservers, and the other 3 for zks.
> >> > > > > > > By the way, what kind of hardware and environment do you
> >> > suggest
> >> > > > we
> >> > > > > > > have?
> >> > > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > > This configuration seems fine to start with. Later we might
> >> > > experiment
> >> > > > > > running zk on same machines as regionservers and then up
> number
> >> of
> >> > > > > > regionservers to 6 and up the quorum members to 5.
> >> > > > > >
> >> > > > > > St.Ack
> >> > > > > >
> >> > > > > >
> >> > > > > > >
> >> > > > > > > Thank you, very much.
> >> > > > > > > LvZheng.
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>
Re: is there any problem with our environment?
Posted by Zheng Lv <lv...@gmail.com>.
Hello Stack,
>Sorry for taking time getting back to you Lv.
Never mind:).
>I posted Andrew's crawling notes yesterday. Did you see them? I thought
>they might be of use to you.
You mean you'v sent mail to me? I didn't get it.
>This is interesting. You were just looking at message logs and saw the
>above?
We checked /var/log/messages on some servers, and we got many "link down"
and "link up".
>You don't need 6 zk nodes. Make it 3 (In an earlier mail you said you had
3
>only but above would seem to say you have 6 zk nodes). You could up the
>ticktime from 2 to 3 seconds (We learned that maximum session time is
>20*ticktime no matter what session timeout is set to).
>
We will change our cluster as you told me.
>(In an earlier mail you said you had 3 only but above would seem to say you
have 6 zk nodes)
We added some, we thought 3 is insufficient.
>For 8Gs of RAM and a single disk might not be sufficient to carry all of
the
>above daemons on a single node. You have cluster-wide monitoring setup?
>Are these nodes swapping? From the logs I've seen, HDFS is for sure
>struggling.
Our system manager has just deployed mrtg+snmp on those servers, and we will
share the data with you.
>The crawler writes straight into hbase or does it write the
>local disk?
Each crawler has a cache in memory, when it is full, the crawler will flush
webpages in cache to hbase.
>What is the tasktracker doing? Are mapreduce jobs running
>concurrently on these nodes?
Now the tasktracker process is running, but there is not any job on it. But,
when updater, which is another part of our product, is on there will be.
We will restart hbase0.20.2 with DEBUG enabled and monitor it, I'll be back
when we get useful data.
Regards,
Lv Zheng.
2009/11/25 stack <st...@duboce.net>
> I took a look at your logs. Please run with DEBUG enabled going forward
> (if
> you upgraded to 0.20.2, then it'll be on by default).
>
> HDFS is struggling.
>
> Looking at this region:
>
>
> http://blog.blog.tianya.cn/blogger/post_Date.asp?blogID=1818563&CategoryID=1256944&idWriter=0&Key=0&NextPostID=888888888&PageNo=2
>
> ... there seems to be an issue w/ accounting. It looks like it got
> assigned
> to 13 before the logs you gave me but then the regionserver its on, while
> it
> thinks its carrying it in one regard -- its trying to flush it so it can
> let
> go of old WAL files -- it then is telling clients that ask for it that its
> not serving it. Seeing an earlier master and regionserver 13 log would
> help
> figure what happened. Try restart on 0.20.2. Tell me more about the
> loading on these machines, how its done. Do you have monitoring software
> running?
>
> Thanks,
> St.Ack
>
>
>
>
>
> On Tue, Nov 24, 2009 at 10:30 AM, stack <st...@duboce.net> wrote:
>
> > Sorry for taking time getting back to you Lv.
> >
> > I posted Andrew's crawling notes yesterday. Did you see them? I thought
> > they might be of use to you.
> >
> >
> > On Sun, Nov 22, 2009 at 10:30 PM, Zheng Lv <lvzheng19800619@gmail.com
> >wrote:
> >
> >> >How did you fix it?
> >> We found some "link down" and "link up" in some server's syslog, our
> >> system
> >> manager suggested to change the network cards on those servers, so we
> >> changed them, and the errors disappeared.
> >>
> >
> >
> > This is interesting. You were just looking at message logs and saw the
> > above?
> >
> >
> >
> >>
> >>
> >> >So figure out whats happening to that region by grepping its name in
> the
> >> >master log. Why is it offline so long? Are machines loaded?
> Swapping?
> >>
> >> >Yeah, its split. Thats normal. Whats not normal is the client not
> >> finding
> >> >the daughter split in its new location. Did the daughters get deployed
> >> >promptly?
> >> I'm sorry, I don't know exactly how I can get these infomation from the
> >> log.
> >> Can you tell me? And I have upload the logs to sky drives
> >>
> >>
> http://cid-a331bb289a14fbef.skydrive.live.com/browse.aspx/.Public/1123?uc=1&isFromRichUpload=1&lc=2052
> >> .
> >> There are two master logs, which are from different days, and two
> >> regionserver logs, which are from the first 2 shutdown servers.
> >>
> >>
> > I'm looking at these logs now.
> >
> >
> >
> >> >
> >> >Are the crawlers running on same machines as hbase?
> >> Yes, they are. And the cluster we are using is like that:
> >> 1 master/namenode/jobtracker/crawler server.
> >> 6 rs/zk/datanode/tasktracker/crawler client.
> >>
> >>
> > You don't need 6 zk nodes. Make it 3 (In an earlier mail you said you
> had
> > 3 only but above would seem to say you have 6 zk nodes). You could up
> the
> > ticktime from 2 to 3 seconds (We learned that maximum session time is
> > 20*ticktime no matter what session timeout is set to).
> >
> > For 8Gs of RAM and a single disk might not be sufficient to carry all of
> > the above daemons on a single node. You have cluster-wide monitoring
> > setup? Are these nodes swapping? From the logs I've seen, HDFS is for
> sure
> > struggling. The crawler writes straight into hbase or does it write the
> > local disk? What is the tasktracker doing? Are mapreduce jobs running
> > concurrently on these nodes?
> >
> >
> >> >Can you update to hbase 0.20.2? It has a bunch of fixes that could be
> >> >related to the above.
> >> We have updated to hbase 0.20.2 this morning, and we will re-run the
> >> crawler
> >> in a minute, and we will share the result with you.
> >>
> >> Good stuff.
> >
> > I'll be back after some study of your logs.
> > St.Ack
> >
> >
> >
> >> Regards,
> >> LvZheng.
> >>
> >>
> >> 2009/11/21 stack <st...@duboce.net>
> >>
> >> > On Fri, Nov 20, 2009 at 12:28 AM, Zheng Lv <lvzheng19800619@gmail.com
> >> > >wrote:
> >> >
> >> > > Hello Stack,
> >> > > Remember the "no route to host" exceptions last time? Now there
> isn't
> >> any
> >> > > more, and the test program can be running for several days.
> >> >
> >> >
> >> > How did you fix it?
> >> >
> >> >
> >> >
> >> > > Thank you.
> >> > > Recently we started running our crawling program, which crawls
> >> webpages
> >> > and
> >> > > then insert them to hbase.
> >> > > But we got so many
> "org.apache.hadoop.hbase.NotServingRegionException"
> >> > like
> >> > > that:
> >> > >
> >> > > 2009-11-20 12:36:41,898 ERROR
> >> > > org.apache.hadoop.hbase.regionserver.HRegionServer:
> >> > > org.apache.hadoop.hbase.NotServingRegionException: webpage,
> >> > >
> >> > >
> >> >
> >>
> http://bbs.city.tianya.cn/tianyacity/Content/178/1/536629.shtml,1258691377544
> >> > >
> >> >
> >> > So figure out whats happening to that region by grepping its name in
> the
> >> > master log. Why is it offline so long? Are machines loaded?
> Swapping?
> >> >
> >> > Are the crawlers running on same machines as hbase?
> >> >
> >> > What crawler are you using?
> >> >
> >> > Andrew Purtell has written up some notes on getting a nice balance
> >> between
> >> > crawl process and hbase such that all runs smoothly in private
> >> > correspondence. Let me ask him if its ok to forward the list.
> >> >
> >> >
> >> > ....
> >> >
> >> > 2009-11-20 12:36:25,259 INFO
> >> org.apache.hadoop.hbase.master.ServerManager:
> >> > > Processing MSG_REPORT_SPLIT:
> >> > > webpage,http:\x2F\x2Fbbs.city.tianya.cn<http://x2fbbs.city.tianya.cn/><
> >> http://x2fbbs.city.tianya.cn/>
> >> > > \x2Ftianyacity\x2FContent\x2F178\x2F1\x2F536629.shtml,1258691377544:
> >> > > Daughters; webpage,http:\x2F\x2Fbbs.city.tianya.cn<http://x2fbbs.city.tianya.cn/>
> <
> >> http://x2fbbs.city.tianya.cn/>
> >> > > \x2Ftianyacity\x2FContent\x2F178\x2F1\x2F536629.shtml,1258691779496,
> >> > > webpage,http:\x2F\x2Fbbs.city.tianya.cn<http://x2fbbs.city.tianya.cn/><
> >> http://x2fbbs.city.tianya.cn/>
> >> > > \x2Ftianyacity\x2FContent\x2F329\x2F1\x2F164370.shtml,1258691779496
> >> > > from ubuntu12,60020,1258687326554;
> >> > >
> >> > > Yeah, its split. Thats normal. Whats not normal is the client not
> >> > finding
> >> > the daughter split in its new location. Did the daughters get
> deployed
> >> > promptly?
> >> >
> >> >
> >> >
> >> > > And a few hours later, some rs shutdown.
> >> > >
> >> > > I read the mail
> >> > >
> >> > >
> >> >
> >>
> http://mail-archives.apache.org/mod_mbox/hadoop-hbase-user/200907.mbox/%3C9b27a8a60907272122y1bfa6254n95948942d5ca7f88@mail.gmail.com%3E
> >> > > ,
> >> > > which was sent by my partner Angus. In the mail you told us it was a
> >> case
> >> > > of
> >> > > "HBASE-1671", Fix Version of which is 0.20.0, but the hbase version
> we
> >> > are
> >> > > using is just 0.20.0.
> >> > >
> >> >
> >> > Can you update to hbase 0.20.2? It has a bunch of fixes that could be
> >> > related to the above.
> >> > Yours,
> >> > St.Ack
> >> >
> >> >
> >> >
> >> > > Any idea?
> >> > > Best Regards,
> >> > > LvZheng
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > 2009/10/13 stack <st...@duboce.net>
> >> > >
> >> > > > Thanks for posting. Its much easier reading the logs from there.
> >> > > >
> >> > > > Looking in nohup.out I see it can't find region
> 'webpage,http:\x2F\
> >> > > > x2Fnews.163.com <http://x2fnews.163.com/> <
> http://x2fnews.163.com/> <http://x2fnews.163.com/>
> >> > > >
> >> > >
> >> >
> >>
> \x2F09\x2F080\x2F0\x2F5FOO155J0001124J.html1255072992000_751685,1255316061169'.
> >> > > > It never finds it. It looks like it was assigned successfully to
> >> > > > 192.168.33.5 going by the master log. Once you've figured out the
> >> > > > hardware/networking issues, lets work at getting that region back
> on
> >> > > line.
> >> > > >
> >> > > > The master timed out its session against zk because of 'no route
> to
> >> > > host'.
> >> > > >
> >> > > > St.Ack
> >> > > >
> >> > > > On Mon, Oct 12, 2009 at 12:23 AM, Zheng Lv <
> >> lvzheng19800619@gmail.com
> >> > > > >wrote:
> >> > > >
> >> > > > > Hello Stack,
> >> > > > > I have enabled DEBUG and restarted the test program. This
> time
> >> the
> >> > > > > master shut down, and I have put the logs on skydrive.
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> http://cid-a331bb289a14fbef.skydrive.live.com/browse.aspx/.Public?uc=2&isFromRichUpload=1
> >> > > > > .
> >> > > > > "nohup.out" is our test program log,
> >> > "hbase-cyd-master-ubuntu6.log"
> >> > > is
> >> > > > > master log.
> >> > > > >
> >> > > > > On the other hand, today we found that when we run "dmesg",
> >> there
> >> > > were
> >> > > > > many logs like "[3641697.122769] r8169: eth0: link down". And I
> >> think
> >> > > > this
> >> > > > > might be the reason of so many "no route to host" and "Time
> Out".
> >> Now
> >> > > our
> >> > > > > system manager is checking, if we have a result we will let you
> >> > know.:)
> >> > > > > Thanks,
> >> > > > > LvZheng.
> >> > > > >
> >> > > > > 2009/10/11 stack <st...@duboce.net>
> >> > > > >
> >> > > > > > On Fri, Oct 9, 2009 at 3:18 AM, Zheng Lv <
> >> > lvzheng19800619@gmail.com>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > ...
> >> > > > > > > so,
> >> > > > > > > > please remove the delay so hbase fails faster so it
> >> doesn't
> >> > > take
> >> > > > > so
> >> > > > > > > long to
> >> > > > > > > > figure the issue.
> >> > > > > > > > Are you inserting every 10ms because hbase is falling
> >> over
> >> > on
> >> > > > you?
> >> > > > > > If
> >> > > > > > > Yes I inserted every 10ms because I'm afraid hbase would
> >> fall
> >> > > > over.
> >> > > > > > Now
> >> > > > > > > I have removed the delay.
> >> > > > > > >
> >> > > > > > > After doing these, We have run the test program again,
> and
> >> one
> >> > > > > region
> >> > > > > > > server shut down after about 2 hours, another one 3.
> >> > > > > > > I will post the logs on these two servers in following
> >> reply
> >> > > > mails.
> >> > > > > > >
> >> > > > > > >
> >> > > > > > Thanks for doing the above.
> >> > > > > >
> >> > > > > > For the future, debugging, please enable DEBUG and put your
> logs
> >> > > > > somewhere
> >> > > > > > where I can pull them or put them up in pastebin. Logs in
> email
> >> > > > messages
> >> > > > > > are hard to follow. Thanks.
> >> > > > > >
> >> > > > > >
> >> > > > > > > > Ok. So this is hbase 0.20.0? Tell us about your
> >> hardware.
> >> > > > What
> >> > > > > > kind
> >> > > > > > > is
> >> > > > > > > > it? CPU/RAM/Disks.
> >> > > > > > > Yes we are using hbase 0.20.0. And the following is our
> >> > > > hardware:
> >> > > > > > >
> >> > > > > > > CPU:amd x3 710
> >> > > > > > > RAM:8g ddr2 800
> >> > > > > > > Disk:270g(raid0)
> >> > > > > > >
> >> > > > > > >
> >> > > > > > Thats an interesting chip -- 3 cores! The above should be
> fine
> >> as
> >> > > long
> >> > > > > as
> >> > > > > > you coral your mapreduce jobs running on same cluster.
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > > > We have 7 servers with above hardware, one for master,
> >> three
> >> > for
> >> > > > > > > namenodes / regionservers, and the other 3 for zks.
> >> > > > > > > By the way, what kind of hardware and environment do you
> >> > suggest
> >> > > > we
> >> > > > > > > have?
> >> > > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > > This configuration seems fine to start with. Later we might
> >> > > experiment
> >> > > > > > running zk on same machines as regionservers and then up
> number
> >> of
> >> > > > > > regionservers to 6 and up the quorum members to 5.
> >> > > > > >
> >> > > > > > St.Ack
> >> > > > > >
> >> > > > > >
> >> > > > > > >
> >> > > > > > > Thank you, very much.
> >> > > > > > > LvZheng.
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>
Re: is there any problem with our environment?
Posted by stack <st...@duboce.net>.
I took a look at your logs. Please run with DEBUG enabled going forward (if
you upgraded to 0.20.2, then it'll be on by default).
HDFS is struggling.
Looking at this region:
http://blog.blog.tianya.cn/blogger/post_Date.asp?blogID=1818563&CategoryID=1256944&idWriter=0&Key=0&NextPostID=888888888&PageNo=2
... there seems to be an issue w/ accounting. It looks like it got assigned
to 13 before the logs you gave me but then the regionserver its on, while it
thinks its carrying it in one regard -- its trying to flush it so it can let
go of old WAL files -- it then is telling clients that ask for it that its
not serving it. Seeing an earlier master and regionserver 13 log would help
figure what happened. Try restart on 0.20.2. Tell me more about the
loading on these machines, how its done. Do you have monitoring software
running?
Thanks,
St.Ack
On Tue, Nov 24, 2009 at 10:30 AM, stack <st...@duboce.net> wrote:
> Sorry for taking time getting back to you Lv.
>
> I posted Andrew's crawling notes yesterday. Did you see them? I thought
> they might be of use to you.
>
>
> On Sun, Nov 22, 2009 at 10:30 PM, Zheng Lv <lv...@gmail.com>wrote:
>
>> >How did you fix it?
>> We found some "link down" and "link up" in some server's syslog, our
>> system
>> manager suggested to change the network cards on those servers, so we
>> changed them, and the errors disappeared.
>>
>
>
> This is interesting. You were just looking at message logs and saw the
> above?
>
>
>
>>
>>
>> >So figure out whats happening to that region by grepping its name in the
>> >master log. Why is it offline so long? Are machines loaded? Swapping?
>>
>> >Yeah, its split. Thats normal. Whats not normal is the client not
>> finding
>> >the daughter split in its new location. Did the daughters get deployed
>> >promptly?
>> I'm sorry, I don't know exactly how I can get these infomation from the
>> log.
>> Can you tell me? And I have upload the logs to sky drives
>>
>> http://cid-a331bb289a14fbef.skydrive.live.com/browse.aspx/.Public/1123?uc=1&isFromRichUpload=1&lc=2052
>> .
>> There are two master logs, which are from different days, and two
>> regionserver logs, which are from the first 2 shutdown servers.
>>
>>
> I'm looking at these logs now.
>
>
>
>> >
>> >Are the crawlers running on same machines as hbase?
>> Yes, they are. And the cluster we are using is like that:
>> 1 master/namenode/jobtracker/crawler server.
>> 6 rs/zk/datanode/tasktracker/crawler client.
>>
>>
> You don't need 6 zk nodes. Make it 3 (In an earlier mail you said you had
> 3 only but above would seem to say you have 6 zk nodes). You could up the
> ticktime from 2 to 3 seconds (We learned that maximum session time is
> 20*ticktime no matter what session timeout is set to).
>
> For 8Gs of RAM and a single disk might not be sufficient to carry all of
> the above daemons on a single node. You have cluster-wide monitoring
> setup? Are these nodes swapping? From the logs I've seen, HDFS is for sure
> struggling. The crawler writes straight into hbase or does it write the
> local disk? What is the tasktracker doing? Are mapreduce jobs running
> concurrently on these nodes?
>
>
>> >Can you update to hbase 0.20.2? It has a bunch of fixes that could be
>> >related to the above.
>> We have updated to hbase 0.20.2 this morning, and we will re-run the
>> crawler
>> in a minute, and we will share the result with you.
>>
>> Good stuff.
>
> I'll be back after some study of your logs.
> St.Ack
>
>
>
>> Regards,
>> LvZheng.
>>
>>
>> 2009/11/21 stack <st...@duboce.net>
>>
>> > On Fri, Nov 20, 2009 at 12:28 AM, Zheng Lv <lvzheng19800619@gmail.com
>> > >wrote:
>> >
>> > > Hello Stack,
>> > > Remember the "no route to host" exceptions last time? Now there isn't
>> any
>> > > more, and the test program can be running for several days.
>> >
>> >
>> > How did you fix it?
>> >
>> >
>> >
>> > > Thank you.
>> > > Recently we started running our crawling program, which crawls
>> webpages
>> > and
>> > > then insert them to hbase.
>> > > But we got so many "org.apache.hadoop.hbase.NotServingRegionException"
>> > like
>> > > that:
>> > >
>> > > 2009-11-20 12:36:41,898 ERROR
>> > > org.apache.hadoop.hbase.regionserver.HRegionServer:
>> > > org.apache.hadoop.hbase.NotServingRegionException: webpage,
>> > >
>> > >
>> >
>> http://bbs.city.tianya.cn/tianyacity/Content/178/1/536629.shtml,1258691377544
>> > >
>> >
>> > So figure out whats happening to that region by grepping its name in the
>> > master log. Why is it offline so long? Are machines loaded? Swapping?
>> >
>> > Are the crawlers running on same machines as hbase?
>> >
>> > What crawler are you using?
>> >
>> > Andrew Purtell has written up some notes on getting a nice balance
>> between
>> > crawl process and hbase such that all runs smoothly in private
>> > correspondence. Let me ask him if its ok to forward the list.
>> >
>> >
>> > ....
>> >
>> > 2009-11-20 12:36:25,259 INFO
>> org.apache.hadoop.hbase.master.ServerManager:
>> > > Processing MSG_REPORT_SPLIT:
>> > > webpage,http:\x2F\x2Fbbs.city.tianya.cn <
>> http://x2fbbs.city.tianya.cn/>
>> > > \x2Ftianyacity\x2FContent\x2F178\x2F1\x2F536629.shtml,1258691377544:
>> > > Daughters; webpage,http:\x2F\x2Fbbs.city.tianya.cn<
>> http://x2fbbs.city.tianya.cn/>
>> > > \x2Ftianyacity\x2FContent\x2F178\x2F1\x2F536629.shtml,1258691779496,
>> > > webpage,http:\x2F\x2Fbbs.city.tianya.cn <
>> http://x2fbbs.city.tianya.cn/>
>> > > \x2Ftianyacity\x2FContent\x2F329\x2F1\x2F164370.shtml,1258691779496
>> > > from ubuntu12,60020,1258687326554;
>> > >
>> > > Yeah, its split. Thats normal. Whats not normal is the client not
>> > finding
>> > the daughter split in its new location. Did the daughters get deployed
>> > promptly?
>> >
>> >
>> >
>> > > And a few hours later, some rs shutdown.
>> > >
>> > > I read the mail
>> > >
>> > >
>> >
>> http://mail-archives.apache.org/mod_mbox/hadoop-hbase-user/200907.mbox/%3C9b27a8a60907272122y1bfa6254n95948942d5ca7f88@mail.gmail.com%3E
>> > > ,
>> > > which was sent by my partner Angus. In the mail you told us it was a
>> case
>> > > of
>> > > "HBASE-1671", Fix Version of which is 0.20.0, but the hbase version we
>> > are
>> > > using is just 0.20.0.
>> > >
>> >
>> > Can you update to hbase 0.20.2? It has a bunch of fixes that could be
>> > related to the above.
>> > Yours,
>> > St.Ack
>> >
>> >
>> >
>> > > Any idea?
>> > > Best Regards,
>> > > LvZheng
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > 2009/10/13 stack <st...@duboce.net>
>> > >
>> > > > Thanks for posting. Its much easier reading the logs from there.
>> > > >
>> > > > Looking in nohup.out I see it can't find region 'webpage,http:\x2F\
>> > > > x2Fnews.163.com <http://x2fnews.163.com/> <http://x2fnews.163.com/>
>> > > >
>> > >
>> >
>> \x2F09\x2F080\x2F0\x2F5FOO155J0001124J.html1255072992000_751685,1255316061169'.
>> > > > It never finds it. It looks like it was assigned successfully to
>> > > > 192.168.33.5 going by the master log. Once you've figured out the
>> > > > hardware/networking issues, lets work at getting that region back on
>> > > line.
>> > > >
>> > > > The master timed out its session against zk because of 'no route to
>> > > host'.
>> > > >
>> > > > St.Ack
>> > > >
>> > > > On Mon, Oct 12, 2009 at 12:23 AM, Zheng Lv <
>> lvzheng19800619@gmail.com
>> > > > >wrote:
>> > > >
>> > > > > Hello Stack,
>> > > > > I have enabled DEBUG and restarted the test program. This time
>> the
>> > > > > master shut down, and I have put the logs on skydrive.
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> http://cid-a331bb289a14fbef.skydrive.live.com/browse.aspx/.Public?uc=2&isFromRichUpload=1
>> > > > > .
>> > > > > "nohup.out" is our test program log,
>> > "hbase-cyd-master-ubuntu6.log"
>> > > is
>> > > > > master log.
>> > > > >
>> > > > > On the other hand, today we found that when we run "dmesg",
>> there
>> > > were
>> > > > > many logs like "[3641697.122769] r8169: eth0: link down". And I
>> think
>> > > > this
>> > > > > might be the reason of so many "no route to host" and "Time Out".
>> Now
>> > > our
>> > > > > system manager is checking, if we have a result we will let you
>> > know.:)
>> > > > > Thanks,
>> > > > > LvZheng.
>> > > > >
>> > > > > 2009/10/11 stack <st...@duboce.net>
>> > > > >
>> > > > > > On Fri, Oct 9, 2009 at 3:18 AM, Zheng Lv <
>> > lvzheng19800619@gmail.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > ...
>> > > > > > > so,
>> > > > > > > > please remove the delay so hbase fails faster so it
>> doesn't
>> > > take
>> > > > > so
>> > > > > > > long to
>> > > > > > > > figure the issue.
>> > > > > > > > Are you inserting every 10ms because hbase is falling
>> over
>> > on
>> > > > you?
>> > > > > > If
>> > > > > > > Yes I inserted every 10ms because I'm afraid hbase would
>> fall
>> > > > over.
>> > > > > > Now
>> > > > > > > I have removed the delay.
>> > > > > > >
>> > > > > > > After doing these, We have run the test program again, and
>> one
>> > > > > region
>> > > > > > > server shut down after about 2 hours, another one 3.
>> > > > > > > I will post the logs on these two servers in following
>> reply
>> > > > mails.
>> > > > > > >
>> > > > > > >
>> > > > > > Thanks for doing the above.
>> > > > > >
>> > > > > > For the future, debugging, please enable DEBUG and put your logs
>> > > > > somewhere
>> > > > > > where I can pull them or put them up in pastebin. Logs in email
>> > > > messages
>> > > > > > are hard to follow. Thanks.
>> > > > > >
>> > > > > >
>> > > > > > > > Ok. So this is hbase 0.20.0? Tell us about your
>> hardware.
>> > > > What
>> > > > > > kind
>> > > > > > > is
>> > > > > > > > it? CPU/RAM/Disks.
>> > > > > > > Yes we are using hbase 0.20.0. And the following is our
>> > > > hardware:
>> > > > > > >
>> > > > > > > CPU:amd x3 710
>> > > > > > > RAM:8g ddr2 800
>> > > > > > > Disk:270g(raid0)
>> > > > > > >
>> > > > > > >
>> > > > > > Thats an interesting chip -- 3 cores! The above should be fine
>> as
>> > > long
>> > > > > as
>> > > > > > you coral your mapreduce jobs running on same cluster.
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > > We have 7 servers with above hardware, one for master,
>> three
>> > for
>> > > > > > > namenodes / regionservers, and the other 3 for zks.
>> > > > > > > By the way, what kind of hardware and environment do you
>> > suggest
>> > > > we
>> > > > > > > have?
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > > > This configuration seems fine to start with. Later we might
>> > > experiment
>> > > > > > running zk on same machines as regionservers and then up number
>> of
>> > > > > > regionservers to 6 and up the quorum members to 5.
>> > > > > >
>> > > > > > St.Ack
>> > > > > >
>> > > > > >
>> > > > > > >
>> > > > > > > Thank you, very much.
>> > > > > > > LvZheng.
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>
Re: is there any problem with our environment?
Posted by stack <st...@duboce.net>.
Sorry for taking time getting back to you Lv.
I posted Andrew's crawling notes yesterday. Did you see them? I thought
they might be of use to you.
On Sun, Nov 22, 2009 at 10:30 PM, Zheng Lv <lv...@gmail.com>wrote:
> >How did you fix it?
> We found some "link down" and "link up" in some server's syslog, our system
> manager suggested to change the network cards on those servers, so we
> changed them, and the errors disappeared.
>
This is interesting. You were just looking at message logs and saw the
above?
>
>
> >So figure out whats happening to that region by grepping its name in the
> >master log. Why is it offline so long? Are machines loaded? Swapping?
>
> >Yeah, its split. Thats normal. Whats not normal is the client not
> finding
> >the daughter split in its new location. Did the daughters get deployed
> >promptly?
> I'm sorry, I don't know exactly how I can get these infomation from the
> log.
> Can you tell me? And I have upload the logs to sky drives
>
> http://cid-a331bb289a14fbef.skydrive.live.com/browse.aspx/.Public/1123?uc=1&isFromRichUpload=1&lc=2052
> .
> There are two master logs, which are from different days, and two
> regionserver logs, which are from the first 2 shutdown servers.
>
>
I'm looking at these logs now.
> >
> >Are the crawlers running on same machines as hbase?
> Yes, they are. And the cluster we are using is like that:
> 1 master/namenode/jobtracker/crawler server.
> 6 rs/zk/datanode/tasktracker/crawler client.
>
>
You don't need 6 zk nodes. Make it 3 (In an earlier mail you said you had 3
only but above would seem to say you have 6 zk nodes). You could up the
ticktime from 2 to 3 seconds (We learned that maximum session time is
20*ticktime no matter what session timeout is set to).
For 8Gs of RAM and a single disk might not be sufficient to carry all of the
above daemons on a single node. You have cluster-wide monitoring setup?
Are these nodes swapping? From the logs I've seen, HDFS is for sure
struggling. The crawler writes straight into hbase or does it write the
local disk? What is the tasktracker doing? Are mapreduce jobs running
concurrently on these nodes?
> >Can you update to hbase 0.20.2? It has a bunch of fixes that could be
> >related to the above.
> We have updated to hbase 0.20.2 this morning, and we will re-run the
> crawler
> in a minute, and we will share the result with you.
>
> Good stuff.
I'll be back after some study of your logs.
St.Ack
> Regards,
> LvZheng.
>
>
> 2009/11/21 stack <st...@duboce.net>
>
> > On Fri, Nov 20, 2009 at 12:28 AM, Zheng Lv <lvzheng19800619@gmail.com
> > >wrote:
> >
> > > Hello Stack,
> > > Remember the "no route to host" exceptions last time? Now there isn't
> any
> > > more, and the test program can be running for several days.
> >
> >
> > How did you fix it?
> >
> >
> >
> > > Thank you.
> > > Recently we started running our crawling program, which crawls webpages
> > and
> > > then insert them to hbase.
> > > But we got so many "org.apache.hadoop.hbase.NotServingRegionException"
> > like
> > > that:
> > >
> > > 2009-11-20 12:36:41,898 ERROR
> > > org.apache.hadoop.hbase.regionserver.HRegionServer:
> > > org.apache.hadoop.hbase.NotServingRegionException: webpage,
> > >
> > >
> >
> http://bbs.city.tianya.cn/tianyacity/Content/178/1/536629.shtml,1258691377544
> > >
> >
> > So figure out whats happening to that region by grepping its name in the
> > master log. Why is it offline so long? Are machines loaded? Swapping?
> >
> > Are the crawlers running on same machines as hbase?
> >
> > What crawler are you using?
> >
> > Andrew Purtell has written up some notes on getting a nice balance
> between
> > crawl process and hbase such that all runs smoothly in private
> > correspondence. Let me ask him if its ok to forward the list.
> >
> >
> > ....
> >
> > 2009-11-20 12:36:25,259 INFO
> org.apache.hadoop.hbase.master.ServerManager:
> > > Processing MSG_REPORT_SPLIT:
> > > webpage,http:\x2F\x2Fbbs.city.tianya.cn <http://x2fbbs.city.tianya.cn/
> >
> > > \x2Ftianyacity\x2FContent\x2F178\x2F1\x2F536629.shtml,1258691377544:
> > > Daughters; webpage,http:\x2F\x2Fbbs.city.tianya.cn<
> http://x2fbbs.city.tianya.cn/>
> > > \x2Ftianyacity\x2FContent\x2F178\x2F1\x2F536629.shtml,1258691779496,
> > > webpage,http:\x2F\x2Fbbs.city.tianya.cn <http://x2fbbs.city.tianya.cn/
> >
> > > \x2Ftianyacity\x2FContent\x2F329\x2F1\x2F164370.shtml,1258691779496
> > > from ubuntu12,60020,1258687326554;
> > >
> > > Yeah, its split. Thats normal. Whats not normal is the client not
> > finding
> > the daughter split in its new location. Did the daughters get deployed
> > promptly?
> >
> >
> >
> > > And a few hours later, some rs shutdown.
> > >
> > > I read the mail
> > >
> > >
> >
> http://mail-archives.apache.org/mod_mbox/hadoop-hbase-user/200907.mbox/%3C9b27a8a60907272122y1bfa6254n95948942d5ca7f88@mail.gmail.com%3E
> > > ,
> > > which was sent by my partner Angus. In the mail you told us it was a
> case
> > > of
> > > "HBASE-1671", Fix Version of which is 0.20.0, but the hbase version we
> > are
> > > using is just 0.20.0.
> > >
> >
> > Can you update to hbase 0.20.2? It has a bunch of fixes that could be
> > related to the above.
> > Yours,
> > St.Ack
> >
> >
> >
> > > Any idea?
> > > Best Regards,
> > > LvZheng
> > >
> > >
> > >
> > >
> > >
> > > 2009/10/13 stack <st...@duboce.net>
> > >
> > > > Thanks for posting. Its much easier reading the logs from there.
> > > >
> > > > Looking in nohup.out I see it can't find region 'webpage,http:\x2F\
> > > > x2Fnews.163.com <http://x2fnews.163.com/> <http://x2fnews.163.com/>
> > > >
> > >
> >
> \x2F09\x2F080\x2F0\x2F5FOO155J0001124J.html1255072992000_751685,1255316061169'.
> > > > It never finds it. It looks like it was assigned successfully to
> > > > 192.168.33.5 going by the master log. Once you've figured out the
> > > > hardware/networking issues, lets work at getting that region back on
> > > line.
> > > >
> > > > The master timed out its session against zk because of 'no route to
> > > host'.
> > > >
> > > > St.Ack
> > > >
> > > > On Mon, Oct 12, 2009 at 12:23 AM, Zheng Lv <
> lvzheng19800619@gmail.com
> > > > >wrote:
> > > >
> > > > > Hello Stack,
> > > > > I have enabled DEBUG and restarted the test program. This time
> the
> > > > > master shut down, and I have put the logs on skydrive.
> > > > >
> > > > >
> > > >
> > >
> >
> http://cid-a331bb289a14fbef.skydrive.live.com/browse.aspx/.Public?uc=2&isFromRichUpload=1
> > > > > .
> > > > > "nohup.out" is our test program log,
> > "hbase-cyd-master-ubuntu6.log"
> > > is
> > > > > master log.
> > > > >
> > > > > On the other hand, today we found that when we run "dmesg",
> there
> > > were
> > > > > many logs like "[3641697.122769] r8169: eth0: link down". And I
> think
> > > > this
> > > > > might be the reason of so many "no route to host" and "Time Out".
> Now
> > > our
> > > > > system manager is checking, if we have a result we will let you
> > know.:)
> > > > > Thanks,
> > > > > LvZheng.
> > > > >
> > > > > 2009/10/11 stack <st...@duboce.net>
> > > > >
> > > > > > On Fri, Oct 9, 2009 at 3:18 AM, Zheng Lv <
> > lvzheng19800619@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > ...
> > > > > > > so,
> > > > > > > > please remove the delay so hbase fails faster so it
> doesn't
> > > take
> > > > > so
> > > > > > > long to
> > > > > > > > figure the issue.
> > > > > > > > Are you inserting every 10ms because hbase is falling over
> > on
> > > > you?
> > > > > > If
> > > > > > > Yes I inserted every 10ms because I'm afraid hbase would
> fall
> > > > over.
> > > > > > Now
> > > > > > > I have removed the delay.
> > > > > > >
> > > > > > > After doing these, We have run the test program again, and
> one
> > > > > region
> > > > > > > server shut down after about 2 hours, another one 3.
> > > > > > > I will post the logs on these two servers in following reply
> > > > mails.
> > > > > > >
> > > > > > >
> > > > > > Thanks for doing the above.
> > > > > >
> > > > > > For the future, debugging, please enable DEBUG and put your logs
> > > > > somewhere
> > > > > > where I can pull them or put them up in pastebin. Logs in email
> > > > messages
> > > > > > are hard to follow. Thanks.
> > > > > >
> > > > > >
> > > > > > > > Ok. So this is hbase 0.20.0? Tell us about your
> hardware.
> > > > What
> > > > > > kind
> > > > > > > is
> > > > > > > > it? CPU/RAM/Disks.
> > > > > > > Yes we are using hbase 0.20.0. And the following is our
> > > > hardware:
> > > > > > >
> > > > > > > CPU:amd x3 710
> > > > > > > RAM:8g ddr2 800
> > > > > > > Disk:270g(raid0)
> > > > > > >
> > > > > > >
> > > > > > Thats an interesting chip -- 3 cores! The above should be fine
> as
> > > long
> > > > > as
> > > > > > you coral your mapreduce jobs running on same cluster.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > > We have 7 servers with above hardware, one for master, three
> > for
> > > > > > > namenodes / regionservers, and the other 3 for zks.
> > > > > > > By the way, what kind of hardware and environment do you
> > suggest
> > > > we
> > > > > > > have?
> > > > > > >
> > > > > >
> > > > > >
> > > > > > This configuration seems fine to start with. Later we might
> > > experiment
> > > > > > running zk on same machines as regionservers and then up number
> of
> > > > > > regionservers to 6 and up the quorum members to 5.
> > > > > >
> > > > > > St.Ack
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > Thank you, very much.
> > > > > > > LvZheng.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: is there any problem with our environment?
Posted by Zheng Lv <lv...@gmail.com>.
Hello Stack,
>How did you fix it?
We found some "link down" and "link up" in some server's syslog, our system
manager suggested to change the network cards on those servers, so we
changed them, and the errors disappeared.
>So figure out whats happening to that region by grepping its name in the
>master log. Why is it offline so long? Are machines loaded? Swapping?
>Yeah, its split. Thats normal. Whats not normal is the client not finding
>the daughter split in its new location. Did the daughters get deployed
>promptly?
I'm sorry, I don't know exactly how I can get these infomation from the log.
Can you tell me? And I have upload the logs to sky drives
http://cid-a331bb289a14fbef.skydrive.live.com/browse.aspx/.Public/1123?uc=1&isFromRichUpload=1&lc=2052.
There are two master logs, which are from different days, and two
regionserver logs, which are from the first 2 shutdown servers.
>
>Are the crawlers running on same machines as hbase?
Yes, they are. And the cluster we are using is like that:
1 master/namenode/jobtracker/crawler server.
6 rs/zk/datanode/tasktracker/crawler client.
>
>What crawler are you using?
The crawler we are using is developed by ourselves, which are composed of a
server and several client, communicating with RMI.
>
>Andrew Purtell has written up some notes on getting a nice balance between
>crawl process and hbase such that all runs smoothly in private
>correspondence. Let me ask him if its ok to forward the list.
If so, we will be very happy and thankful.
>Can you update to hbase 0.20.2? It has a bunch of fixes that could be
>related to the above.
We have updated to hbase 0.20.2 this morning, and we will re-run the crawler
in a minute, and we will share the result with you.
Regards,
LvZheng.
2009/11/21 stack <st...@duboce.net>
> On Fri, Nov 20, 2009 at 12:28 AM, Zheng Lv <lvzheng19800619@gmail.com
> >wrote:
>
> > Hello Stack,
> > Remember the "no route to host" exceptions last time? Now there isn't any
> > more, and the test program can be running for several days.
>
>
> How did you fix it?
>
>
>
> > Thank you.
> > Recently we started running our crawling program, which crawls webpages
> and
> > then insert them to hbase.
> > But we got so many "org.apache.hadoop.hbase.NotServingRegionException"
> like
> > that:
> >
> > 2009-11-20 12:36:41,898 ERROR
> > org.apache.hadoop.hbase.regionserver.HRegionServer:
> > org.apache.hadoop.hbase.NotServingRegionException: webpage,
> >
> >
> http://bbs.city.tianya.cn/tianyacity/Content/178/1/536629.shtml,1258691377544
> >
>
> So figure out whats happening to that region by grepping its name in the
> master log. Why is it offline so long? Are machines loaded? Swapping?
>
> Are the crawlers running on same machines as hbase?
>
> What crawler are you using?
>
> Andrew Purtell has written up some notes on getting a nice balance between
> crawl process and hbase such that all runs smoothly in private
> correspondence. Let me ask him if its ok to forward the list.
>
>
> ....
>
> 2009-11-20 12:36:25,259 INFO org.apache.hadoop.hbase.master.ServerManager:
> > Processing MSG_REPORT_SPLIT:
> > webpage,http:\x2F\x2Fbbs.city.tianya.cn <http://x2fbbs.city.tianya.cn/>
> > \x2Ftianyacity\x2FContent\x2F178\x2F1\x2F536629.shtml,1258691377544:
> > Daughters; webpage,http:\x2F\x2Fbbs.city.tianya.cn<http://x2fbbs.city.tianya.cn/>
> > \x2Ftianyacity\x2FContent\x2F178\x2F1\x2F536629.shtml,1258691779496,
> > webpage,http:\x2F\x2Fbbs.city.tianya.cn <http://x2fbbs.city.tianya.cn/>
> > \x2Ftianyacity\x2FContent\x2F329\x2F1\x2F164370.shtml,1258691779496
> > from ubuntu12,60020,1258687326554;
> >
> > Yeah, its split. Thats normal. Whats not normal is the client not
> finding
> the daughter split in its new location. Did the daughters get deployed
> promptly?
>
>
>
> > And a few hours later, some rs shutdown.
> >
> > I read the mail
> >
> >
> http://mail-archives.apache.org/mod_mbox/hadoop-hbase-user/200907.mbox/%3C9b27a8a60907272122y1bfa6254n95948942d5ca7f88@mail.gmail.com%3E
> > ,
> > which was sent by my partner Angus. In the mail you told us it was a case
> > of
> > "HBASE-1671", Fix Version of which is 0.20.0, but the hbase version we
> are
> > using is just 0.20.0.
> >
>
> Can you update to hbase 0.20.2? It has a bunch of fixes that could be
> related to the above.
> Yours,
> St.Ack
>
>
>
> > Any idea?
> > Best Regards,
> > LvZheng
> >
> >
> >
> >
> >
> > 2009/10/13 stack <st...@duboce.net>
> >
> > > Thanks for posting. Its much easier reading the logs from there.
> > >
> > > Looking in nohup.out I see it can't find region 'webpage,http:\x2F\
> > > x2Fnews.163.com <http://x2fnews.163.com/> <http://x2fnews.163.com/>
> > >
> >
> \x2F09\x2F080\x2F0\x2F5FOO155J0001124J.html1255072992000_751685,1255316061169'.
> > > It never finds it. It looks like it was assigned successfully to
> > > 192.168.33.5 going by the master log. Once you've figured out the
> > > hardware/networking issues, lets work at getting that region back on
> > line.
> > >
> > > The master timed out its session against zk because of 'no route to
> > host'.
> > >
> > > St.Ack
> > >
> > > On Mon, Oct 12, 2009 at 12:23 AM, Zheng Lv <lvzheng19800619@gmail.com
> > > >wrote:
> > >
> > > > Hello Stack,
> > > > I have enabled DEBUG and restarted the test program. This time the
> > > > master shut down, and I have put the logs on skydrive.
> > > >
> > > >
> > >
> >
> http://cid-a331bb289a14fbef.skydrive.live.com/browse.aspx/.Public?uc=2&isFromRichUpload=1
> > > > .
> > > > "nohup.out" is our test program log,
> "hbase-cyd-master-ubuntu6.log"
> > is
> > > > master log.
> > > >
> > > > On the other hand, today we found that when we run "dmesg", there
> > were
> > > > many logs like "[3641697.122769] r8169: eth0: link down". And I think
> > > this
> > > > might be the reason of so many "no route to host" and "Time Out". Now
> > our
> > > > system manager is checking, if we have a result we will let you
> know.:)
> > > > Thanks,
> > > > LvZheng.
> > > >
> > > > 2009/10/11 stack <st...@duboce.net>
> > > >
> > > > > On Fri, Oct 9, 2009 at 3:18 AM, Zheng Lv <
> lvzheng19800619@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > ...
> > > > > > so,
> > > > > > > please remove the delay so hbase fails faster so it doesn't
> > take
> > > > so
> > > > > > long to
> > > > > > > figure the issue.
> > > > > > > Are you inserting every 10ms because hbase is falling over
> on
> > > you?
> > > > > If
> > > > > > Yes I inserted every 10ms because I'm afraid hbase would fall
> > > over.
> > > > > Now
> > > > > > I have removed the delay.
> > > > > >
> > > > > > After doing these, We have run the test program again, and one
> > > > region
> > > > > > server shut down after about 2 hours, another one 3.
> > > > > > I will post the logs on these two servers in following reply
> > > mails.
> > > > > >
> > > > > >
> > > > > Thanks for doing the above.
> > > > >
> > > > > For the future, debugging, please enable DEBUG and put your logs
> > > > somewhere
> > > > > where I can pull them or put them up in pastebin. Logs in email
> > > messages
> > > > > are hard to follow. Thanks.
> > > > >
> > > > >
> > > > > > > Ok. So this is hbase 0.20.0? Tell us about your hardware.
> > > What
> > > > > kind
> > > > > > is
> > > > > > > it? CPU/RAM/Disks.
> > > > > > Yes we are using hbase 0.20.0. And the following is our
> > > hardware:
> > > > > >
> > > > > > CPU:amd x3 710
> > > > > > RAM:8g ddr2 800
> > > > > > Disk:270g(raid0)
> > > > > >
> > > > > >
> > > > > Thats an interesting chip -- 3 cores! The above should be fine as
> > long
> > > > as
> > > > > you coral your mapreduce jobs running on same cluster.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > > We have 7 servers with above hardware, one for master, three
> for
> > > > > > namenodes / regionservers, and the other 3 for zks.
> > > > > > By the way, what kind of hardware and environment do you
> suggest
> > > we
> > > > > > have?
> > > > > >
> > > > >
> > > > >
> > > > > This configuration seems fine to start with. Later we might
> > experiment
> > > > > running zk on same machines as regionservers and then up number of
> > > > > regionservers to 6 and up the quorum members to 5.
> > > > >
> > > > > St.Ack
> > > > >
> > > > >
> > > > > >
> > > > > > Thank you, very much.
> > > > > > LvZheng.
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: is there any problem with our environment?
Posted by stack <st...@duboce.net>.
On Fri, Nov 20, 2009 at 12:28 AM, Zheng Lv <lv...@gmail.com>wrote:
> Hello Stack,
> Remember the "no route to host" exceptions last time? Now there isn't any
> more, and the test program can be running for several days.
How did you fix it?
> Thank you.
> Recently we started running our crawling program, which crawls webpages and
> then insert them to hbase.
> But we got so many "org.apache.hadoop.hbase.NotServingRegionException" like
> that:
>
> 2009-11-20 12:36:41,898 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> org.apache.hadoop.hbase.NotServingRegionException: webpage,
>
> http://bbs.city.tianya.cn/tianyacity/Content/178/1/536629.shtml,1258691377544
>
So figure out whats happening to that region by grepping its name in the
master log. Why is it offline so long? Are machines loaded? Swapping?
Are the crawlers running on same machines as hbase?
What crawler are you using?
Andrew Purtell has written up some notes on getting a nice balance between
crawl process and hbase such that all runs smoothly in private
correspondence. Let me ask him if its ok to forward the list.
....
2009-11-20 12:36:25,259 INFO org.apache.hadoop.hbase.master.ServerManager:
> Processing MSG_REPORT_SPLIT:
> webpage,http:\x2F\x2Fbbs.city.tianya.cn
> \x2Ftianyacity\x2FContent\x2F178\x2F1\x2F536629.shtml,1258691377544:
> Daughters; webpage,http:\x2F\x2Fbbs.city.tianya.cn
> \x2Ftianyacity\x2FContent\x2F178\x2F1\x2F536629.shtml,1258691779496,
> webpage,http:\x2F\x2Fbbs.city.tianya.cn
> \x2Ftianyacity\x2FContent\x2F329\x2F1\x2F164370.shtml,1258691779496
> from ubuntu12,60020,1258687326554;
>
> Yeah, its split. Thats normal. Whats not normal is the client not finding
the daughter split in its new location. Did the daughters get deployed
promptly?
> And a few hours later, some rs shutdown.
>
> I read the mail
>
> http://mail-archives.apache.org/mod_mbox/hadoop-hbase-user/200907.mbox/%3C9b27a8a60907272122y1bfa6254n95948942d5ca7f88@mail.gmail.com%3E
> ,
> which was sent by my partner Angus. In the mail you told us it was a case
> of
> "HBASE-1671", Fix Version of which is 0.20.0, but the hbase version we are
> using is just 0.20.0.
>
Can you update to hbase 0.20.2? It has a bunch of fixes that could be
related to the above.
Yours,
St.Ack
> Any idea?
> Best Regards,
> LvZheng
>
>
>
>
>
> 2009/10/13 stack <st...@duboce.net>
>
> > Thanks for posting. Its much easier reading the logs from there.
> >
> > Looking in nohup.out I see it can't find region 'webpage,http:\x2F\
> > x2Fnews.163.com <http://x2fnews.163.com/>
> >
> \x2F09\x2F080\x2F0\x2F5FOO155J0001124J.html1255072992000_751685,1255316061169'.
> > It never finds it. It looks like it was assigned successfully to
> > 192.168.33.5 going by the master log. Once you've figured out the
> > hardware/networking issues, lets work at getting that region back on
> line.
> >
> > The master timed out its session against zk because of 'no route to
> host'.
> >
> > St.Ack
> >
> > On Mon, Oct 12, 2009 at 12:23 AM, Zheng Lv <lvzheng19800619@gmail.com
> > >wrote:
> >
> > > Hello Stack,
> > > I have enabled DEBUG and restarted the test program. This time the
> > > master shut down, and I have put the logs on skydrive.
> > >
> > >
> >
> http://cid-a331bb289a14fbef.skydrive.live.com/browse.aspx/.Public?uc=2&isFromRichUpload=1
> > > .
> > > "nohup.out" is our test program log, "hbase-cyd-master-ubuntu6.log"
> is
> > > master log.
> > >
> > > On the other hand, today we found that when we run "dmesg", there
> were
> > > many logs like "[3641697.122769] r8169: eth0: link down". And I think
> > this
> > > might be the reason of so many "no route to host" and "Time Out". Now
> our
> > > system manager is checking, if we have a result we will let you know.:)
> > > Thanks,
> > > LvZheng.
> > >
> > > 2009/10/11 stack <st...@duboce.net>
> > >
> > > > On Fri, Oct 9, 2009 at 3:18 AM, Zheng Lv <lv...@gmail.com>
> > > > wrote:
> > > >
> > > > > ...
> > > > > so,
> > > > > > please remove the delay so hbase fails faster so it doesn't
> take
> > > so
> > > > > long to
> > > > > > figure the issue.
> > > > > > Are you inserting every 10ms because hbase is falling over on
> > you?
> > > > If
> > > > > Yes I inserted every 10ms because I'm afraid hbase would fall
> > over.
> > > > Now
> > > > > I have removed the delay.
> > > > >
> > > > > After doing these, We have run the test program again, and one
> > > region
> > > > > server shut down after about 2 hours, another one 3.
> > > > > I will post the logs on these two servers in following reply
> > mails.
> > > > >
> > > > >
> > > > Thanks for doing the above.
> > > >
> > > > For the future, debugging, please enable DEBUG and put your logs
> > > somewhere
> > > > where I can pull them or put them up in pastebin. Logs in email
> > messages
> > > > are hard to follow. Thanks.
> > > >
> > > >
> > > > > > Ok. So this is hbase 0.20.0? Tell us about your hardware.
> > What
> > > > kind
> > > > > is
> > > > > > it? CPU/RAM/Disks.
> > > > > Yes we are using hbase 0.20.0. And the following is our
> > hardware:
> > > > >
> > > > > CPU:amd x3 710
> > > > > RAM:8g ddr2 800
> > > > > Disk:270g(raid0)
> > > > >
> > > > >
> > > > Thats an interesting chip -- 3 cores! The above should be fine as
> long
> > > as
> > > > you coral your mapreduce jobs running on same cluster.
> > > >
> > > >
> > > >
> > > >
> > > > > We have 7 servers with above hardware, one for master, three for
> > > > > namenodes / regionservers, and the other 3 for zks.
> > > > > By the way, what kind of hardware and environment do you suggest
> > we
> > > > > have?
> > > > >
> > > >
> > > >
> > > > This configuration seems fine to start with. Later we might
> experiment
> > > > running zk on same machines as regionservers and then up number of
> > > > regionservers to 6 and up the quorum members to 5.
> > > >
> > > > St.Ack
> > > >
> > > >
> > > > >
> > > > > Thank you, very much.
> > > > > LvZheng.
> > > > >
> > > >
> > >
> >
>
Re: is there any problem with our environment?
Posted by Zheng Lv <lv...@gmail.com>.
Hello Stack,
Remember the "no route to host" exceptions last time? Now there isn't any
more, and the test program can be running for several days. Thank you.
Recently we started running our crawling program, which crawls webpages and
then insert them to hbase.
But we got so many "org.apache.hadoop.hbase.NotServingRegionException" like
that:
2009-11-20 12:36:41,898 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
org.apache.hadoop.hbase.NotServingRegionException: webpage,
http://bbs.city.tianya.cn/tianyacity/Content/178/1/536629.shtml,1258691377544
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2261)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1767)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
And in master we can find that corresponding log:
2009-11-20 12:36:25,259 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_SPLIT:
webpage,http:\x2F\x2Fbbs.city.tianya.cn\x2Ftianyacity\x2FContent\x2F178\x2F1\x2F536629.shtml,1258691377544:
Daughters; webpage,http:\x2F\x2Fbbs.city.tianya.cn\x2Ftianyacity\x2FContent\x2F178\x2F1\x2F536629.shtml,1258691779496,
webpage,http:\x2F\x2Fbbs.city.tianya.cn\x2Ftianyacity\x2FContent\x2F329\x2F1\x2F164370.shtml,1258691779496
from ubuntu12,60020,1258687326554;
And a few hours later, some rs shutdown.
I read the mail
http://mail-archives.apache.org/mod_mbox/hadoop-hbase-user/200907.mbox/%3C9b27a8a60907272122y1bfa6254n95948942d5ca7f88@mail.gmail.com%3E,
which was sent by my partner Angus. In the mail you told us it was a case of
"HBASE-1671", Fix Version of which is 0.20.0, but the hbase version we are
using is just 0.20.0.
Any idea?
Best Regards,
LvZheng
2009/10/13 stack <st...@duboce.net>
> Thanks for posting. Its much easier reading the logs from there.
>
> Looking in nohup.out I see it can't find region 'webpage,http:\x2F\
> x2Fnews.163.com <http://x2fnews.163.com/>
> \x2F09\x2F080\x2F0\x2F5FOO155J0001124J.html1255072992000_751685,1255316061169'.
> It never finds it. It looks like it was assigned successfully to
> 192.168.33.5 going by the master log. Once you've figured out the
> hardware/networking issues, lets work at getting that region back on line.
>
> The master timed out its session against zk because of 'no route to host'.
>
> St.Ack
>
> On Mon, Oct 12, 2009 at 12:23 AM, Zheng Lv <lvzheng19800619@gmail.com
> >wrote:
>
> > Hello Stack,
> > I have enabled DEBUG and restarted the test program. This time the
> > master shut down, and I have put the logs on skydrive.
> >
> >
> http://cid-a331bb289a14fbef.skydrive.live.com/browse.aspx/.Public?uc=2&isFromRichUpload=1
> > .
> > "nohup.out" is our test program log, "hbase-cyd-master-ubuntu6.log" is
> > master log.
> >
> > On the other hand, today we found that when we run "dmesg", there were
> > many logs like "[3641697.122769] r8169: eth0: link down". And I think
> this
> > might be the reason of so many "no route to host" and "Time Out". Now our
> > system manager is checking, if we have a result we will let you know.:)
> > Thanks,
> > LvZheng.
> >
> > 2009/10/11 stack <st...@duboce.net>
> >
> > > On Fri, Oct 9, 2009 at 3:18 AM, Zheng Lv <lv...@gmail.com>
> > > wrote:
> > >
> > > > ...
> > > > so,
> > > > > please remove the delay so hbase fails faster so it doesn't take
> > so
> > > > long to
> > > > > figure the issue.
> > > > > Are you inserting every 10ms because hbase is falling over on
> you?
> > > If
> > > > Yes I inserted every 10ms because I'm afraid hbase would fall
> over.
> > > Now
> > > > I have removed the delay.
> > > >
> > > > After doing these, We have run the test program again, and one
> > region
> > > > server shut down after about 2 hours, another one 3.
> > > > I will post the logs on these two servers in following reply
> mails.
> > > >
> > > >
> > > Thanks for doing the above.
> > >
> > > For the future, debugging, please enable DEBUG and put your logs
> > somewhere
> > > where I can pull them or put them up in pastebin. Logs in email
> messages
> > > are hard to follow. Thanks.
> > >
> > >
> > > > > Ok. So this is hbase 0.20.0? Tell us about your hardware.
> What
> > > kind
> > > > is
> > > > > it? CPU/RAM/Disks.
> > > > Yes we are using hbase 0.20.0. And the following is our
> hardware:
> > > >
> > > > CPU:amd x3 710
> > > > RAM:8g ddr2 800
> > > > Disk:270g(raid0)
> > > >
> > > >
> > > Thats an interesting chip -- 3 cores! The above should be fine as long
> > as
> > > you coral your mapreduce jobs running on same cluster.
> > >
> > >
> > >
> > >
> > > > We have 7 servers with above hardware, one for master, three for
> > > > namenodes / regionservers, and the other 3 for zks.
> > > > By the way, what kind of hardware and environment do you suggest
> we
> > > > have?
> > > >
> > >
> > >
> > > This configuration seems fine to start with. Later we might experiment
> > > running zk on same machines as regionservers and then up number of
> > > regionservers to 6 and up the quorum members to 5.
> > >
> > > St.Ack
> > >
> > >
> > > >
> > > > Thank you, very much.
> > > > LvZheng.
> > > >
> > >
> >
>
Re: is there any problem with our environment?
Posted by stack <st...@duboce.net>.
Thanks for posting. Its much easier reading the logs from there.
Looking in nohup.out I see it can't find region 'webpage,http:\x2F\
x2Fnews.163.com\x2F09\x2F080\x2F0\x2F5FOO155J0001124J.html1255072992000_751685,1255316061169'.
It never finds it. It looks like it was assigned successfully to
192.168.33.5 going by the master log. Once you've figured out the
hardware/networking issues, lets work at getting that region back on line.
The master timed out its session against zk because of 'no route to host'.
St.Ack
On Mon, Oct 12, 2009 at 12:23 AM, Zheng Lv <lv...@gmail.com>wrote:
> Hello Stack,
> I have enabled DEBUG and restarted the test program. This time the
> master shut down, and I have put the logs on skydrive.
>
> http://cid-a331bb289a14fbef.skydrive.live.com/browse.aspx/.Public?uc=2&isFromRichUpload=1
> .
> "nohup.out" is our test program log, "hbase-cyd-master-ubuntu6.log" is
> master log.
>
> On the other hand, today we found that when we run "dmesg", there were
> many logs like "[3641697.122769] r8169: eth0: link down". And I think this
> might be the reason of so many "no route to host" and "Time Out". Now our
> system manager is checking, if we have a result we will let you know.:)
> Thanks,
> LvZheng.
>
> 2009/10/11 stack <st...@duboce.net>
>
> > On Fri, Oct 9, 2009 at 3:18 AM, Zheng Lv <lv...@gmail.com>
> > wrote:
> >
> > > ...
> > > so,
> > > > please remove the delay so hbase fails faster so it doesn't take
> so
> > > long to
> > > > figure the issue.
> > > > Are you inserting every 10ms because hbase is falling over on you?
> > If
> > > Yes I inserted every 10ms because I'm afraid hbase would fall over.
> > Now
> > > I have removed the delay.
> > >
> > > After doing these, We have run the test program again, and one
> region
> > > server shut down after about 2 hours, another one 3.
> > > I will post the logs on these two servers in following reply mails.
> > >
> > >
> > Thanks for doing the above.
> >
> > For the future, debugging, please enable DEBUG and put your logs
> somewhere
> > where I can pull them or put them up in pastebin. Logs in email messages
> > are hard to follow. Thanks.
> >
> >
> > > > Ok. So this is hbase 0.20.0? Tell us about your hardware. What
> > kind
> > > is
> > > > it? CPU/RAM/Disks.
> > > Yes we are using hbase 0.20.0. And the following is our hardware:
> > >
> > > CPU:amd x3 710
> > > RAM:8g ddr2 800
> > > Disk:270g(raid0)
> > >
> > >
> > Thats an interesting chip -- 3 cores! The above should be fine as long
> as
> > you coral your mapreduce jobs running on same cluster.
> >
> >
> >
> >
> > > We have 7 servers with above hardware, one for master, three for
> > > namenodes / regionservers, and the other 3 for zks.
> > > By the way, what kind of hardware and environment do you suggest we
> > > have?
> > >
> >
> >
> > This configuration seems fine to start with. Later we might experiment
> > running zk on same machines as regionservers and then up number of
> > regionservers to 6 and up the quorum members to 5.
> >
> > St.Ack
> >
> >
> > >
> > > Thank you, very much.
> > > LvZheng.
> > >
> >
>
Re: is there any problem with our environment?
Posted by Zheng Lv <lv...@gmail.com>.
Hello Stack,
I have enabled DEBUG and restarted the test program. This time the
master shut down, and I have put the logs on skydrive.
http://cid-a331bb289a14fbef.skydrive.live.com/browse.aspx/.Public?uc=2&isFromRichUpload=1
.
"nohup.out" is our test program log, "hbase-cyd-master-ubuntu6.log" is
master log.
On the other hand, today we found that when we run "dmesg", there were
many logs like "[3641697.122769] r8169: eth0: link down". And I think this
might be the reason of so many "no route to host" and "Time Out". Now our
system manager is checking, if we have a result we will let you know.:)
Thanks,
LvZheng.
2009/10/11 stack <st...@duboce.net>
> On Fri, Oct 9, 2009 at 3:18 AM, Zheng Lv <lv...@gmail.com>
> wrote:
>
> > ...
> > so,
> > > please remove the delay so hbase fails faster so it doesn't take so
> > long to
> > > figure the issue.
> > > Are you inserting every 10ms because hbase is falling over on you?
> If
> > Yes I inserted every 10ms because I'm afraid hbase would fall over.
> Now
> > I have removed the delay.
> >
> > After doing these, We have run the test program again, and one region
> > server shut down after about 2 hours, another one 3.
> > I will post the logs on these two servers in following reply mails.
> >
> >
> Thanks for doing the above.
>
> For the future, debugging, please enable DEBUG and put your logs somewhere
> where I can pull them or put them up in pastebin. Logs in email messages
> are hard to follow. Thanks.
>
>
> > > Ok. So this is hbase 0.20.0? Tell us about your hardware. What
> kind
> > is
> > > it? CPU/RAM/Disks.
> > Yes we are using hbase 0.20.0. And the following is our hardware:
> >
> > CPU:amd x3 710
> > RAM:8g ddr2 800
> > Disk:270g(raid0)
> >
> >
> Thats an interesting chip -- 3 cores! The above should be fine as long as
> you coral your mapreduce jobs running on same cluster.
>
>
>
>
> > We have 7 servers with above hardware, one for master, three for
> > namenodes / regionservers, and the other 3 for zks.
> > By the way, what kind of hardware and environment do you suggest we
> > have?
> >
>
>
> This configuration seems fine to start with. Later we might experiment
> running zk on same machines as regionservers and then up number of
> regionservers to 6 and up the quorum members to 5.
>
> St.Ack
>
>
> >
> > Thank you, very much.
> > LvZheng.
> >
>
Re: is there any problem with our environment?
Posted by stack <st...@duboce.net>.
On Fri, Oct 9, 2009 at 3:18 AM, Zheng Lv <lv...@gmail.com> wrote:
> ...
> so,
> > please remove the delay so hbase fails faster so it doesn't take so
> long to
> > figure the issue.
> > Are you inserting every 10ms because hbase is falling over on you? If
> Yes I inserted every 10ms because I'm afraid hbase would fall over. Now
> I have removed the delay.
>
> After doing these, We have run the test program again, and one region
> server shut down after about 2 hours, another one 3.
> I will post the logs on these two servers in following reply mails.
>
>
Thanks for doing the above.
For the future, debugging, please enable DEBUG and put your logs somewhere
where I can pull them or put them up in pastebin. Logs in email messages
are hard to follow. Thanks.
> > Ok. So this is hbase 0.20.0? Tell us about your hardware. What kind
> is
> > it? CPU/RAM/Disks.
> Yes we are using hbase 0.20.0. And the following is our hardware:
>
> CPU:amd x3 710
> RAM:8g ddr2 800
> Disk:270g(raid0)
>
>
Thats an interesting chip -- 3 cores! The above should be fine as long as
you coral your mapreduce jobs running on same cluster.
> We have 7 servers with above hardware, one for master, three for
> namenodes / regionservers, and the other 3 for zks.
> By the way, what kind of hardware and environment do you suggest we
> have?
>
This configuration seems fine to start with. Later we might experiment
running zk on same machines as regionservers and then up number of
regionservers to 6 and up the quorum members to 5.
St.Ack
>
> Thank you, very much.
> LvZheng.
>
Re: is there any problem with our environment?
Posted by Zheng Lv <lv...@gmail.com>.
Hello Stack,
Thank you for your reply. These days I was on vacation. I'm so sorry for
replying too late.
> Are you using IPs? Try using hostnames?
I have changed it.
> Are you inserting every 10ms because hbase is falling over on you? If
so,
> please remove the delay so hbase fails faster so it doesn't take so
long to
> figure the issue.
Yes I inserted every 10ms because I'm afraid hbase would fall over. Now
I have removed the delay.
After doing these, We have run the test program again, and one region
server shut down after about 2 hours, another one 3.
I will post the logs on these two servers in following reply mails.
> Ok. So this is hbase 0.20.0? Tell us about your hardware. What kind
is
> it? CPU/RAM/Disks.
Yes we are using hbase 0.20.0. And the following is our hardware:
CPU:amd x3 710
RAM:8g ddr2 800
Disk:270g(raid0)
We have 7 servers with above hardware, one for master, three for
namenodes / regionservers, and the other 3 for zks.
By the way, what kind of hardware and environment do you suggest we
have?
Thank you, very much.
LvZheng.
Re: is there any problem with our environment?
Posted by Zheng Lv <lv...@gmail.com>.
Hello Stack,
I have posted four mails yesterday, but it seemed that you saw only the
last one, did you? I listed the hardware of our servers and the logs in
regionservers.
Can you tell me if the hardwares are competent enough? What do you
suggest to improve our environment? If you have 7 servers with that
hardware, what will you do? Or we need add more servers?
Thank you,
LvZheng
2009/10/9 stack <st...@duboce.net>
> No route to host would seem to indicate a problem with your networking
> (It'd
> likely provoke 'time out').
> St.Ack
>
> On Fri, Oct 9, 2009 at 3:28 AM, Zheng Lv <lv...@gmail.com>
> wrote:
>
> > I noticed that there are so many "no route to host", "time out" and so
> on.
> > I
> > doubt that there are some problems with our LAN. What do you think?
> >
>
Re: is there any problem with our environment?
Posted by stack <st...@duboce.net>.
No route to host would seem to indicate a problem with your networking (It'd
likely provoke 'time out').
St.Ack
On Fri, Oct 9, 2009 at 3:28 AM, Zheng Lv <lv...@gmail.com> wrote:
> I noticed that there are so many "no route to host", "time out" and so on.
> I
> doubt that there are some problems with our LAN. What do you think?
>
Re: is there any problem with our environment?
Posted by Zheng Lv <lv...@gmail.com>.
I noticed that there are so many "no route to host", "time out" and so on. I
doubt that there are some problems with our LAN. What do you think?
Re: is there any problem with our environment?
Posted by stack <st...@duboce.net>.
On Fri, Oct 9, 2009 at 3:20 AM, Zheng Lv <lv...@gmail.com> wrote:
> These are the first shut down server's logs:
>
> 2009-10-09 13:55:34,708 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x124375a36140003 to sun.nio.ch.SelectionKeyImpl@73d1f6da
> java.io.IOException: TIMED OUT
>
So, this is timeout against your zk quorum. Was the machine under load?
This or the zk cluster? Was there lots of i/o or garbage collecting going
on? The timeout is more plain later in the log:
2009-10-09 14:01:55,715 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: Expired, type: None, path: null
2009-10-09 14:01:55,715 WARN org.apache.zookeeper.
>
> ClientCnxn: Exception
> closing session 0x124375a36140000 to sun.nio.ch.SelectionKeyImpl@15f4a8c8
> java.io.IOException: Session Expired
>
... or later
2009-10-09 14:01:24,067 INFO org.apache.zookeeper.
>
> ClientCnxn: Attempting
> connection to server ubuntu12/192.168.33.12:2222
> 2009-10-09 14:01:36,328 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x124375a36140003 to sun.nio.ch.SelectionKeyImpl@7624a116
> java.io.IOException: TIMED OUT
>
Can you figure whats going on here?
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
> 2009-10-09 13:55:34,720 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x124375a36140000 to sun.nio.ch.SelectionKeyImpl@240ede73
> java.io.IOException: TIMED OUT
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
> 2009-10-09 13:55:34,821 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
> state: Disconnected, type: None, path: null
> 2009-10-09 13:55:34,897 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server ubuntu10/192.168.33.10:2222
> 2009-10-09 13:55:35,510 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server ubuntu2/192.168.33.9:2222
> 2009-10-09 13:55:38,518 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x124375a36140003 to sun.nio.ch.SelectionKeyImpl@54547a3a
> java.net.NoRouteToHostException: No route to host
>
No route to host would indicate a networking issue on your cluster. Can you
try and figure what it is? Is it transient? Does this cluster ever come up
and take on load?
St.Ack
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
> 2009-10-09 13:55:38,519 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-10-09 13:55:38,519 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-10-09 13:55:38,707 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server ubuntu12/192.168.33.12:2222
> 2009-10-09 13:55:46,589 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: ubuntu6/192.168.33.7:9000. Already tried 0 time(s).
> 2009-10-09 13:55:48,228 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x124375a36140000 to sun.nio.ch.SelectionKeyImpl@25cd7751
> java.io.IOException: TIMED OUT
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
> 2009-10-09 13:55:48,229 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown output
> java.net.SocketException: Transport endpoint is not connected
> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-10-09 13:55:48,487 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server ubuntu2/192.168.33.9:2222
> 2009-10-09 13:57:20,244 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Starting compaction on region webpage,http:\x2F\x2Fnews.163.com
>
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255067890842_526799,1255068212004
> 2009-10-09 13:57:26,146 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC
> Server handler 8 on 60020 took 1714ms appending an edit to hlog;
> editcount=1537
> ... ...
> 2009-10-09 14:00:55,549 INFO
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META
> updated, and report to master all successful. Old region=REGION => {NAME =>
> 'webpage,http:\x2F\x2Fnews.163.com
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255068061432_537933,1255068399681',
> STARTKEY => 'http:\x2F\x2Fnews.163.com
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255068061432_537933',
> ENDKEY => '', ENCODED => 1740841210, OFFLINE => true, SPLIT => true, TABLE
> => {{NAME => 'webpage', FAMILIES => [{NAME => 'CF_CONTENT', COMPRESSION =>
> 'NONE', VERSIONS => '2', TTL => '2147483647', BLOCKSIZE => '65536',
> IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'CF_INFORMATION',
> COMPRESSION => 'NONE', VERSIONS => '1', TTL => '2147483647', BLOCKSIZE =>
> '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, new regions:
> webpage,http:\x2F\x2Fnews.163.com
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255068061432_537933,1255068399682,
> webpage,http:\x2F\x2Fnews.163.com
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255068126127_540247,1255068399682.
> Split took 3sec
> 2009-10-09 14:01:22,998 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x124375a36140000 to sun.nio.ch.SelectionKeyImpl@1ce8f17d
> java.io.IOException: TIMED OUT
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
> 2009-10-09 14:01:23,099 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
> state: Disconnected, type: None, path: null
> 2009-10-09 14:01:24,067 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server ubuntu12/192.168.33.12:2222
> 2009-10-09 14:01:36,328 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x124375a36140003 to sun.nio.ch.SelectionKeyImpl@7624a116
> java.io.IOException: TIMED OUT
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
> 2009-10-09 14:01:37,317 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server ubuntu10/192.168.33.10:2222
> 2009-10-09 14:01:37,398 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x124375a36140000 to sun.nio.ch.SelectionKeyImpl@a5612e0
> java.io.IOException: TIMED OUT
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
> 2009-10-09 14:01:37,398 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown output
> java.net.SocketException: Transport endpoint is not connected
> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-10-09 14:01:38,042 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server ubuntu10/192.168.33.10:2222
> 2009-10-09 14:01:40,318 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x124375a36140003 to sun.nio.ch.SelectionKeyImpl@18f1356d
> java.net.NoRouteToHostException: No route to host
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
> 2009-10-09 14:01:40,319 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-10-09 14:01:40,319 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-10-09 14:01:40,320 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x124375a36140000 to sun.nio.ch.SelectionKeyImpl@3ea93e6
> java.net.NoRouteToHostException: No route to host
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
> 2009-10-09 14:01:40,320 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-10-09 14:01:40,320 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-10-09 14:01:40,621 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: ubuntu6/192.168.33.7:9000. Already tried 0 time(s).
> 2009-10-09 14:01:41,404 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server ubuntu2/192.168.33.9:2222
> 2009-10-09 14:01:42,402 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server ubuntu2/192.168.33.9:2222
> 2009-10-09 14:01:54,738 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x124375a36140003 to sun.nio.ch.SelectionKeyImpl@581f37ff
> java.io.IOException: TIMED OUT
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
> 2009-10-09 14:01:54,739 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown output
> java.net.SocketException: Transport endpoint is not connected
> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-10-09 14:01:55,499 INFO org.apache.zookeeper.ClientCnxn: Priming
> connection to java.nio.channels.SocketChannel[connected local=/
> 192.168.33.5:38517 remote=ubuntu2/192.168.33.9:2222]
> 2009-10-09 14:01:55,500 INFO org.apache.zookeeper.ClientCnxn: Server
> connection successful
> 2009-10-09 14:01:55,507 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x124375a36140000 to sun.nio.ch.SelectionKeyImpl@6995738b
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcher.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
> at sun.nio.ch.IOUtil.read(IOUtil.java:200)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
> at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:651)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
> 2009-10-09 14:01:55,507 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown input
> java.net.SocketException: Transport endpoint is not connected
> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-10-09 14:01:55,508 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown output
> java.net.SocketException: Transport endpoint is not connected
> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-10-09 14:01:55,712 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server ubuntu12/192.168.33.12:2222
> 2009-10-09 14:01:55,713 INFO org.apache.zookeeper.ClientCnxn: Priming
> connection to java.nio.channels.SocketChannel[connected local=/
> 192.168.33.5:57273 remote=ubuntu12/192.168.33.12:2222]
> 2009-10-09 14:01:55,713 INFO org.apache.zookeeper.ClientCnxn: Server
> connection successful
> 2009-10-09 14:01:55,715 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
> state: Expired, type: None, path: null
> 2009-10-09 14:01:55,715 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x124375a36140000 to sun.nio.ch.SelectionKeyImpl@15f4a8c8
> java.io.IOException: Session Expired
> at
>
> org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:548)
> at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:661)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
> 2009-10-09 14:01:55,715 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session
> expired
> 2009-10-09 14:01:55,803 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
> request=0.0, regions=168, stores=335, storefiles=337,
> storefileIndexSize=28,
> memstoreSize=0, usedHeap=237, maxHeap=2993, blockCacheSize=5151568,
> blockCacheFree=622696432, blockCacheCount=0, blockCacheHitRatio=0
> 2009-10-09 14:01:56,156 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server ubuntu12/192.168.33.12:2222
> 2009-10-09 14:01:56,156 INFO org.apache.zookeeper.ClientCnxn: Priming
> connection to java.nio.channels.SocketChannel[connected local=/
> 192.168.33.5:57274 remote=ubuntu12/192.168.33.12:2222]
> 2009-10-09 14:01:56,156 INFO org.apache.zookeeper.ClientCnxn: Server
> connection successful
> 2009-10-09 14:01:56,507 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: ubuntu6/192.168.33.7:9000. Already tried 0 time(s).
> 2009-10-09 14:01:58,861 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: worker thread exiting
> 2009-10-09 14:02:00,399 INFO
> org.apache.hadoop.hbase.regionserver.LogFlusher:
> regionserver/192.168.33.5:60020.logFlusher exiting
> 2009-10-09 14:02:01,277 INFO
> org.apache.hadoop.hbase.regionserver.LogRoller:
> LogRoller exiting.
> 2009-10-09 14:02:02,713 INFO
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher:
> regionserver/192.168.33.5:60020.cacheFlusher exiting
> 2009-10-09 14:02:06,216 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 7 on 60020, call getClosestRowBefore([B@3925421d, [B@3d179b3a,
> [B@58d01c2b) from 192.168.33.7:46653: error: java.io.IOException: Server
> not
> running, aborting
> java.io.IOException: Server not running, aborting
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1748)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> 2009-10-09 14:02:07,771 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 3 on 60020, call put([B@9ecead5,
> [Lorg.apache.hadoop.hbase.client.Put;@212d4b0a) from 192.168.33.6:52759:
> error: java.io.IOException: Server not running, aborting
> java.io.IOException: Server not running, aborting
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1806)
> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> 2009-10-09 14:02:08,228 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 8 on 60020, call getClosestRowBefore([B@24998e8a, [B@5d5d18cb,
> [B@b06e750) from 192.168.33.7:46653: error: java.io.IOException: Server
> not
> running, aborting
> java.io.IOException: Server not running, aborting
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1748)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> 2009-10-09 14:02:09,850 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 4 on 60020, call put([B@723c3ad7,
> [Lorg.apache.hadoop.hbase.client.Put;@503b33d8) from 192.168.33.6:52759:
> error: java.io.IOException: Server not running, aborting
> java.io.IOException: Server not running, aborting
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1806)
> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> 2009-10-09 14:02:10,233 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 6 on 60020, call getClosestRowBefore([B@5ec8d30e, [B@52986cbf,
> [B@150b99f6) from 192.168.33.7:46653: error: java.io.IOException: Server
> not
> running, aborting
> java.io.IOException: Server not running, aborting
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1748)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> 2009-10-09 14:02:11,856 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 5 on 60020, call put([B@5a8b6e61,
> [Lorg.apache.hadoop.hbase.client.Put;@77fbd0b7) from 192.168.33.6:52759:
> error: java.io.IOException: Server not running, aborting
> java.io.IOException: Server not running, aborting
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1806)
> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> 2009-10-09 14:02:12,241 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 1 on 60020, call getClosestRowBefore([B@3543dbeb, [B@7a1b3c9e,
> [B@1228e7b3) from 192.168.33.7:46653: error: java.io.IOException: Server
> not
> running, aborting
> java.io.IOException: Server not running, aborting
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1748)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> 2009-10-09 14:02:15,550 INFO
> org.apache.hadoop.hbase.regionserver.CompactSplitThread:
> regionserver/192.168.33.5:60020.compactor exiting
> 2009-10-09 14:02:16,246 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 0 on 60020, call getClosestRowBefore([B@3f062f39, [B@2f066782,
> [B@4d759fec) from 192.168.33.7:46653: error: java.io.IOException: Server
> not
> running, aborting
> java.io.IOException: Server not running, aborting
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1748)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> 2009-10-09 14:02:18,070 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 9 on 60020, call put([B@b6f5d7c,
> [Lorg.apache.hadoop.hbase.client.Put;@3c627dc1) from 192.168.33.7:46658:
> error: java.io.IOException: Server not running, aborting
> java.io.IOException: Server not running, aborting
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1806)
> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> 2009-10-09 14:02:33,974 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 2 on 60020, call put([B@2be39fa6,
> [Lorg.apache.hadoop.hbase.client.Put;@6d0661cc) from 192.168.33.8:33100:
> error: java.io.IOException: Server not running, aborting
> java.io.IOException: Server not running, aborting
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1806)
> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> 2009-10-09 14:03:18,164 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
> (Retry: 0)
> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
> at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
> at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
> at java.lang.Thread.run(Thread.java:619)
> 2009-10-09 14:03:18,165 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Stop was requested,
> clearing the toDo despite of the exception
> 2009-10-09 14:03:18,165 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> server on 60020
> 2009-10-09 14:03:18,166 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 0 on 60020: exiting
> 2009-10-09 14:03:18,166 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 7 on 60020: exiting
> 2009-10-09 14:03:18,173 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> IPC
> Server listener on 60020
> 2009-10-09 14:03:18,173 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 2 on 60020: exiting
> 2009-10-09 14:03:18,174 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 6 on 60020: exiting
> 2009-10-09 14:03:18,173 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> IPC
> Server Responder
> 2009-10-09 14:03:18,173 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
> 2009-10-09 14:03:18,186 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 9 on 60020: exiting
> 2009-10-09 14:03:18,176 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 4 on 60020: exiting
> 2009-10-09 14:03:18,176 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 5 on 60020: exiting
> 2009-10-09 14:03:18,174 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 8 on 60020: exiting
> 2009-10-09 14:03:18,174 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 3 on 60020: exiting
> 2009-10-09 14:03:18,177 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 1 on 60020: exiting
> 2009-10-09 14:03:18,430 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer$MajorCompactionChecker:
> regionserver/192.168.33.5:60020.majorCompactionChecker exiting
> 2009-10-09 14:03:24,535 INFO org.apache.hadoop.hbase.Leases:
> regionserver/192.168.33.5:60020.leaseChecker closing leases
> 2009-10-09 14:03:24,535 INFO org.apache.hadoop.hbase.Leases:
> regionserver/192.168.33.5:60020.leaseChecker closed leases
> 2009-10-09 14:04:09,451 WARN org.apache.hadoop.hdfs.DFSClient:
> DFSOutputStream ResponseProcessor exception for block
> blk_1789127546305167321_21177java.io.IOException: Bad response 1 for block
> blk_1789127546305167321_21177 from datanode 192.168.33.8:50010
> at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2393)
> 2009-10-09 14:04:09,452 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_1789127546305167321_21177 bad datanode[2]
> 192.168.33.8:50010
> 2009-10-09 14:04:09,452 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_1789127546305167321_21177 in pipeline
> 192.168.33.5:50010, 192.168.33.6:50010, 192.168.33.8:50010: bad datanode
> 192.168.33.8:50010
> 2009-10-09 14:04:09,532 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_1789127546305167321_21177 failed because recovery
> from primary datanode 192.168.33.5:50010 failed 1 times. Pipeline was
> 192.168.33.5:50010, 192.168.33.6:50010, 192.168.33.8:50010. Will retry...
> 2009-10-09 14:04:10,532 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_1789127546305167321_21177 bad datanode[2]
> 192.168.33.8:50010
> 2009-10-09 14:04:10,532 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_1789127546305167321_21177 in pipeline
> 192.168.33.5:50010, 192.168.33.6:50010, 192.168.33.8:50010: bad datanode
> 192.168.33.8:50010
> 2009-10-09 14:04:10,564 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_1789127546305167321_21177 failed because recovery
> from primary datanode 192.168.33.5:50010 failed 2 times. Pipeline was
> 192.168.33.5:50010, 192.168.33.6:50010, 192.168.33.8:50010. Will retry...
> 2009-10-09 14:04:11,564 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_1789127546305167321_21177 bad datanode[2]
> 192.168.33.8:50010
> 2009-10-09 14:04:11,564 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_1789127546305167321_21177 in pipeline
> 192.168.33.5:50010, 192.168.33.6:50010, 192.168.33.8:50010: bad datanode
> 192.168.33.8:50010
> 2009-10-09 14:04:11,573 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_1789127546305167321_21177 failed because recovery
> from primary datanode 192.168.33.5:50010 failed 3 times. Pipeline was
> 192.168.33.5:50010, 192.168.33.6:50010, 192.168.33.8:50010. Will retry...
> 2009-10-09 14:04:12,574 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_1789127546305167321_21177 bad datanode[2]
> 192.168.33.8:50010
> 2009-10-09 14:04:12,574 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_1789127546305167321_21177 in pipeline
> 192.168.33.5:50010, 192.168.33.6:50010, 192.168.33.8:50010: bad datanode
> 192.168.33.8:50010
> 2009-10-09 14:04:12,590 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_1789127546305167321_21177 failed because recovery
> from primary datanode 192.168.33.5:50010 failed 4 times. Pipeline was
> 192.168.33.5:50010, 192.168.33.6:50010, 192.168.33.8:50010. Will retry...
> 2009-10-09 14:04:13,590 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_1789127546305167321_21177 bad datanode[2]
> 192.168.33.8:50010
> 2009-10-09 14:04:13,591 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_1789127546305167321_21177 in pipeline
> 192.168.33.5:50010, 192.168.33.6:50010, 192.168.33.8:50010: bad datanode
> 192.168.33.8:50010
> 2009-10-09 14:04:13,603 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_1789127546305167321_21177 failed because recovery
> from primary datanode 192.168.33.5:50010 failed 5 times. Pipeline was
> 192.168.33.5:50010, 192.168.33.6:50010, 192.168.33.8:50010. Will retry...
> 2009-10-09 14:04:14,603 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_1789127546305167321_21177 bad datanode[2]
> 192.168.33.8:50010
> 2009-10-09 14:04:14,603 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_1789127546305167321_21177 in pipeline
> 192.168.33.5:50010, 192.168.33.6:50010, 192.168.33.8:50010: bad datanode
> 192.168.33.8:50010
> 2009-10-09 14:04:16,242 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_1789127546305167321_21177 failed because recovery
> from primary datanode 192.168.33.5:50010 failed 6 times. Pipeline was
> 192.168.33.5:50010, 192.168.33.6:50010, 192.168.33.8:50010. Marking
> primary
> datanode as bad.
> 2009-10-09 14:04:17,310 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_1789127546305167321_21177 failed because recovery
> from primary datanode 192.168.33.6:50010 failed 1 times. Pipeline was
> 192.168.33.6:50010, 192.168.33.8:50010. Will retry...
> 2009-10-09 14:04:18,641 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_1789127546305167321_21177 failed because recovery
> from primary datanode 192.168.33.6:50010 failed 2 times. Pipeline was
> 192.168.33.6:50010, 192.168.33.8:50010. Will retry...
> 2009-10-09 14:04:19,654 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_1789127546305167321_21177 failed because recovery
> from primary datanode 192.168.33.6:50010 failed 3 times. Pipeline was
> 192.168.33.6:50010, 192.168.33.8:50010. Will retry...
> 2009-10-09 14:04:20,668 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_1789127546305167321_21177 failed because recovery
> from primary datanode 192.168.33.6:50010 failed 4 times. Pipeline was
> 192.168.33.6:50010, 192.168.33.8:50010. Will retry...
> 2009-10-09 14:04:21,684 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_1789127546305167321_21177 failed because recovery
> from primary datanode 192.168.33.6:50010 failed 5 times. Pipeline was
> 192.168.33.6:50010, 192.168.33.8:50010. Will retry...
> 2009-10-09 14:04:22,708 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_1789127546305167321_21177 failed because recovery
> from primary datanode 192.168.33.6:50010 failed 6 times. Pipeline was
> 192.168.33.6:50010, 192.168.33.8:50010. Marking primary datanode as bad.
> 2009-10-09 14:04:23,755 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_1789127546305167321_21177 failed because recovery
> from primary datanode 192.168.33.8:50010 failed 1 times. Pipeline was
> 192.168.33.8:50010. Will retry...
> 2009-10-09 14:04:24,765 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_1789127546305167321_21177 failed because recovery
> from primary datanode 192.168.33.8:50010 failed 2 times. Pipeline was
> 192.168.33.8:50010. Will retry...
> 2009-10-09 14:04:25,773 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_1789127546305167321_21177 failed because recovery
> from primary datanode 192.168.33.8:50010 failed 3 times. Pipeline was
> 192.168.33.8:50010. Will retry...
> 2009-10-09 14:04:26,783 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_1789127546305167321_21177 failed because recovery
> from primary datanode 192.168.33.8:50010 failed 4 times. Pipeline was
> 192.168.33.8:50010. Will retry...
> 2009-10-09 14:04:27,792 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_1789127546305167321_21177 failed because recovery
> from primary datanode 192.168.33.8:50010 failed 5 times. Pipeline was
> 192.168.33.8:50010. Will retry...
> 2009-10-09 14:04:28,800 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_1789127546305167321_21177 failed because recovery
> from primary datanode 192.168.33.8:50010 failed 6 times. Pipeline was
> 192.168.33.8:50010. Aborting...
> 2009-10-09 14:04:28,800 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to close log in
> abort
> java.io.IOException: Error Recovery for block blk_1789127546305167321_21177
> failed because recovery from primary datanode 192.168.33.8:50010 failed 6
> times. Pipeline was 192.168.33.8:50010. Aborting...
> at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2554)
> at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2048)
> at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2211)
> 2009-10-09 14:04:28,802 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Closed webpage,http:\x2F\x2Fnews.163.com
>
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255067721595_511167,1255067964885
> 2009-10-09 14:04:28,802 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Closed webpage,http:\x2F\x2Fnews.163.com
>
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255061989830_96119,1255062349801
> ... ...
> 2009-10-09 14:04:28,854 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Closed webpage,http:\x2F\x2Fnews.163.com
>
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254221224742_62772,1254221761034
> 2009-10-09 14:04:28,854 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Closed webpage,http:\x2F\x2Fnews.163.com
>
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255067666341_505743,1255067964883
> 2009-10-09 14:04:28,854 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: aborting server at:
> 192.168.33.5:60020
> 2009-10-09 14:04:28,854 INFO org.apache.zookeeper.ZooKeeper: Closing
> session: 0x124375a36140000
> 2009-10-09 14:04:28,854 INFO org.apache.zookeeper.ClientCnxn: Closing
> ClientCnxn for session: 0x124375a36140000
> 2009-10-09 14:04:28,855 INFO org.apache.zookeeper.ClientCnxn: Disconnecting
> ClientCnxn for session: 0x124375a36140000
> 2009-10-09 14:04:28,855 INFO org.apache.zookeeper.ZooKeeper: Session:
> 0x124375a36140000 closed
> 2009-10-09 14:04:28,855 INFO org.apache.zookeeper.ClientCnxn: EventThread
> shut down
> 2009-10-09 14:04:28,899 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver/
> 192.168.33.5:60020 exiting
> 2009-10-09 14:04:28,909 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Starting shutdown
> thread.
> 2009-10-09 14:04:28,909 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Shutdown thread
> complete
>
Re: is there any problem with our environment?
Posted by Zheng Lv <lv...@gmail.com>.
These are the first shut down server's logs:
2009-10-09 13:55:34,708 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x124375a36140003 to sun.nio.ch.SelectionKeyImpl@73d1f6da
java.io.IOException: TIMED OUT
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-10-09 13:55:34,720 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x124375a36140000 to sun.nio.ch.SelectionKeyImpl@240ede73
java.io.IOException: TIMED OUT
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-10-09 13:55:34,821 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: Disconnected, type: None, path: null
2009-10-09 13:55:34,897 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu10/192.168.33.10:2222
2009-10-09 13:55:35,510 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu2/192.168.33.9:2222
2009-10-09 13:55:38,518 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x124375a36140003 to sun.nio.ch.SelectionKeyImpl@54547a3a
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
2009-10-09 13:55:38,519 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown input
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-10-09 13:55:38,519 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-10-09 13:55:38,707 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu12/192.168.33.12:2222
2009-10-09 13:55:46,589 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: ubuntu6/192.168.33.7:9000. Already tried 0 time(s).
2009-10-09 13:55:48,228 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x124375a36140000 to sun.nio.ch.SelectionKeyImpl@25cd7751
java.io.IOException: TIMED OUT
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-10-09 13:55:48,229 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.net.SocketException: Transport endpoint is not connected
at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-10-09 13:55:48,487 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu2/192.168.33.9:2222
2009-10-09 13:57:20,244 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255067890842_526799,1255068212004
2009-10-09 13:57:26,146 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC
Server handler 8 on 60020 took 1714ms appending an edit to hlog;
editcount=1537
... ...
2009-10-09 14:00:55,549 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META
updated, and report to master all successful. Old region=REGION => {NAME =>
'webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255068061432_537933,1255068399681',
STARTKEY => 'http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255068061432_537933',
ENDKEY => '', ENCODED => 1740841210, OFFLINE => true, SPLIT => true, TABLE
=> {{NAME => 'webpage', FAMILIES => [{NAME => 'CF_CONTENT', COMPRESSION =>
'NONE', VERSIONS => '2', TTL => '2147483647', BLOCKSIZE => '65536',
IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'CF_INFORMATION',
COMPRESSION => 'NONE', VERSIONS => '1', TTL => '2147483647', BLOCKSIZE =>
'65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, new regions:
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255068061432_537933,1255068399682,
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255068126127_540247,1255068399682.
Split took 3sec
2009-10-09 14:01:22,998 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x124375a36140000 to sun.nio.ch.SelectionKeyImpl@1ce8f17d
java.io.IOException: TIMED OUT
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-10-09 14:01:23,099 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: Disconnected, type: None, path: null
2009-10-09 14:01:24,067 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu12/192.168.33.12:2222
2009-10-09 14:01:36,328 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x124375a36140003 to sun.nio.ch.SelectionKeyImpl@7624a116
java.io.IOException: TIMED OUT
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-10-09 14:01:37,317 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu10/192.168.33.10:2222
2009-10-09 14:01:37,398 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x124375a36140000 to sun.nio.ch.SelectionKeyImpl@a5612e0
java.io.IOException: TIMED OUT
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-10-09 14:01:37,398 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.net.SocketException: Transport endpoint is not connected
at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-10-09 14:01:38,042 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu10/192.168.33.10:2222
2009-10-09 14:01:40,318 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x124375a36140003 to sun.nio.ch.SelectionKeyImpl@18f1356d
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
2009-10-09 14:01:40,319 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown input
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-10-09 14:01:40,319 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-10-09 14:01:40,320 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x124375a36140000 to sun.nio.ch.SelectionKeyImpl@3ea93e6
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
2009-10-09 14:01:40,320 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown input
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-10-09 14:01:40,320 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-10-09 14:01:40,621 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: ubuntu6/192.168.33.7:9000. Already tried 0 time(s).
2009-10-09 14:01:41,404 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu2/192.168.33.9:2222
2009-10-09 14:01:42,402 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu2/192.168.33.9:2222
2009-10-09 14:01:54,738 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x124375a36140003 to sun.nio.ch.SelectionKeyImpl@581f37ff
java.io.IOException: TIMED OUT
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-10-09 14:01:54,739 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.net.SocketException: Transport endpoint is not connected
at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-10-09 14:01:55,499 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.33.5:38517 remote=ubuntu2/192.168.33.9:2222]
2009-10-09 14:01:55,500 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-10-09 14:01:55,507 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x124375a36140000 to sun.nio.ch.SelectionKeyImpl@6995738b
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
at sun.nio.ch.IOUtil.read(IOUtil.java:200)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:651)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
2009-10-09 14:01:55,507 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown input
java.net.SocketException: Transport endpoint is not connected
at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-10-09 14:01:55,508 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.net.SocketException: Transport endpoint is not connected
at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-10-09 14:01:55,712 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu12/192.168.33.12:2222
2009-10-09 14:01:55,713 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.33.5:57273 remote=ubuntu12/192.168.33.12:2222]
2009-10-09 14:01:55,713 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-10-09 14:01:55,715 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: Expired, type: None, path: null
2009-10-09 14:01:55,715 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x124375a36140000 to sun.nio.ch.SelectionKeyImpl@15f4a8c8
java.io.IOException: Session Expired
at
org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:548)
at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:661)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
2009-10-09 14:01:55,715 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session
expired
2009-10-09 14:01:55,803 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
request=0.0, regions=168, stores=335, storefiles=337, storefileIndexSize=28,
memstoreSize=0, usedHeap=237, maxHeap=2993, blockCacheSize=5151568,
blockCacheFree=622696432, blockCacheCount=0, blockCacheHitRatio=0
2009-10-09 14:01:56,156 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu12/192.168.33.12:2222
2009-10-09 14:01:56,156 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.33.5:57274 remote=ubuntu12/192.168.33.12:2222]
2009-10-09 14:01:56,156 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-10-09 14:01:56,507 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: ubuntu6/192.168.33.7:9000. Already tried 0 time(s).
2009-10-09 14:01:58,861 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: worker thread exiting
2009-10-09 14:02:00,399 INFO
org.apache.hadoop.hbase.regionserver.LogFlusher:
regionserver/192.168.33.5:60020.logFlusher exiting
2009-10-09 14:02:01,277 INFO org.apache.hadoop.hbase.regionserver.LogRoller:
LogRoller exiting.
2009-10-09 14:02:02,713 INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher:
regionserver/192.168.33.5:60020.cacheFlusher exiting
2009-10-09 14:02:06,216 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 7 on 60020, call getClosestRowBefore([B@3925421d, [B@3d179b3a,
[B@58d01c2b) from 192.168.33.7:46653: error: java.io.IOException: Server not
running, aborting
java.io.IOException: Server not running, aborting
at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1748)
at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-09 14:02:07,771 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 3 on 60020, call put([B@9ecead5,
[Lorg.apache.hadoop.hbase.client.Put;@212d4b0a) from 192.168.33.6:52759:
error: java.io.IOException: Server not running, aborting
java.io.IOException: Server not running, aborting
at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1806)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-09 14:02:08,228 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 8 on 60020, call getClosestRowBefore([B@24998e8a, [B@5d5d18cb,
[B@b06e750) from 192.168.33.7:46653: error: java.io.IOException: Server not
running, aborting
java.io.IOException: Server not running, aborting
at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1748)
at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-09 14:02:09,850 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 4 on 60020, call put([B@723c3ad7,
[Lorg.apache.hadoop.hbase.client.Put;@503b33d8) from 192.168.33.6:52759:
error: java.io.IOException: Server not running, aborting
java.io.IOException: Server not running, aborting
at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1806)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-09 14:02:10,233 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 6 on 60020, call getClosestRowBefore([B@5ec8d30e, [B@52986cbf,
[B@150b99f6) from 192.168.33.7:46653: error: java.io.IOException: Server not
running, aborting
java.io.IOException: Server not running, aborting
at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1748)
at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-09 14:02:11,856 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 5 on 60020, call put([B@5a8b6e61,
[Lorg.apache.hadoop.hbase.client.Put;@77fbd0b7) from 192.168.33.6:52759:
error: java.io.IOException: Server not running, aborting
java.io.IOException: Server not running, aborting
at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1806)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-09 14:02:12,241 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 1 on 60020, call getClosestRowBefore([B@3543dbeb, [B@7a1b3c9e,
[B@1228e7b3) from 192.168.33.7:46653: error: java.io.IOException: Server not
running, aborting
java.io.IOException: Server not running, aborting
at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1748)
at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-09 14:02:15,550 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread:
regionserver/192.168.33.5:60020.compactor exiting
2009-10-09 14:02:16,246 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 0 on 60020, call getClosestRowBefore([B@3f062f39, [B@2f066782,
[B@4d759fec) from 192.168.33.7:46653: error: java.io.IOException: Server not
running, aborting
java.io.IOException: Server not running, aborting
at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1748)
at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-09 14:02:18,070 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 9 on 60020, call put([B@b6f5d7c,
[Lorg.apache.hadoop.hbase.client.Put;@3c627dc1) from 192.168.33.7:46658:
error: java.io.IOException: Server not running, aborting
java.io.IOException: Server not running, aborting
at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1806)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-09 14:02:33,974 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 2 on 60020, call put([B@2be39fa6,
[Lorg.apache.hadoop.hbase.client.Put;@6d0661cc) from 192.168.33.8:33100:
error: java.io.IOException: Server not running, aborting
java.io.IOException: Server not running, aborting
at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1806)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-10-09 14:03:18,164 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
(Retry: 0)
org.apache.hadoop.hbase.Leases$LeaseStillHeldException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
at java.lang.Thread.run(Thread.java:619)
2009-10-09 14:03:18,165 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Stop was requested,
clearing the toDo despite of the exception
2009-10-09 14:03:18,165 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
server on 60020
2009-10-09 14:03:18,166 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 0 on 60020: exiting
2009-10-09 14:03:18,166 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 7 on 60020: exiting
2009-10-09 14:03:18,173 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC
Server listener on 60020
2009-10-09 14:03:18,173 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 2 on 60020: exiting
2009-10-09 14:03:18,174 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 6 on 60020: exiting
2009-10-09 14:03:18,173 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC
Server Responder
2009-10-09 14:03:18,173 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
2009-10-09 14:03:18,186 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 9 on 60020: exiting
2009-10-09 14:03:18,176 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 4 on 60020: exiting
2009-10-09 14:03:18,176 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 5 on 60020: exiting
2009-10-09 14:03:18,174 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 8 on 60020: exiting
2009-10-09 14:03:18,174 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 3 on 60020: exiting
2009-10-09 14:03:18,177 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 1 on 60020: exiting
2009-10-09 14:03:18,430 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer$MajorCompactionChecker:
regionserver/192.168.33.5:60020.majorCompactionChecker exiting
2009-10-09 14:03:24,535 INFO org.apache.hadoop.hbase.Leases:
regionserver/192.168.33.5:60020.leaseChecker closing leases
2009-10-09 14:03:24,535 INFO org.apache.hadoop.hbase.Leases:
regionserver/192.168.33.5:60020.leaseChecker closed leases
2009-10-09 14:04:09,451 WARN org.apache.hadoop.hdfs.DFSClient:
DFSOutputStream ResponseProcessor exception for block
blk_1789127546305167321_21177java.io.IOException: Bad response 1 for block
blk_1789127546305167321_21177 from datanode 192.168.33.8:50010
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2393)
2009-10-09 14:04:09,452 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 bad datanode[2]
192.168.33.8:50010
2009-10-09 14:04:09,452 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 in pipeline
192.168.33.5:50010, 192.168.33.6:50010, 192.168.33.8:50010: bad datanode
192.168.33.8:50010
2009-10-09 14:04:09,532 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 failed because recovery
from primary datanode 192.168.33.5:50010 failed 1 times. Pipeline was
192.168.33.5:50010, 192.168.33.6:50010, 192.168.33.8:50010. Will retry...
2009-10-09 14:04:10,532 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 bad datanode[2]
192.168.33.8:50010
2009-10-09 14:04:10,532 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 in pipeline
192.168.33.5:50010, 192.168.33.6:50010, 192.168.33.8:50010: bad datanode
192.168.33.8:50010
2009-10-09 14:04:10,564 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 failed because recovery
from primary datanode 192.168.33.5:50010 failed 2 times. Pipeline was
192.168.33.5:50010, 192.168.33.6:50010, 192.168.33.8:50010. Will retry...
2009-10-09 14:04:11,564 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 bad datanode[2]
192.168.33.8:50010
2009-10-09 14:04:11,564 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 in pipeline
192.168.33.5:50010, 192.168.33.6:50010, 192.168.33.8:50010: bad datanode
192.168.33.8:50010
2009-10-09 14:04:11,573 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 failed because recovery
from primary datanode 192.168.33.5:50010 failed 3 times. Pipeline was
192.168.33.5:50010, 192.168.33.6:50010, 192.168.33.8:50010. Will retry...
2009-10-09 14:04:12,574 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 bad datanode[2]
192.168.33.8:50010
2009-10-09 14:04:12,574 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 in pipeline
192.168.33.5:50010, 192.168.33.6:50010, 192.168.33.8:50010: bad datanode
192.168.33.8:50010
2009-10-09 14:04:12,590 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 failed because recovery
from primary datanode 192.168.33.5:50010 failed 4 times. Pipeline was
192.168.33.5:50010, 192.168.33.6:50010, 192.168.33.8:50010. Will retry...
2009-10-09 14:04:13,590 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 bad datanode[2]
192.168.33.8:50010
2009-10-09 14:04:13,591 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 in pipeline
192.168.33.5:50010, 192.168.33.6:50010, 192.168.33.8:50010: bad datanode
192.168.33.8:50010
2009-10-09 14:04:13,603 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 failed because recovery
from primary datanode 192.168.33.5:50010 failed 5 times. Pipeline was
192.168.33.5:50010, 192.168.33.6:50010, 192.168.33.8:50010. Will retry...
2009-10-09 14:04:14,603 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 bad datanode[2]
192.168.33.8:50010
2009-10-09 14:04:14,603 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 in pipeline
192.168.33.5:50010, 192.168.33.6:50010, 192.168.33.8:50010: bad datanode
192.168.33.8:50010
2009-10-09 14:04:16,242 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 failed because recovery
from primary datanode 192.168.33.5:50010 failed 6 times. Pipeline was
192.168.33.5:50010, 192.168.33.6:50010, 192.168.33.8:50010. Marking primary
datanode as bad.
2009-10-09 14:04:17,310 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 failed because recovery
from primary datanode 192.168.33.6:50010 failed 1 times. Pipeline was
192.168.33.6:50010, 192.168.33.8:50010. Will retry...
2009-10-09 14:04:18,641 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 failed because recovery
from primary datanode 192.168.33.6:50010 failed 2 times. Pipeline was
192.168.33.6:50010, 192.168.33.8:50010. Will retry...
2009-10-09 14:04:19,654 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 failed because recovery
from primary datanode 192.168.33.6:50010 failed 3 times. Pipeline was
192.168.33.6:50010, 192.168.33.8:50010. Will retry...
2009-10-09 14:04:20,668 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 failed because recovery
from primary datanode 192.168.33.6:50010 failed 4 times. Pipeline was
192.168.33.6:50010, 192.168.33.8:50010. Will retry...
2009-10-09 14:04:21,684 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 failed because recovery
from primary datanode 192.168.33.6:50010 failed 5 times. Pipeline was
192.168.33.6:50010, 192.168.33.8:50010. Will retry...
2009-10-09 14:04:22,708 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 failed because recovery
from primary datanode 192.168.33.6:50010 failed 6 times. Pipeline was
192.168.33.6:50010, 192.168.33.8:50010. Marking primary datanode as bad.
2009-10-09 14:04:23,755 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 failed because recovery
from primary datanode 192.168.33.8:50010 failed 1 times. Pipeline was
192.168.33.8:50010. Will retry...
2009-10-09 14:04:24,765 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 failed because recovery
from primary datanode 192.168.33.8:50010 failed 2 times. Pipeline was
192.168.33.8:50010. Will retry...
2009-10-09 14:04:25,773 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 failed because recovery
from primary datanode 192.168.33.8:50010 failed 3 times. Pipeline was
192.168.33.8:50010. Will retry...
2009-10-09 14:04:26,783 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 failed because recovery
from primary datanode 192.168.33.8:50010 failed 4 times. Pipeline was
192.168.33.8:50010. Will retry...
2009-10-09 14:04:27,792 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 failed because recovery
from primary datanode 192.168.33.8:50010 failed 5 times. Pipeline was
192.168.33.8:50010. Will retry...
2009-10-09 14:04:28,800 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1789127546305167321_21177 failed because recovery
from primary datanode 192.168.33.8:50010 failed 6 times. Pipeline was
192.168.33.8:50010. Aborting...
2009-10-09 14:04:28,800 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to close log in
abort
java.io.IOException: Error Recovery for block blk_1789127546305167321_21177
failed because recovery from primary datanode 192.168.33.8:50010 failed 6
times. Pipeline was 192.168.33.8:50010. Aborting...
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2554)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2048)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2211)
2009-10-09 14:04:28,802 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255067721595_511167,1255067964885
2009-10-09 14:04:28,802 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255061989830_96119,1255062349801
... ...
2009-10-09 14:04:28,854 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254221224742_62772,1254221761034
2009-10-09 14:04:28,854 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1255067666341_505743,1255067964883
2009-10-09 14:04:28,854 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: aborting server at:
192.168.33.5:60020
2009-10-09 14:04:28,854 INFO org.apache.zookeeper.ZooKeeper: Closing
session: 0x124375a36140000
2009-10-09 14:04:28,854 INFO org.apache.zookeeper.ClientCnxn: Closing
ClientCnxn for session: 0x124375a36140000
2009-10-09 14:04:28,855 INFO org.apache.zookeeper.ClientCnxn: Disconnecting
ClientCnxn for session: 0x124375a36140000
2009-10-09 14:04:28,855 INFO org.apache.zookeeper.ZooKeeper: Session:
0x124375a36140000 closed
2009-10-09 14:04:28,855 INFO org.apache.zookeeper.ClientCnxn: EventThread
shut down
2009-10-09 14:04:28,899 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver/
192.168.33.5:60020 exiting
2009-10-09 14:04:28,909 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Starting shutdown
thread.
2009-10-09 14:04:28,909 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Shutdown thread complete
Re: is there any problem with our environment?
Posted by stack <st...@duboce.net>.
On Sun, Sep 27, 2009 at 2:45 AM, Zheng Lv <lv...@gmail.com> wrote:
> Hello Everyone,
> About one or two months ago, we made a test program which inserts one
> web page per 10ms into hbase, but after running it we got many exceptions
> and many regionservers shutdown very soon.
Are you inserting every 10ms because hbase is falling over on you? If so,
please remove the delay so hbase fails faster so it doesn't take so long to
figure the issue.
> Many people suggested me to make
> zks exclusive. And Now we have built a cluster including one master, three
> regionservers, and three zookeepers, exclusively, but after we ran the same
> test program, regionservers shutdown too, even sooner.
>
Ok. So this is hbase 0.20.0? Tell us about your hardware. What kind is
it? CPU/RAM/Disks.
> Now I'm confused. Is there any problem with the environment? Or some
> other errors? Someone give some suggestions?
>
When the regionservers shutdown, what are they complaining about? OOME?
Session timeout against zookeeper?
Thanks,
St.Ack
>
> BTW, I noticed that when hbase starting up, there are some exceptions in
> logging files, both master, regionservers, zookeepers.
> I will reply this mail to post up all the logs.
>
> Thank you,
> LvZheng
>
Re: is there any problem with our environment?
Posted by stack <st...@duboce.net>.
On Sun, Sep 27, 2009 at 2:50 AM, Zheng Lv <lv...@gmail.com> wrote:
> following is the logs in one of shut down regionservers:
>
>
> Sun Sep 27 15:40:35 CST 2009 Starting regionserver on ubuntu9
> ulimit -n 1024
>
Please see the 'Getting Started' for hbase:
http://hadoop.apache.org/hbase/docs/r0.20.0/api/overview-summary.html#overview_description.
See the section where it says hbase is a file handle hog. There are other
recommendations there too.
Enable DEBUG in your hbase cluster if you are having issues (which it looks
like going by the below). See FAQ for how.
St.Ack
> Recovery for block blk_-2072248271029020079_1234 in pipeline
> 192.168.33.6:50010, 192.168.33.8:50010: bad datanode 192.168.33.6:50010
> 2009-09-27 16:34:32,685 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC
> Server handler 7 on 60020 took 67075ms appending an edit to hlog;
> editcount=6073
> 2009-09-27 16:34:32,686 WARN org.apache.hadoop.hbase.regionserver.HLog:
> regionserver/192.168.33.5:60020.logFlusher took 58173ms optional sync'ing
> hlog; editcount=6074
> 2009-09-27 16:34:32,998 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_-2143462669730028869_1282 bad datanode[0]
> 192.168.33.5:50010
> 2009-09-27 16:34:32,998 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block blk_-2143462669730028869_1282 in pipeline
> 192.168.33.5:50010, 192.168.33.6:50010, 192.168.33.8:50010: bad datanode
> 192.168.33.5:50010
> 2009-09-27 16:34:33,007 INFO org.apache.hadoop.hbase.regionserver.HLog:
> Roll
> /hbase/.logs/ubuntu9,60020,1254037278829/hlog.dat.1254040073957,
> entries=6193, calcsize=63755028, filesize=63097127. New hlog
> /hbase/.logs/ubuntu9,60020,1254037278829/hlog.dat.1254040472966
> 2009-09-27 16:34:45,212 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC
> Server handler 3 on 60020 took 9528ms appending an edit to hlog;
> editcount=1152
> 2009-09-27 16:34:45,213 WARN org.apache.hadoop.hbase.regionserver.HLog:
> regionserver/192.168.33.5:60020.logFlusher took 2527ms optional sync'ing
> hlog; editcount=1158
> 2009-09-27 16:34:46,593 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> compaction completed on region
> webpage,http:\x2F\x2Fnews.163.com
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040043019_9471,1254040612499
> in 1mins, 25sec
> 2009-09-27 16:34:46,593 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Starting compaction on region webpage,http:\x2F\x2Fnews.163.com
>
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040106544_11251,1254040612499
> 2009-09-27 16:34:58,504 INFO org.apache.hadoop.hbase.regionserver.HLog:
> Roll
> /hbase/.logs/ubuntu9,60020,1254037278829/hlog.dat.1254040472966,
> entries=6187, calcsize=63759284, filesize=63101288. New hlog
> /hbase/.logs/ubuntu9,60020,1254037278829/hlog.dat.1254040498496
> 2009-09-27 16:34:59,153 INFO
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Too many store files
> for region webpage,http:\x2F\x2Fnews.163.com
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040106544_11251,1254040612499:
> 5, requesting compaction and waiting
> 2009-09-27 16:35:08,662 INFO
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Compaction has
> completed, we waited 10000ms, finishing flush of region webpage,http:\x2F\
> x2Fnews.163.com
>
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040106544_11251,1254040612499
> 2009-09-27 16:35:09,828 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> compaction completed on region
> webpage,http:\x2F\x2Fnews.163.com
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040106544_11251,1254040612499
> in 23sec
> 2009-09-27 16:35:09,828 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Starting split of region webpage,http:\x2F\x2Fnews.163.com
>
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040106544_11251,1254040612499
> 2009-09-27 16:35:09,836 WARN org.apache.hadoop.hbase.regionserver.HRegion:
> Clock skew; parent regions id is 1254040612499 but current time here is
> 1254040509836
> 2009-09-27 16:35:13,132 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC
> Server handler 9 on 60020 took 1322ms appending an edit to hlog;
> editcount=4679
> 2009-09-27 16:35:13,634 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Closed webpage,http:\x2F\x2Fnews.163.com
>
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040106544_11251,1254040612499
> 2009-09-27 16:35:14,437 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> region webpage,http:\x2F\x2Fnews.163.com
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040106544_11251,1254040612500/512381466
> available; sequence id is 140212
> 2009-09-27 16:35:14,438 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Closed webpage,http:\x2F\x2Fnews.163.com
>
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040106544_11251,1254040612500
> 2009-09-27 16:35:14,638 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> region webpage,http:\x2F\x2Fnews.163.com
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040237301_15439,1254040612500/95607190
> available; sequence id is 140213
> 2009-09-27 16:35:14,639 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Closed webpage,http:\x2F\x2Fnews.163.com
>
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040237301_15439,1254040612500
> 2009-09-27 16:35:14,652 INFO
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META
> updated, and report to master all successful. Old region=REGION => {NAME =>
> 'webpage,http:\x2F\x2Fnews.163.com
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040106544_11251,1254040612499',
> STARTKEY => 'http:\x2F\x2Fnews.163.com
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040106544_11251',
> ENDKEY => '', ENCODED => 2135003529, OFFLINE => true, SPLIT => true, TABLE
> => {{NAME => 'webpage', FAMILIES => [{NAME => 'CF_CONTENT', COMPRESSION =>
> 'NONE', VERSIONS => '2', TTL => '2147483647', BLOCKSIZE => '65536',
> IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'CF_INFORMATION',
> COMPRESSION => 'NONE', VERSIONS => '1', TTL => '2147483647', BLOCKSIZE =>
> '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, new regions:
> webpage,http:\x2F\x2Fnews.163.com
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040106544_11251,1254040612500,
> webpage,http:\x2F\x2Fnews.163.com
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040237301_15439,1254040612500.
> Split took 4sec
> 2009-09-27 16:35:15,613 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
> webpage,http:\x2F\x2Fnews.163.com
>
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040106544_11251,1254040612500
> 2009-09-27 16:35:15,613 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
> webpage,http:\x2F\x2Fnews.163.com
>
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040237301_15439,1254040612500
> 2009-09-27 16:35:15,613 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Worker:
> MSG_REGION_OPEN:
> webpage,http:\x2F\x2Fnews.163.com
>
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040106544_11251,1254040612500
> 2009-09-27 16:35:15,769 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> region webpage,http:\x2F\x2Fnews.163.com
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040106544_11251,1254040612500/512381466
> available; sequence id is 140212
> 2009-09-27 16:35:15,770 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Worker:
> MSG_REGION_OPEN:
> webpage,http:\x2F\x2Fnews.163.com
>
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040237301_15439,1254040612500
> 2009-09-27 16:35:15,770 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Starting compaction on region webpage,http:\x2F\x2Fnews.163.com
>
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040106544_11251,1254040612500
> 2009-09-27 16:35:15,945 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> region webpage,http:\x2F\x2Fnews.163.com
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040237301_15439,1254040612500/95607190
> available; sequence id is 140213
> 2009-09-27 16:35:25,402 INFO org.apache.hadoop.hbase.regionserver.HLog:
> Roll
> /hbase/.logs/ubuntu9,60020,1254037278829/hlog.dat.1254040498496,
> entries=6194, calcsize=63761088, filesize=63102912. New hlog
> /hbase/.logs/ubuntu9,60020,1254037278829/hlog.dat.1254040525388
> 2009-09-27 16:35:25,967 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> compaction completed on region
> webpage,http:\x2F\x2Fnews.163.com
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040106544_11251,1254040612500
> in 10sec
> 2009-09-27 16:35:25,968 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Starting compaction on region webpage,http:\x2F\x2Fnews.163.com
>
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040237301_15439,1254040612500
> 2009-09-27 16:36:09,228 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC
> Server handler 3 on 60020 took 39592ms appending an edit to hlog;
> editcount=1415
> 2009-09-27 16:36:09,228 WARN org.apache.hadoop.hbase.regionserver.HLog:
> regionserver/192.168.33.5:60020.logFlusher took 36542ms optional sync'ing
> hlog; editcount=1421
> 2009-09-27 16:36:15,344 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC
> Server handler 3 on 60020 took 1053ms appending an edit to hlog;
> editcount=3575
> 2009-09-27 16:36:19,598 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> compaction completed on region
> webpage,http:\x2F\x2Fnews.163.com
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040237301_15439,1254040612500
> in 53sec
> 2009-09-27 16:36:19,598 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Starting split of region webpage,http:\x2F\x2Fnews.163.com
>
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040237301_15439,1254040612500
> 2009-09-27 16:36:19,606 WARN org.apache.hadoop.hbase.regionserver.HRegion:
> Clock skew; parent regions id is 1254040612500 but current time here is
> 1254040579606
> 2009-09-27 16:36:22,416 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Closed webpage,http:\x2F\x2Fnews.163.com
>
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040237301_15439,1254040612500
> 2009-09-27 16:36:22,966 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> region webpage,http:\x2F\x2Fnews.163.com
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040237301_15439,1254040612501/2088743043
> available; sequence id is 146389
> 2009-09-27 16:36:22,966 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Closed webpage,http:\x2F\x2Fnews.163.com
>
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040237301_15439,1254040612501
> 2009-09-27 16:36:23,092 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> region webpage,http:\x2F\x2Fnews.163.com
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040368751_18455,1254040612501/1245778272
> available; sequence id is 146390
> 2009-09-27 16:36:23,093 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Closed webpage,http:\x2F\x2Fnews.163.com
>
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040368751_18455,1254040612501
> 2009-09-27 16:36:23,109 INFO
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META
> updated, and report to master all successful. Old region=REGION => {NAME =>
> 'webpage,http:\x2F\x2Fnews.163.com
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040237301_15439,1254040612500',
> STARTKEY => 'http:\x2F\x2Fnews.163.com
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040237301_15439',
> ENDKEY => '', ENCODED => 95607190, OFFLINE => true, SPLIT => true, TABLE =>
> {{NAME => 'webpage', FAMILIES => [{NAME => 'CF_CONTENT', COMPRESSION =>
> 'NONE', VERSIONS => '2', TTL => '2147483647', BLOCKSIZE => '65536',
> IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'CF_INFORMATION',
> COMPRESSION => 'NONE', VERSIONS => '1', TTL => '2147483647', BLOCKSIZE =>
> '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, new regions:
> webpage,http:\x2F\x2Fnews.163.com
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040237301_15439,1254040612501,
> webpage,http:\x2F\x2Fnews.163.com
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040368751_18455,1254040612501.
> Split took 3sec
> 2009-09-27 16:37:34,798 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x123fa71ce2e0002 to sun.nio.ch.SelectionKeyImpl@29dc7e1f
> java.io.IOException: TIMED OUT
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
> 2009-09-27 16:37:34,820 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x123fa71ce2e0006 to sun.nio.ch.SelectionKeyImpl@6c96f1cb
> java.io.IOException: TIMED OUT
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
> 2009-09-27 16:37:34,899 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
> state: Disconnected, type: None, path: null
> 2009-09-27 16:37:35,275 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server ubuntu12/192.168.33.12:2222
> 2009-09-27 16:37:35,880 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server ubuntu12/192.168.33.12:2222
> 2009-09-27 16:37:38,281 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x123fa71ce2e0002 to sun.nio.ch.SelectionKeyImpl@41d77c9f
> java.net.NoRouteToHostException: No route to host
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
> 2009-09-27 16:37:38,281 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-09-27 16:37:38,282 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-09-27 16:37:38,282 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x123fa71ce2e0006 to sun.nio.ch.SelectionKeyImpl@2a36cc7e
> java.net.NoRouteToHostException: No route to host
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
> 2009-09-27 16:37:38,282 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-09-27 16:37:38,283 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-09-27 16:37:38,769 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server ubuntu10/192.168.33.10:2222
> 2009-09-27 16:37:39,111 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: ubuntu6/192.168.33.7:9000. Already tried 0 time(s).
> 2009-09-27 16:37:39,158 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server ubuntu10/192.168.33.10:2222
> 2009-09-27 16:37:52,108 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x123fa71ce2e0006 to sun.nio.ch.SelectionKeyImpl@73ab79d8
> java.io.IOException: TIMED OUT
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
> 2009-09-27 16:37:52,109 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown output
> java.net.SocketException: Transport endpoint is not connected
> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-09-27 16:37:52,498 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x123fa71ce2e0002 to sun.nio.ch.SelectionKeyImpl@3305fed5
> java.io.IOException: TIMED OUT
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
> 2009-09-27 16:37:52,498 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown output
> java.net.SocketException: Transport endpoint is not connected
> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-09-27 16:37:53,857 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server ubuntu2/192.168.33.9:2222
> 2009-09-27 16:37:53,938 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server ubuntu2/192.168.33.9:2222
> 2009-09-27 16:38:00,121 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: ubuntu6/192.168.33.7:9000. Already tried 1 time(s).
> 2009-09-27 16:38:04,502 INFO org.apache.zookeeper.ClientCnxn: Priming
> connection to java.nio.channels.SocketChannel[connected local=/
> 192.168.33.5:37122 remote=ubuntu2/192.168.33.9:2222]
> 2009-09-27 16:38:04,502 INFO org.apache.zookeeper.ClientCnxn: Server
> connection successful
> 2009-09-27 16:38:04,503 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x123fa71ce2e0002 to sun.nio.ch.SelectionKeyImpl@41254de4
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcher.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
> at sun.nio.ch.IOUtil.read(IOUtil.java:200)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
> at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:651)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
> 2009-09-27 16:38:04,504 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown input
> java.net.SocketException: Transport endpoint is not connected
> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-09-27 16:38:04,505 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown output
> java.net.SocketException: Transport endpoint is not connected
> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-09-27 16:38:04,513 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x123fa71ce2e0006 to sun.nio.ch.SelectionKeyImpl@4e9d6c1e
> java.net.SocketException: Connection reset by peer
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
> 2009-09-27 16:38:04,513 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-09-27 16:38:04,514 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-09-27 16:38:04,693 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server ubuntu12/192.168.33.12:2222
> 2009-09-27 16:38:04,693 INFO org.apache.zookeeper.ClientCnxn: Priming
> connection to java.nio.channels.SocketChannel[connected local=/
> 192.168.33.5:45155 remote=ubuntu12/192.168.33.12:2222]
> 2009-09-27 16:38:04,694 INFO org.apache.zookeeper.ClientCnxn: Server
> connection successful
> 2009-09-27 16:38:04,695 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
> state: Expired, type: None, path: null
> 2009-09-27 16:38:04,695 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x123fa71ce2e0002 to sun.nio.ch.SelectionKeyImpl@3eac3b6
> java.io.IOException: Session Expired
> at
>
> org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:548)
> at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:661)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
> 2009-09-27 16:38:04,695 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session
> expired
> 2009-09-27 16:38:04,699 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
> request=0.0, regions=5, stores=9, storefiles=9, storefileIndexSize=0,
> memstoreSize=0, usedHeap=104, maxHeap=2993, blockCacheSize=5152344,
> blockCacheFree=622695656, blockCacheCount=1, blockCacheHitRatio=0
> 2009-09-27 16:38:05,092 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server ubuntu12/192.168.33.12:2222
> 2009-09-27 16:38:05,093 INFO org.apache.zookeeper.ClientCnxn: Priming
> connection to java.nio.channels.SocketChannel[connected local=/
> 192.168.33.5:45156 remote=ubuntu12/192.168.33.12:2222]
> 2009-09-27 16:38:05,093 INFO org.apache.zookeeper.ClientCnxn: Server
> connection successful
> 2009-09-27 16:38:05,094 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x123fa71ce2e0006 to sun.nio.ch.SelectionKeyImpl@1e8516be
> java.io.IOException: Session Expired
> at
>
> org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:548)
> at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:661)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
> 2009-09-27 16:38:05,094 INFO org.apache.zookeeper.ZooKeeper: Closing
> session: 0x123fa71ce2e0006
> 2009-09-27 16:38:05,095 INFO org.apache.zookeeper.ClientCnxn: Closing
> ClientCnxn for session: 0x123fa71ce2e0006
> 2009-09-27 16:38:05,095 INFO org.apache.zookeeper.ClientCnxn: Disconnecting
> ClientCnxn for session: 0x123fa71ce2e0006
> 2009-09-27 16:38:05,095 INFO org.apache.zookeeper.ZooKeeper: Session:
> 0x123fa71ce2e0006 closed
> 2009-09-27 16:38:05,096 INFO org.apache.zookeeper.ClientCnxn: EventThread
> shut down
> 2009-09-27 16:38:05,297 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 5 on 60020, call openScanner([B@6d3c1c92, startRow=, stopRow=,
> maxVersions=1, caching=1, timeRange=[0,9223372036854775807),
> families={(family=info, columns={}}) from 192.168.33.7:60950: error:
> java.io.IOException: Server not running, aborting
> java.io.IOException: Server not running, aborting
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1865)
> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> 2009-09-27 16:38:05,405 INFO
> org.apache.hadoop.hbase.regionserver.LogRoller:
> LogRoller exiting.
> 2009-09-27 16:38:05,952 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: worker thread exiting
> 2009-09-27 16:38:07,249 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 4 on 60020, call getClosestRowBefore([B@47a4fd1, [B@7a0e24a2,
> [B@36874648) from 192.168.33.7:60956: error: java.io.IOException: Server
> not
> running, aborting
> java.io.IOException: Server not running, aborting
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1748)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> 2009-09-27 16:38:07,511 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> server on 60020
> 2009-09-27 16:38:07,512 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 0 on 60020: exiting
> 2009-09-27 16:38:07,512 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 5 on 60020: exiting
> 2009-09-27 16:38:07,520 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 7 on 60020: exiting
> 2009-09-27 16:38:07,520 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 2 on 60020: exiting
> 2009-09-27 16:38:07,521 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
> 2009-09-27 16:38:07,521 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> IPC
> Server Responder
> 2009-09-27 16:38:07,522 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 9 on 60020: exiting
> 2009-09-27 16:38:07,523 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 8 on 60020: exiting
> 2009-09-27 16:38:07,523 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 6 on 60020: exiting
> 2009-09-27 16:38:07,524 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 3 on 60020: exiting
> 2009-09-27 16:38:07,524 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 1 on 60020: exiting
> 2009-09-27 16:38:07,539 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 4 on 60020: exiting
> 2009-09-27 16:38:07,562 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> IPC
> Server listener on 60020
> 2009-09-27 16:38:07,683 INFO
> org.apache.hadoop.hbase.regionserver.LogFlusher:
> regionserver/192.168.33.5:60020.logFlusher exiting
> 2009-09-27 16:38:07,683 INFO
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher:
> regionserver/192.168.33.5:60020.cacheFlusher exiting
> 2009-09-27 16:38:07,684 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer$MajorCompactionChecker:
> regionserver/192.168.33.5:60020.majorCompactionChecker exiting
> 2009-09-27 16:38:07,687 INFO
> org.apache.hadoop.hbase.regionserver.CompactSplitThread:
> regionserver/192.168.33.5:60020.compactor exiting
> 2009-09-27 16:38:07,726 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: On abort, closed hlog
> 2009-09-27 16:38:07,733 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Closed webpage,http:\x2F\x2Fnews.163.com
>
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040106544_11251,1254040612500
> 2009-09-27 16:38:07,733 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Closed webpage,http:\x2F\x2Fnews.163.com
>
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039598290_2497,1254040068963
> 2009-09-27 16:38:07,733 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Closed -ROOT-,,0
> 2009-09-27 16:38:07,734 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Closed webpage,http:\x2F\x2Fnews.163.com
>
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039498012_112,1254040068962
> 2009-09-27 16:38:07,734 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Closed webpage,http:\x2F\x2Fnews.163.com
>
> \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040043019_9471,1254040612499
> 2009-09-27 16:38:07,734 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: aborting server at:
> 192.168.33.5:60020
> 2009-09-27 16:38:07,734 INFO org.apache.zookeeper.ZooKeeper: Closing
> session: 0x123fa71ce2e0002
> 2009-09-27 16:38:07,734 INFO org.apache.zookeeper.ClientCnxn: Closing
> ClientCnxn for session: 0x123fa71ce2e0002
> 2009-09-27 16:38:07,735 INFO org.apache.zookeeper.ClientCnxn: Disconnecting
> ClientCnxn for session: 0x123fa71ce2e0002
> 2009-09-27 16:38:07,735 INFO org.apache.zookeeper.ZooKeeper: Session:
> 0x123fa71ce2e0002 closed
> 2009-09-27 16:38:07,735 INFO org.apache.zookeeper.ClientCnxn: EventThread
> shut down
> 2009-09-27 16:38:07,837 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver/
> 192.168.33.5:60020 exiting
> 2009-09-27 16:38:09,670 INFO org.apache.hadoop.hbase.Leases:
> regionserver/192.168.33.5:60020.leaseChecker closing leases
> 2009-09-27 16:38:09,670 INFO org.apache.hadoop.hbase.Leases:
> regionserver/192.168.33.5:60020.leaseChecker closed leases
> 2009-09-27 16:38:09,672 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Starting shutdown
> thread.
> 2009-09-27 16:38:09,673 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Shutdown thread
> complete
>
Re: is there any problem with our environment?
Posted by stack <st...@duboce.net>.
Are you using IPs? Try using hostnames?
St.Ack
2009/9/27 Zheng Lv <lv...@gmail.com>
> following is the logs in another one of shutdown regionservers:
>
>
> 2009年 09月 27日 星期日 15:40:20 CST Starting regionserver on ubuntu7
> ulimit -n 1024
> 2009-09-27 15:40:20,951 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> vmInputArguments=[-Xmx3000m, -Djava.net.preferIPv4Stack=true,
> -XX:+HeapDumpOnOutOfMemoryError, -XX:+UseConcMarkSweepGC,
> -XX:+CMSIncrementalMode, -Dhbase.log.dir=/opt/hadoop/hbase/bin/../logs,
> -Dhbase.log.file=hbase-cyd-regionserver-ubuntu7.log,
> -Dhbase.home.dir=/opt/hadoop/hbase/bin/.., -Dhbase.id.str=cyd,
> -Dhbase.root.logger=INFO,DRFA,
> -Djava.library.path=/opt/hadoop/hbase/bin/../lib/native/Linux-amd64-64]
> 2009-09-27 15:40:20,961 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: My address is
> ubuntu7:60020
> 2009-09-27 15:40:21,003 INFO org.apache.hadoop.hbase.ipc.HBaseRpcMetrics:
> Initializing RPC Metrics with hostName=HRegionServer, port=60020
> 2009-09-27 15:40:21,011 INFO
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher:
> globalMemStoreLimit=1.2g, globalMemStoreLimitLowMark=748.5m, maxHeap=2.9g
> 2009-09-27 15:40:21,013 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Runs every 10000000ms
> 2009-09-27 15:40:21,027 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:zookeeper.version=3.2.0--1, built on 05/15/2009 06:05 GMT
> 2009-09-27 15:40:21,027 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:host.name=ubuntu7
> 2009-09-27 15:40:21,027 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.version=1.6.0_14
> 2009-09-27 15:40:21,027 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.vendor=Sun Microsystems Inc.
> 2009-09-27 15:40:21,027 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.home=/usr/lib/jvm/java-6-sun-1.6.0.14/jre
> 2009-09-27 15:40:21,027 INFO org.apache.zookeeper.ZooKeeper: Client
>
> environment:java.class.path=/home/cyd/conf/hbase:/usr/lib/jvm/java-6-sun/lib/tools.jar:/opt/hadoop/hbase/bin/..:/opt/hadoop/hbase/bin/../hbase-0.20.0.jar:/opt/hadoop/hbase/bin/../lib/AgileJSON-2009-03-30.jar:/opt/hadoop/hbase/bin/../lib/commons-cli-2.0-SNAPSHOT.jar:/opt/hadoop/hbase/bin/../lib/commons-el-from-jetty-5.1.4.jar:/opt/hadoop/hbase/bin/../lib/commons-httpclient-3.0.1.jar:/opt/hadoop/hbase/bin/../lib/commons-logging-1.0.4.jar:/opt/hadoop/hbase/bin/../lib/commons-logging-api-1.0.4.jar:/opt/hadoop/hbase/bin/../lib/commons-math-1.1.jar:/opt/hadoop/hbase/bin/../lib/hadoop-0.20.0-plus4681-core.jar:/opt/hadoop/hbase/bin/../lib/hadoop-0.20.0-test.jar:/opt/hadoop/hbase/bin/../lib/jasper-compiler-5.5.12.jar:/opt/hadoop/hbase/bin/../lib/jasper-runtime-5.5.12.jar:/opt/hadoop/hbase/bin/../lib/jetty-6.1.14.jar:/opt/hadoop/hbase/bin/../lib/jetty-util-6.1.14.jar:/opt/hadoop/hbase/bin/../lib/jruby-complete-1.2.0.jar:/opt/hadoop/hbase/bin/../lib/json.jar:/opt/hadoop/hbase/bin/../lib/junit-3.8.1.jar:/opt/hadoop/hbase/bin/../lib/libthrift-r771587.jar:/opt/hadoop/hbase/bin/../lib/log4j-1.2.15.jar:/opt/hadoop/hbase/bin/../lib/lucene-core-2.2.0.jar:/opt/hadoop/hbase/bin/../lib/servlet-api-2.5-6.1.14.jar:/opt/hadoop/hbase/bin/../lib/xmlenc-0.52.jar:/opt/hadoop/hbase/bin/../lib/zookeeper-r785019-hbase-1329.jar:/opt/hadoop/hbase/bin/../lib/jsp-2.1/jsp-2.1.jar:/opt/hadoop/hbase/bin/../lib/jsp-2.1/jsp-api-2.1.jar
> 2009-09-27 15:40:21,027 INFO org.apache.zookeeper.ZooKeeper: Client
>
> environment:java.library.path=/opt/hadoop/hbase/bin/../lib/native/Linux-amd64-64
> 2009-09-27 15:40:21,027 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.io.tmpdir=/tmp
> 2009-09-27 15:40:21,028 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.compiler=<NA>
> 2009-09-27 15:40:21,028 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:os.name=Linux
> 2009-09-27 15:40:21,028 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:os.arch=amd64
> 2009-09-27 15:40:21,028 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:os.version=2.6.28-11-server
> 2009-09-27 15:40:21,028 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:user.name=cyd
> 2009-09-27 15:40:21,028 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:user.home=/home/cyd
> 2009-09-27 15:40:21,028 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:user.dir=/opt/hadoop/hbase-0.20.0
> 2009-09-27 15:40:21,029 INFO org.apache.zookeeper.ZooKeeper: Initiating
> client connection, host=ubuntu12:2222,ubuntu10:2222,ubuntu2:2222
> sessionTimeout=240000
> watcher=org.apache.hadoop.hbase.regionserver.HRegionServer@6719dc16
> 2009-09-27 15:40:21,030 INFO org.apache.zookeeper.ClientCnxn:
> zookeeper.disableAutoWatchReset is false
> 2009-09-27 15:40:21,042 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server ubuntu10/192.168.33.10:2222
> 2009-09-27 15:40:21,048 INFO org.apache.zookeeper.ClientCnxn: Priming
> connection to java.nio.channels.SocketChannel[connected local=/
> 192.168.33.6:36316 remote=ubuntu10/192.168.33.10:2222]
> 2009-09-27 15:40:21,060 INFO org.apache.zookeeper.ClientCnxn: Server
> connection successful
> 2009-09-27 15:40:21,138 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
> state: SyncConnected, type: None, path: null
> 2009-09-27 15:40:21,362 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to read master
> address from ZooKeeper. Retrying. Error was:
> java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode for /hbase/master
> at
>
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:331)
> at
>
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:240)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1316)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1348)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:428)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode for /hbase/master
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:750)
> at
>
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:327)
> ... 5 more
> 2009-09-27 15:40:21,455 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
> state: SyncConnected, type: NodeCreated, path: /hbase/master
> 2009-09-27 15:40:21,469 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Telling master at
> 192.168.33.7:60000 that we are up
> 2009-09-27 15:40:24,368 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Telling master at
> 192.168.33.7:60000 that we are up
> 2009-09-27 15:41:03,732 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Master passed us
> address
> to use. Was=192.168.33.6:60020, Now=192.168.33.6
> 2009-09-27 15:41:04,267 INFO org.apache.hadoop.hbase.regionserver.HLog:
> HLog
> configuration: blocksize=67108864, rollsize=63753420, enabled=true,
> flushlogentries=100, optionallogflushinternal=10000ms
> 2009-09-27 15:41:04,468 INFO org.apache.hadoop.hbase.regionserver.HLog: New
> hlog /hbase/.logs/ubuntu7,60020,1254037263642/hlog.dat.1254037264267
> 2009-09-27 15:41:04,477 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=RegionServer,
> sessionId=regionserver/192.168.33.6:60020
> 2009-09-27 15:41:04,478 INFO
> org.apache.hadoop.hbase.regionserver.metrics.RegionServerMetrics:
> Initialized
> 2009-09-27 15:41:04,818 INFO org.apache.hadoop.http.HttpServer: Jetty bound
> to port 60030
> 2009-09-27 15:41:05,167 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> Responder: starting
> 2009-09-27 15:41:05,167 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> listener on 60020: starting
> 2009-09-27 15:41:05,168 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 0 on 60020: starting
> 2009-09-27 15:41:05,168 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 1 on 60020: starting
> 2009-09-27 15:41:05,168 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 2 on 60020: starting
> 2009-09-27 15:41:05,168 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 3 on 60020: starting
> 2009-09-27 15:41:05,169 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 4 on 60020: starting
> 2009-09-27 15:41:05,169 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 5 on 60020: starting
> 2009-09-27 15:41:05,169 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 6 on 60020: starting
> 2009-09-27 15:41:05,169 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 7 on 60020: starting
> 2009-09-27 15:41:05,169 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 8 on 60020: starting
> 2009-09-27 15:41:05,169 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 9 on 60020: starting
> 2009-09-27 15:41:05,169 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: HRegionServer started
> at: 192.168.33.6:60020
> 2009-09-27 15:41:05,198 INFO
> org.apache.hadoop.hbase.regionserver.StoreFile:
> Allocating LruBlockCache with maximum size 598.8m
> 2009-09-27 15:41:08,229 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
> .META.,,1
> 2009-09-27 15:41:08,237 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Worker:
> MSG_REGION_OPEN:
> .META.,,1
> 2009-09-27 15:41:08,333 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> region .META.,,1/1028785192 available; sequence id is 0
> 2009-09-27 15:50:48,238 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: compactions no longer
> limited
> 2009-09-27 16:19:49,375 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x123fa71ce2e0000 to sun.nio.ch.SelectionKeyImpl@7745c859
> java.io.IOException: TIMED OUT
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
> 2009-09-27 16:19:49,476 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
> state: Disconnected, type: None, path: null
> 2009-09-27 16:19:50,182 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server ubuntu12/192.168.33.12:2222
> 2009-09-27 16:19:53,188 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x123fa71ce2e0000 to sun.nio.ch.SelectionKeyImpl@16e334a
> java.net.NoRouteToHostException: No route to host
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
> 2009-09-27 16:19:53,189 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-09-27 16:19:53,189 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-09-27 16:19:53,919 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server ubuntu2/192.168.33.9:2222
> 2009-09-27 16:19:56,018 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: ubuntu6/192.168.33.7:9000. Already tried 0 time(s).
> 2009-09-27 16:19:56,928 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x123fa71ce2e0000 to sun.nio.ch.SelectionKeyImpl@506937f9
> java.net.NoRouteToHostException: No route to host
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
> 2009-09-27 16:19:56,928 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-09-27 16:19:56,928 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-09-27 16:19:58,605 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server ubuntu10/192.168.33.10:2222
> 2009-09-27 16:20:01,617 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x123fa71ce2e0000 to sun.nio.ch.SelectionKeyImpl@6e75d758
> java.net.NoRouteToHostException: No route to host
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
> 2009-09-27 16:20:01,618 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-09-27 16:20:01,618 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-09-27 16:20:02,323 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server ubuntu12/192.168.33.12:2222
> 2009-09-27 16:20:05,328 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x123fa71ce2e0000 to sun.nio.ch.SelectionKeyImpl@7a8d0b0f
> java.net.NoRouteToHostException: No route to host
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
> 2009-09-27 16:20:05,328 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-09-27 16:20:05,329 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-09-27 16:20:05,879 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server ubuntu2/192.168.33.9:2222
> 2009-09-27 16:20:08,887 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x123fa71ce2e0000 to sun.nio.ch.SelectionKeyImpl@41d144a4
> java.net.NoRouteToHostException: No route to host
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
> 2009-09-27 16:20:08,888 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-09-27 16:20:08,889 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-09-27 16:20:10,535 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server ubuntu10/192.168.33.10:2222
> 2009-09-27 16:20:13,527 INFO org.apache.zookeeper.ClientCnxn: Priming
> connection to java.nio.channels.SocketChannel[connected local=/
> 192.168.33.6:60330 remote=ubuntu10/192.168.33.10:2222]
> 2009-09-27 16:20:13,528 INFO org.apache.zookeeper.ClientCnxn: Server
> connection successful
> 2009-09-27 16:20:13,532 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
> state: SyncConnected, type: None, path: null
> 2009-09-27 16:21:33,555 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x123fa71ce2e0000 to sun.nio.ch.SelectionKeyImpl@11f18cbe
> java.io.IOException: TIMED OUT
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
> 2009-09-27 16:21:33,656 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
> state: Disconnected, type: None, path: null
> 2009-09-27 16:21:33,713 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server ubuntu12/192.168.33.12:2222
> 2009-09-27 16:21:47,047 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x123fa71ce2e0000 to sun.nio.ch.SelectionKeyImpl@66e8c7db
> java.io.IOException: TIMED OUT
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
> 2009-09-27 16:21:47,048 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown output
> java.net.SocketException: Transport endpoint is not connected
> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-09-27 16:21:47,227 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server ubuntu2/192.168.33.9:2222
> 2009-09-27 16:22:00,566 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x123fa71ce2e0000 to sun.nio.ch.SelectionKeyImpl@6d0718b7
> java.io.IOException: TIMED OUT
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
> 2009-09-27 16:22:00,567 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown output
> java.net.SocketException: Transport endpoint is not connected
> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-09-27 16:22:02,264 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
> (Retry: 0)
> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
> at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
> at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
> at java.lang.Thread.run(Thread.java:619)
> 2009-09-27 16:22:02,271 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
> (Retry: 1)
> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
> at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
> at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
> at java.lang.Thread.run(Thread.java:619)
> 2009-09-27 16:22:02,276 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
> (Retry: 2)
> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
> at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
> at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
> at java.lang.Thread.run(Thread.java:619)
> 2009-09-27 16:22:02,281 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
> (Retry: 3)
> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
> at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
> at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
> at java.lang.Thread.run(Thread.java:619)
> 2009-09-27 16:22:02,286 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
> (Retry: 4)
> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
> at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
> at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
> at java.lang.Thread.run(Thread.java:619)
> 2009-09-27 16:22:02,290 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
> (Retry: 5)
> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
> at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
> at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
> at java.lang.Thread.run(Thread.java:619)
> 2009-09-27 16:22:02,294 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server ubuntu10/192.168.33.10:2222
> 2009-09-27 16:22:02,295 INFO org.apache.zookeeper.ClientCnxn: Priming
> connection to java.nio.channels.SocketChannel[connected local=/
> 192.168.33.6:60343 remote=ubuntu10/192.168.33.10:2222]
> 2009-09-27 16:22:02,295 INFO org.apache.zookeeper.ClientCnxn: Server
> connection successful
> 2009-09-27 16:22:02,296 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
> (Retry: 6)
> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
> at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
> at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
> at java.lang.Thread.run(Thread.java:619)
> 2009-09-27 16:22:02,298 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x123fa71ce2e0000 to sun.nio.ch.SelectionKeyImpl@23bf8cd5
> java.io.IOException: Session Expired
> at
>
> org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:548)
> at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:661)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
> 2009-09-27 16:22:02,298 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
> state: Expired, type: None, path: null
> 2009-09-27 16:22:02,299 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session
> expired
> 2009-09-27 16:22:02,301 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
> (Retry: 7)
> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
> at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
> at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
> at java.lang.Thread.run(Thread.java:619)
> 2009-09-27 16:22:02,303 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
> request=0.0, regions=1, stores=2, storefiles=0, storefileIndexSize=0,
> memstoreSize=0, usedHeap=29, maxHeap=2993, blockCacheSize=5151568,
> blockCacheFree=622696432, blockCacheCount=0, blockCacheHitRatio=0
> 2009-09-27 16:22:02,307 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
> (Retry: 8)
> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
> at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
> at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
> at java.lang.Thread.run(Thread.java:619)
> 2009-09-27 16:22:02,307 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Stop was requested,
> clearing the toDo despite of the exception
> 2009-09-27 16:22:02,308 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> server on 60020
> 2009-09-27 16:22:02,308 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 0 on 60020: exiting
> 2009-09-27 16:22:02,309 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 1 on 60020: exiting
> 2009-09-27 16:22:02,309 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 3 on 60020: exiting
> 2009-09-27 16:22:02,310 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> IPC
> Server listener on 60020
> 2009-09-27 16:22:02,310 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> IPC
> Server Responder
> 2009-09-27 16:22:02,310 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
> 2009-09-27 16:22:02,325 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 4 on 60020: exiting
> 2009-09-27 16:22:02,325 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 5 on 60020: exiting
> 2009-09-27 16:22:02,328 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 6 on 60020: exiting
> 2009-09-27 16:22:02,329 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 7 on 60020: exiting
> 2009-09-27 16:22:02,329 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 8 on 60020: exiting
> 2009-09-27 16:22:02,330 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 2 on 60020: exiting
> 2009-09-27 16:22:02,330 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 9 on 60020: exiting
> 2009-09-27 16:22:02,465 INFO
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher:
> regionserver/192.168.33.6:60020.cacheFlusher exiting
> 2009-09-27 16:22:02,465 INFO
> org.apache.hadoop.hbase.regionserver.LogFlusher:
> regionserver/192.168.33.6:60020.logFlusher exiting
> 2009-09-27 16:22:02,465 INFO
> org.apache.hadoop.hbase.regionserver.LogRoller:
> LogRoller exiting.
> 2009-09-27 16:22:02,466 INFO
> org.apache.hadoop.hbase.regionserver.CompactSplitThread:
> regionserver/192.168.33.6:60020.compactor exiting
> 2009-09-27 16:22:02,466 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer$MajorCompactionChecker:
> regionserver/192.168.33.6:60020.majorCompactionChecker exiting
> 2009-09-27 16:22:02,478 INFO org.apache.hadoop.hdfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
> /hbase/.logs/ubuntu7,60020,1254037263642/hlog.dat.1254037264267 File does
> not exist. Holder DFSClient_679922832 does not have any open files.
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1317)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1308)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1236)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> at org.apache.hadoop.ipc.Client.call(Client.java:739)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at $Proxy1.addBlock(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at $Proxy1.addBlock(Unknown Source)
> at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2875)
> at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2757)
> at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2048)
> at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2234)
> 2009-09-27 16:22:02,478 WARN org.apache.hadoop.hdfs.DFSClient:
> NotReplicatedYetException sleeping
> /hbase/.logs/ubuntu7,60020,1254037263642/hlog.dat.1254037264267 retries
> left
> 4
> 2009-09-27 16:22:02,883 INFO org.apache.hadoop.hdfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
> /hbase/.logs/ubuntu7,60020,1254037263642/hlog.dat.1254037264267 File does
> not exist. Holder DFSClient_679922832 does not have any open files.
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1317)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1308)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1236)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> at org.apache.hadoop.ipc.Client.call(Client.java:739)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at $Proxy1.addBlock(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at $Proxy1.addBlock(Unknown Source)
> at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2875)
> at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2757)
> at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2048)
> at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2234)
> 2009-09-27 16:22:02,883 WARN org.apache.hadoop.hdfs.DFSClient:
> NotReplicatedYetException sleeping
> /hbase/.logs/ubuntu7,60020,1254037263642/hlog.dat.1254037264267 retries
> left
> 3
> 2009-09-27 16:22:03,687 INFO org.apache.hadoop.hdfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
> /hbase/.logs/ubuntu7,60020,1254037263642/hlog.dat.1254037264267 File does
> not exist. Holder DFSClient_679922832 does not have any open files.
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1317)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1308)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1236)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> at org.apache.hadoop.ipc.Client.call(Client.java:739)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at $Proxy1.addBlock(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at $Proxy1.addBlock(Unknown Source)
> at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2875)
> at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2757)
> at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2048)
> at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2234)
> 2009-09-27 16:22:03,687 WARN org.apache.hadoop.hdfs.DFSClient:
> NotReplicatedYetException sleeping
> /hbase/.logs/ubuntu7,60020,1254037263642/hlog.dat.1254037264267 retries
> left
> 2
> 2009-09-27 16:22:04,520 INFO org.apache.hadoop.hbase.Leases:
> regionserver/192.168.33.6:60020.leaseChecker closing leases
> 2009-09-27 16:22:04,521 INFO org.apache.hadoop.hbase.Leases:
> regionserver/192.168.33.6:60020.leaseChecker closed leases
> 2009-09-27 16:22:05,292 INFO org.apache.hadoop.hdfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
> /hbase/.logs/ubuntu7,60020,1254037263642/hlog.dat.1254037264267 File does
> not exist. Holder DFSClient_679922832 does not have any open files.
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1317)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1308)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1236)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> at org.apache.hadoop.ipc.Client.call(Client.java:739)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at $Proxy1.addBlock(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at $Proxy1.addBlock(Unknown Source)
> at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2875)
> at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2757)
> at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2048)
> at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2234)
> 2009-09-27 16:22:05,292 WARN org.apache.hadoop.hdfs.DFSClient:
> NotReplicatedYetException sleeping
> /hbase/.logs/ubuntu7,60020,1254037263642/hlog.dat.1254037264267 retries
> left
> 1
> 2009-09-27 16:22:08,370 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: worker thread exiting
> 2009-09-27 16:22:08,497 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer
> Exception: org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
> /hbase/.logs/ubuntu7,60020,1254037263642/hlog.dat.1254037264267 File does
> not exist. Holder DFSClient_679922832 does not have any open files.
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1317)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1308)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1236)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> at org.apache.hadoop.ipc.Client.call(Client.java:739)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at $Proxy1.addBlock(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at $Proxy1.addBlock(Unknown Source)
> at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2875)
> at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2757)
> at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2048)
> at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2234)
> 2009-09-27 16:22:08,498 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block null bad datanode[0] nodes == null
> 2009-09-27 16:22:08,498 WARN org.apache.hadoop.hdfs.DFSClient: Could not
> get
> block locations. Source file
> "/hbase/.logs/ubuntu7,60020,1254037263642/hlog.dat.1254037264267" -
> Aborting...
> 2009-09-27 16:22:08,499 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to close log in
> abort
> org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException:
> org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
> /hbase/.logs/ubuntu7,60020,1254037263642/hlog.dat.1254037264267 File does
> not exist. Holder DFSClient_679922832 does not have any open files.
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1317)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1308)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1236)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
> at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:637)
> at java.lang.Thread.run(Thread.java:619)
> 2009-09-27 16:22:08,501 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Closed .META.,,1
> 2009-09-27 16:22:08,502 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: aborting server at:
> 192.168.33.6:60020
> 2009-09-27 16:22:08,502 INFO org.apache.zookeeper.ZooKeeper: Closing
> session: 0x123fa71ce2e0000
> 2009-09-27 16:22:08,502 INFO org.apache.zookeeper.ClientCnxn: Closing
> ClientCnxn for session: 0x123fa71ce2e0000
> 2009-09-27 16:22:08,502 INFO org.apache.zookeeper.ClientCnxn: Disconnecting
> ClientCnxn for session: 0x123fa71ce2e0000
> 2009-09-27 16:22:08,502 INFO org.apache.zookeeper.ZooKeeper: Session:
> 0x123fa71ce2e0000 closed
> 2009-09-27 16:22:08,503 INFO org.apache.zookeeper.ClientCnxn: EventThread
> shut down
> 2009-09-27 16:22:08,505 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver/
> 192.168.33.6:60020 exiting
> 2009-09-27 16:22:08,507 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Starting shutdown
> thread.
> 2009-09-27 16:22:08,507 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Shutdown thread
> complete
>
Re: is there any problem with our environment?
Posted by Zheng Lv <lv...@gmail.com>.
no one can give some suggestions??
2009/9/27 Zheng Lv <lv...@gmail.com>
> Any suggestion?
> Thank you.
>
Re: is there any problem with our environment?
Posted by Zheng Lv <lv...@gmail.com>.
Any suggestion?
Thank you.
Re: is there any problem with our environment?
Posted by Zheng Lv <lv...@gmail.com>.
following is the logs of my program:
09/09/27 16:19:10 INFO zookeeper.ZooKeeper: Client
environment:zookeeper.version=3.2.0--1, built on 05/15/2009 06:05 GMT
09/09/27 16:19:10 INFO zookeeper.ZooKeeper: Client environment:host.name
=ubuntu6
09/09/27 16:19:10 INFO zookeeper.ZooKeeper: Client
environment:java.version=1.6.0_14
09/09/27 16:19:10 INFO zookeeper.ZooKeeper: Client
environment:java.vendor=Sun Microsystems Inc.
09/09/27 16:19:10 INFO zookeeper.ZooKeeper: Client
environment:java.home=/usr/lib/jvm/java-6-sun-1.6.0.14/jre
09/09/27 16:19:10 INFO zookeeper.ZooKeeper: Client
environment:java.class.path=/home/cyd/conf/hbase/:./hbasetest.jar:./commons-logging-1.0.4.jar:./zookeeper-r785019-hbase-1329.jar:./json.jar:./commons-cli-2.0-SNAPSHOT.jar:./servlet-api-2.5-6.1.14.jar:./jruby-complete-1.2.0.jar:./commons-math-1.1.jar:./AgileJSON-2009-03-30.jar:./lucene-core-2.2.0.jar:./commons-logging-api-1.0.4.jar:./jasper-runtime-5.5.12.jar:./jetty-util-6.1.14.jar:./hbase-0.20.0.jar:./commons-httpclient-3.0.1.jar:./hadoop-0.20.0-plus4681-core.jar:./libthrift-r771587.jar:./hadoop-0.20.0-test.jar:./jasper-compiler-5.5.12.jar:./junit-3.8.1.jar:./xmlenc-0.52.jar:./log4j-1.2.15.jar:./commons-el-from-jetty-5.1.4.jar:./jetty-6.1.14.jar
09/09/27 16:19:10 INFO zookeeper.ZooKeeper: Client
environment:java.library.path=/usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/server:/usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64:/usr/lib/jvm/java-6-sun-1.6.0.14/jre/../lib/amd64:/usr/java/packages/lib/amd64:/lib:/usr/lib
09/09/27 16:19:10 INFO zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=/tmp
09/09/27 16:19:10 INFO zookeeper.ZooKeeper: Client
environment:java.compiler=<NA>
09/09/27 16:19:10 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
09/09/27 16:19:10 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
09/09/27 16:19:10 INFO zookeeper.ZooKeeper: Client
environment:os.version=2.6.28-14-server
09/09/27 16:19:10 INFO zookeeper.ZooKeeper: Client environment:user.name=cyd
09/09/27 16:19:10 INFO zookeeper.ZooKeeper: Client
environment:user.home=/home/cyd
09/09/27 16:19:10 INFO zookeeper.ZooKeeper: Client
environment:user.dir=/usr/hbase2test
09/09/27 16:19:10 INFO zookeeper.ZooKeeper: Initiating client connection,
host=ubuntu12:2222,ubuntu10:2222,ubuntu2:2222 sessionTimeout=240000
watcher=org.apache.hadoop.hbase.client.HConnectionManager$ClientZKWatcher@2c41d05d
09/09/27 16:19:10 INFO zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset
is false
09/09/27 16:19:10 INFO zookeeper.ClientCnxn: Attempting connection to server
ubuntu2/192.168.33.9:2222
09/09/27 16:19:10 INFO zookeeper.ClientCnxn: Priming connection to
java.nio.channels.SocketChannel[connected
local=/192.168.33.7:43687remote=ubuntu2/
192.168.33.9:2222]
09/09/27 16:19:10 INFO zookeeper.ClientCnxn: Server connection successful
insertting 0 webpages need 697 ms.
insertting 1000 webpages need 16336 ms.
insertting 2000 webpages need 38556 ms.
insertting 3000 webpages need 55451 ms.
insertting 4000 webpages need 116184 ms.
insertting 5000 webpages need 170163 ms.
09/09/27 16:24:15 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.6:60020. Already tried 0 time(s).
09/09/27 16:24:16 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.6:60020. Already tried 1 time(s).
09/09/27 16:24:17 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.6:60020. Already tried 2 time(s).
09/09/27 16:24:18 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.6:60020. Already tried 3 time(s).
09/09/27 16:24:19 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.6:60020. Already tried 4 time(s).
09/09/27 16:24:20 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.6:60020. Already tried 5 time(s).
09/09/27 16:24:21 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.6:60020. Already tried 6 time(s).
09/09/27 16:24:22 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.6:60020. Already tried 7 time(s).
09/09/27 16:24:23 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.6:60020. Already tried 8 time(s).
09/09/27 16:24:24 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.6:60020. Already tried 9 time(s).
insertting 6000 webpages need 357611 ms.
insertting 7000 webpages need 373057 ms.
insertting 8000 webpages need 389643 ms.
insertting 9000 webpages need 411328 ms.
insertting 10000 webpages need 501814 ms.
insertting 11000 webpages need 552638 ms.
insertting 12000 webpages need 566738 ms.
insertting 13000 webpages need 604007 ms.
insertting 14000 webpages need 631419 ms.
09/09/27 16:30:04 WARN zookeeper.ClientCnxn: Exception closing session
0x23fa7262cf0001 to sun.nio.ch.SelectionKeyImpl@5557c2bd
java.io.IOException: TIMED OUT
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
09/09/27 16:30:05 INFO zookeeper.ClientCnxn: Attempting connection to server
ubuntu12/192.168.33.12:2222
09/09/27 16:30:18 WARN zookeeper.ClientCnxn: Exception closing session
0x23fa7262cf0001 to sun.nio.ch.SelectionKeyImpl@3b25b27c
java.io.IOException: TIMED OUT
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
09/09/27 16:30:18 WARN zookeeper.ClientCnxn: Ignoring exception during
shutdown output
java.net.SocketException: Transport endpoint is not connected
at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
09/09/27 16:30:19 INFO zookeeper.ClientCnxn: Attempting connection to server
ubuntu10/192.168.33.10:2222
09/09/27 16:30:23 INFO zookeeper.ClientCnxn: Priming connection to
java.nio.channels.SocketChannel[connected
local=/192.168.33.7:36969remote=ubuntu10/
192.168.33.10:2222]
09/09/27 16:30:23 INFO zookeeper.ClientCnxn: Server connection successful
insertting 15000 webpages need 680627 ms.
insertting 16000 webpages need 783023 ms.
insertting 17000 webpages need 797371 ms.
insertting 18000 webpages need 811977 ms.
insertting 19000 webpages need 825966 ms.
insertting 20000 webpages need 972361 ms.
insertting 21000 webpages need 988808 ms.
insertting 22000 webpages need 1054817 ms.
insertting 23000 webpages need 1084032 ms.
insertting 24000 webpages need 1099683 ms.
insertting 25000 webpages need 1114645 ms.
09/09/27 16:38:45 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 0 time(s).
09/09/27 16:38:46 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 1 time(s).
09/09/27 16:38:47 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 2 time(s).
09/09/27 16:38:48 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 3 time(s).
09/09/27 16:38:49 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 4 time(s).
09/09/27 16:38:50 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 5 time(s).
09/09/27 16:38:51 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 6 time(s).
09/09/27 16:38:52 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 7 time(s).
09/09/27 16:38:53 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 8 time(s).
09/09/27 16:38:54 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 9 time(s).
09/09/27 16:38:57 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 0 time(s).
09/09/27 16:38:58 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 1 time(s).
09/09/27 16:38:59 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 2 time(s).
09/09/27 16:39:00 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 3 time(s).
09/09/27 16:39:01 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 4 time(s).
09/09/27 16:39:02 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 5 time(s).
09/09/27 16:39:03 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 6 time(s).
09/09/27 16:39:04 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 7 time(s).
09/09/27 16:39:05 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 8 time(s).
09/09/27 16:39:06 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 9 time(s).
09/09/27 16:39:07 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 0 time(s).
09/09/27 16:39:28 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 0 time(s).
09/09/27 16:39:49 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 1 time(s).
09/09/27 16:39:53 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 1 time(s).
09/09/27 16:39:57 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 2 time(s).
09/09/27 16:40:01 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 3 time(s).
09/09/27 16:40:05 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 4 time(s).
09/09/27 16:40:09 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 5 time(s).
09/09/27 16:40:13 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 6 time(s).
09/09/27 16:40:17 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 7 time(s).
09/09/27 16:40:38 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 2 time(s).
09/09/27 16:40:42 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 8 time(s).
09/09/27 16:40:46 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 9 time(s).
09/09/27 16:41:12 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 0 time(s).
09/09/27 16:41:22 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 0 time(s).
09/09/27 16:41:23 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 1 time(s).
09/09/27 16:41:24 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 2 time(s).
09/09/27 16:41:25 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 3 time(s).
09/09/27 16:41:26 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 4 time(s).
09/09/27 16:41:27 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 5 time(s).
09/09/27 16:41:28 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 6 time(s).
09/09/27 16:41:29 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 7 time(s).
09/09/27 16:41:30 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 8 time(s).
09/09/27 16:41:31 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 9 time(s).
09/09/27 16:41:32 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 0 time(s).
09/09/27 16:41:33 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 1 time(s).
09/09/27 16:41:34 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 2 time(s).
09/09/27 16:41:35 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 3 time(s).
09/09/27 16:41:36 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 4 time(s).
09/09/27 16:41:37 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 5 time(s).
09/09/27 16:41:38 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 6 time(s).
09/09/27 16:41:39 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 7 time(s).
09/09/27 16:41:40 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 8 time(s).
09/09/27 16:41:41 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 9 time(s).
09/09/27 16:41:46 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 0 time(s).
09/09/27 16:41:47 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 1 time(s).
09/09/27 16:41:48 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 2 time(s).
09/09/27 16:41:49 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 3 time(s).
09/09/27 16:41:50 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 4 time(s).
09/09/27 16:41:51 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 5 time(s).
09/09/27 16:41:52 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 6 time(s).
09/09/27 16:41:53 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 7 time(s).
09/09/27 16:41:54 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 8 time(s).
09/09/27 16:41:55 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 9 time(s).
09/09/27 16:41:56 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 0 time(s).
09/09/27 16:41:57 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 1 time(s).
09/09/27 16:41:58 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 2 time(s).
09/09/27 16:41:59 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 3 time(s).
09/09/27 16:42:00 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 4 time(s).
09/09/27 16:42:01 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 5 time(s).
09/09/27 16:42:02 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 6 time(s).
09/09/27 16:42:03 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 7 time(s).
09/09/27 16:42:04 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 8 time(s).
09/09/27 16:42:05 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 9 time(s).
09/09/27 16:42:10 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 0 time(s).
09/09/27 16:42:11 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 1 time(s).
09/09/27 16:42:12 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 2 time(s).
09/09/27 16:42:13 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 3 time(s).
09/09/27 16:42:14 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 4 time(s).
09/09/27 16:42:15 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 5 time(s).
09/09/27 16:42:16 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 6 time(s).
09/09/27 16:42:17 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 7 time(s).
09/09/27 16:42:18 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 8 time(s).
09/09/27 16:42:19 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 9 time(s).
09/09/27 16:42:20 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 0 time(s).
09/09/27 16:42:21 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 1 time(s).
09/09/27 16:42:22 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 2 time(s).
09/09/27 16:42:23 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 3 time(s).
09/09/27 16:42:24 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 4 time(s).
09/09/27 16:42:25 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 5 time(s).
09/09/27 16:42:26 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 6 time(s).
09/09/27 16:42:27 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 7 time(s).
09/09/27 16:42:28 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 8 time(s).
09/09/27 16:42:29 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 9 time(s).
09/09/27 16:42:38 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 0 time(s).
09/09/27 16:42:39 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 1 time(s).
09/09/27 16:42:40 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 2 time(s).
09/09/27 16:42:41 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 3 time(s).
09/09/27 16:42:42 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 4 time(s).
09/09/27 16:42:43 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 5 time(s).
09/09/27 16:42:44 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 6 time(s).
09/09/27 16:42:45 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 7 time(s).
09/09/27 16:42:46 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 8 time(s).
09/09/27 16:42:47 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 9 time(s).
09/09/27 16:42:48 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 0 time(s).
09/09/27 16:42:49 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 1 time(s).
09/09/27 16:42:50 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 2 time(s).
09/09/27 16:42:51 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 3 time(s).
09/09/27 16:42:52 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 4 time(s).
09/09/27 16:42:53 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 5 time(s).
09/09/27 16:42:54 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 6 time(s).
09/09/27 16:42:55 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 7 time(s).
09/09/27 16:42:56 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 8 time(s).
09/09/27 16:42:57 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 9 time(s).
09/09/27 16:43:06 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 0 time(s).
09/09/27 16:43:07 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 1 time(s).
09/09/27 16:43:08 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 2 time(s).
09/09/27 16:43:09 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 3 time(s).
09/09/27 16:43:10 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 4 time(s).
09/09/27 16:43:11 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 5 time(s).
09/09/27 16:43:12 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 6 time(s).
09/09/27 16:43:13 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 7 time(s).
09/09/27 16:43:14 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 8 time(s).
09/09/27 16:43:15 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 9 time(s).
09/09/27 16:43:16 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 0 time(s).
09/09/27 16:43:17 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 1 time(s).
09/09/27 16:43:18 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 2 time(s).
09/09/27 16:43:19 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 3 time(s).
09/09/27 16:43:20 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 4 time(s).
09/09/27 16:43:21 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 5 time(s).
09/09/27 16:43:22 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 6 time(s).
09/09/27 16:43:23 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 7 time(s).
09/09/27 16:43:24 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 8 time(s).
09/09/27 16:43:25 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 9 time(s).
09/09/27 16:43:42 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 0 time(s).
09/09/27 16:43:43 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 1 time(s).
09/09/27 16:43:44 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 2 time(s).
09/09/27 16:43:45 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 3 time(s).
09/09/27 16:43:46 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 4 time(s).
09/09/27 16:43:47 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 5 time(s).
09/09/27 16:43:48 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 6 time(s).
09/09/27 16:43:49 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 7 time(s).
09/09/27 16:43:50 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 8 time(s).
09/09/27 16:43:51 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 9 time(s).
09/09/27 16:43:52 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 0 time(s).
09/09/27 16:43:53 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 1 time(s).
09/09/27 16:43:54 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 2 time(s).
09/09/27 16:43:55 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 3 time(s).
09/09/27 16:43:56 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 4 time(s).
09/09/27 16:43:57 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 5 time(s).
09/09/27 16:43:58 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 6 time(s).
09/09/27 16:43:59 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 7 time(s).
09/09/27 16:44:00 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 8 time(s).
09/09/27 16:44:01 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 9 time(s).
09/09/27 16:44:34 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 0 time(s).
09/09/27 16:44:35 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 1 time(s).
09/09/27 16:44:36 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 2 time(s).
09/09/27 16:44:37 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 3 time(s).
09/09/27 16:44:38 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 4 time(s).
09/09/27 16:44:39 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 5 time(s).
09/09/27 16:44:40 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 6 time(s).
09/09/27 16:44:41 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 7 time(s).
09/09/27 16:44:42 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 8 time(s).
09/09/27 16:44:43 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 9 time(s).
09/09/27 16:44:44 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 0 time(s).
09/09/27 16:44:45 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 1 time(s).
09/09/27 16:44:46 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 2 time(s).
09/09/27 16:44:47 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 3 time(s).
09/09/27 16:44:48 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 4 time(s).
09/09/27 16:44:49 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 5 time(s).
09/09/27 16:44:50 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 6 time(s).
09/09/27 16:44:51 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 7 time(s).
09/09/27 16:44:52 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 8 time(s).
09/09/27 16:44:53 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 9 time(s).
09/09/27 16:45:58 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 0 time(s).
09/09/27 16:45:59 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 1 time(s).
09/09/27 16:46:00 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 2 time(s).
09/09/27 16:46:01 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 3 time(s).
09/09/27 16:46:02 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 4 time(s).
09/09/27 16:46:03 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 5 time(s).
09/09/27 16:46:04 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 6 time(s).
09/09/27 16:46:05 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 7 time(s).
09/09/27 16:46:06 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 8 time(s).
09/09/27 16:46:07 INFO ipc.HBaseClient: Retrying connect to server: /
192.168.33.5:60020. Already tried 9 time(s).
Killed
java -classpath
/home/cyd/conf/hbase/:./hbasetest.jar:./commons-logging-1.0.4.jar:./zookeeper-r785019-hbase-1329.jar:./json.jar:./commons-cli-2.0-SNAPSHOT.jar:./servlet-api-2.5-6.1.14.jar:./jruby-complete-1.2.0.jar:./commons-math-1.1.jar:./AgileJSON-2009-03-30.jar:./lucene-core-2.2.0.jar:./commons-logging-api-1.0.4.jar:./jasper-runtime-5.5.12.jar:./jetty-util-6.1.14.jar:./hbase-0.20.0.jar:./commons-httpclient-3.0.1.jar:./hadoop-0.20.0-plus4681-core.jar:./libthrift-r771587.jar:./hadoop-0.20.0-test.jar:./jasper-compiler-5.5.12.jar:./junit-3.8.1.jar:./xmlenc-0.52.jar:./log4j-1.2.15.jar:./commons-el-from-jetty-5.1.4.jar:./jetty-6.1.14.jar
hbasetest.HbaseTest http://news.163.com/09/0803/01/5FOO155J0001124J.html 1
10
Re: is there any problem with our environment?
Posted by Zheng Lv <lv...@gmail.com>.
following is the logs in another one of shutdown regionservers:
2009年 09月 27日 星期日 15:40:20 CST Starting regionserver on ubuntu7
ulimit -n 1024
2009-09-27 15:40:20,951 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer:
vmInputArguments=[-Xmx3000m, -Djava.net.preferIPv4Stack=true,
-XX:+HeapDumpOnOutOfMemoryError, -XX:+UseConcMarkSweepGC,
-XX:+CMSIncrementalMode, -Dhbase.log.dir=/opt/hadoop/hbase/bin/../logs,
-Dhbase.log.file=hbase-cyd-regionserver-ubuntu7.log,
-Dhbase.home.dir=/opt/hadoop/hbase/bin/.., -Dhbase.id.str=cyd,
-Dhbase.root.logger=INFO,DRFA,
-Djava.library.path=/opt/hadoop/hbase/bin/../lib/native/Linux-amd64-64]
2009-09-27 15:40:20,961 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: My address is
ubuntu7:60020
2009-09-27 15:40:21,003 INFO org.apache.hadoop.hbase.ipc.HBaseRpcMetrics:
Initializing RPC Metrics with hostName=HRegionServer, port=60020
2009-09-27 15:40:21,011 INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher:
globalMemStoreLimit=1.2g, globalMemStoreLimitLowMark=748.5m, maxHeap=2.9g
2009-09-27 15:40:21,013 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Runs every 10000000ms
2009-09-27 15:40:21,027 INFO org.apache.zookeeper.ZooKeeper: Client
environment:zookeeper.version=3.2.0--1, built on 05/15/2009 06:05 GMT
2009-09-27 15:40:21,027 INFO org.apache.zookeeper.ZooKeeper: Client
environment:host.name=ubuntu7
2009-09-27 15:40:21,027 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.version=1.6.0_14
2009-09-27 15:40:21,027 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.vendor=Sun Microsystems Inc.
2009-09-27 15:40:21,027 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.home=/usr/lib/jvm/java-6-sun-1.6.0.14/jre
2009-09-27 15:40:21,027 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.class.path=/home/cyd/conf/hbase:/usr/lib/jvm/java-6-sun/lib/tools.jar:/opt/hadoop/hbase/bin/..:/opt/hadoop/hbase/bin/../hbase-0.20.0.jar:/opt/hadoop/hbase/bin/../lib/AgileJSON-2009-03-30.jar:/opt/hadoop/hbase/bin/../lib/commons-cli-2.0-SNAPSHOT.jar:/opt/hadoop/hbase/bin/../lib/commons-el-from-jetty-5.1.4.jar:/opt/hadoop/hbase/bin/../lib/commons-httpclient-3.0.1.jar:/opt/hadoop/hbase/bin/../lib/commons-logging-1.0.4.jar:/opt/hadoop/hbase/bin/../lib/commons-logging-api-1.0.4.jar:/opt/hadoop/hbase/bin/../lib/commons-math-1.1.jar:/opt/hadoop/hbase/bin/../lib/hadoop-0.20.0-plus4681-core.jar:/opt/hadoop/hbase/bin/../lib/hadoop-0.20.0-test.jar:/opt/hadoop/hbase/bin/../lib/jasper-compiler-5.5.12.jar:/opt/hadoop/hbase/bin/../lib/jasper-runtime-5.5.12.jar:/opt/hadoop/hbase/bin/../lib/jetty-6.1.14.jar:/opt/hadoop/hbase/bin/../lib/jetty-util-6.1.14.jar:/opt/hadoop/hbase/bin/../lib/jruby-complete-1.2.0.jar:/opt/hadoop/hbase/bin/../lib/json.jar:/opt/hadoop/hbase/bin/../lib/junit-3.8.1.jar:/opt/hadoop/hbase/bin/../lib/libthrift-r771587.jar:/opt/hadoop/hbase/bin/../lib/log4j-1.2.15.jar:/opt/hadoop/hbase/bin/../lib/lucene-core-2.2.0.jar:/opt/hadoop/hbase/bin/../lib/servlet-api-2.5-6.1.14.jar:/opt/hadoop/hbase/bin/../lib/xmlenc-0.52.jar:/opt/hadoop/hbase/bin/../lib/zookeeper-r785019-hbase-1329.jar:/opt/hadoop/hbase/bin/../lib/jsp-2.1/jsp-2.1.jar:/opt/hadoop/hbase/bin/../lib/jsp-2.1/jsp-api-2.1.jar
2009-09-27 15:40:21,027 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.library.path=/opt/hadoop/hbase/bin/../lib/native/Linux-amd64-64
2009-09-27 15:40:21,027 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=/tmp
2009-09-27 15:40:21,028 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.compiler=<NA>
2009-09-27 15:40:21,028 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.name=Linux
2009-09-27 15:40:21,028 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.arch=amd64
2009-09-27 15:40:21,028 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.version=2.6.28-11-server
2009-09-27 15:40:21,028 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.name=cyd
2009-09-27 15:40:21,028 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.home=/home/cyd
2009-09-27 15:40:21,028 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.dir=/opt/hadoop/hbase-0.20.0
2009-09-27 15:40:21,029 INFO org.apache.zookeeper.ZooKeeper: Initiating
client connection, host=ubuntu12:2222,ubuntu10:2222,ubuntu2:2222
sessionTimeout=240000
watcher=org.apache.hadoop.hbase.regionserver.HRegionServer@6719dc16
2009-09-27 15:40:21,030 INFO org.apache.zookeeper.ClientCnxn:
zookeeper.disableAutoWatchReset is false
2009-09-27 15:40:21,042 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu10/192.168.33.10:2222
2009-09-27 15:40:21,048 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.33.6:36316 remote=ubuntu10/192.168.33.10:2222]
2009-09-27 15:40:21,060 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-09-27 15:40:21,138 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: SyncConnected, type: None, path: null
2009-09-27 15:40:21,362 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to read master
address from ZooKeeper. Retrying. Error was:
java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for /hbase/master
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:331)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:240)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1316)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1348)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:428)
at java.lang.Thread.run(Thread.java:619)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for /hbase/master
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:750)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:327)
... 5 more
2009-09-27 15:40:21,455 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: SyncConnected, type: NodeCreated, path: /hbase/master
2009-09-27 15:40:21,469 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Telling master at
192.168.33.7:60000 that we are up
2009-09-27 15:40:24,368 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Telling master at
192.168.33.7:60000 that we are up
2009-09-27 15:41:03,732 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Master passed us address
to use. Was=192.168.33.6:60020, Now=192.168.33.6
2009-09-27 15:41:04,267 INFO org.apache.hadoop.hbase.regionserver.HLog: HLog
configuration: blocksize=67108864, rollsize=63753420, enabled=true,
flushlogentries=100, optionallogflushinternal=10000ms
2009-09-27 15:41:04,468 INFO org.apache.hadoop.hbase.regionserver.HLog: New
hlog /hbase/.logs/ubuntu7,60020,1254037263642/hlog.dat.1254037264267
2009-09-27 15:41:04,477 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=RegionServer,
sessionId=regionserver/192.168.33.6:60020
2009-09-27 15:41:04,478 INFO
org.apache.hadoop.hbase.regionserver.metrics.RegionServerMetrics:
Initialized
2009-09-27 15:41:04,818 INFO org.apache.hadoop.http.HttpServer: Jetty bound
to port 60030
2009-09-27 15:41:05,167 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
Responder: starting
2009-09-27 15:41:05,167 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
listener on 60020: starting
2009-09-27 15:41:05,168 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 0 on 60020: starting
2009-09-27 15:41:05,168 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 1 on 60020: starting
2009-09-27 15:41:05,168 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 2 on 60020: starting
2009-09-27 15:41:05,168 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 3 on 60020: starting
2009-09-27 15:41:05,169 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 4 on 60020: starting
2009-09-27 15:41:05,169 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 5 on 60020: starting
2009-09-27 15:41:05,169 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 6 on 60020: starting
2009-09-27 15:41:05,169 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 7 on 60020: starting
2009-09-27 15:41:05,169 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 8 on 60020: starting
2009-09-27 15:41:05,169 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 9 on 60020: starting
2009-09-27 15:41:05,169 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: HRegionServer started
at: 192.168.33.6:60020
2009-09-27 15:41:05,198 INFO org.apache.hadoop.hbase.regionserver.StoreFile:
Allocating LruBlockCache with maximum size 598.8m
2009-09-27 15:41:08,229 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
.META.,,1
2009-09-27 15:41:08,237 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN:
.META.,,1
2009-09-27 15:41:08,333 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region .META.,,1/1028785192 available; sequence id is 0
2009-09-27 15:50:48,238 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: compactions no longer
limited
2009-09-27 16:19:49,375 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x123fa71ce2e0000 to sun.nio.ch.SelectionKeyImpl@7745c859
java.io.IOException: TIMED OUT
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-09-27 16:19:49,476 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: Disconnected, type: None, path: null
2009-09-27 16:19:50,182 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu12/192.168.33.12:2222
2009-09-27 16:19:53,188 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x123fa71ce2e0000 to sun.nio.ch.SelectionKeyImpl@16e334a
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
2009-09-27 16:19:53,189 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown input
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-09-27 16:19:53,189 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-09-27 16:19:53,919 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu2/192.168.33.9:2222
2009-09-27 16:19:56,018 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: ubuntu6/192.168.33.7:9000. Already tried 0 time(s).
2009-09-27 16:19:56,928 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x123fa71ce2e0000 to sun.nio.ch.SelectionKeyImpl@506937f9
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
2009-09-27 16:19:56,928 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown input
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-09-27 16:19:56,928 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-09-27 16:19:58,605 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu10/192.168.33.10:2222
2009-09-27 16:20:01,617 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x123fa71ce2e0000 to sun.nio.ch.SelectionKeyImpl@6e75d758
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
2009-09-27 16:20:01,618 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown input
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-09-27 16:20:01,618 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-09-27 16:20:02,323 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu12/192.168.33.12:2222
2009-09-27 16:20:05,328 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x123fa71ce2e0000 to sun.nio.ch.SelectionKeyImpl@7a8d0b0f
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
2009-09-27 16:20:05,328 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown input
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-09-27 16:20:05,329 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-09-27 16:20:05,879 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu2/192.168.33.9:2222
2009-09-27 16:20:08,887 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x123fa71ce2e0000 to sun.nio.ch.SelectionKeyImpl@41d144a4
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
2009-09-27 16:20:08,888 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown input
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-09-27 16:20:08,889 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-09-27 16:20:10,535 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu10/192.168.33.10:2222
2009-09-27 16:20:13,527 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.33.6:60330 remote=ubuntu10/192.168.33.10:2222]
2009-09-27 16:20:13,528 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-09-27 16:20:13,532 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: SyncConnected, type: None, path: null
2009-09-27 16:21:33,555 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x123fa71ce2e0000 to sun.nio.ch.SelectionKeyImpl@11f18cbe
java.io.IOException: TIMED OUT
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-09-27 16:21:33,656 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: Disconnected, type: None, path: null
2009-09-27 16:21:33,713 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu12/192.168.33.12:2222
2009-09-27 16:21:47,047 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x123fa71ce2e0000 to sun.nio.ch.SelectionKeyImpl@66e8c7db
java.io.IOException: TIMED OUT
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-09-27 16:21:47,048 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.net.SocketException: Transport endpoint is not connected
at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-09-27 16:21:47,227 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu2/192.168.33.9:2222
2009-09-27 16:22:00,566 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x123fa71ce2e0000 to sun.nio.ch.SelectionKeyImpl@6d0718b7
java.io.IOException: TIMED OUT
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-09-27 16:22:00,567 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.net.SocketException: Transport endpoint is not connected
at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-09-27 16:22:02,264 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
(Retry: 0)
org.apache.hadoop.hbase.Leases$LeaseStillHeldException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
at java.lang.Thread.run(Thread.java:619)
2009-09-27 16:22:02,271 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
(Retry: 1)
org.apache.hadoop.hbase.Leases$LeaseStillHeldException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
at java.lang.Thread.run(Thread.java:619)
2009-09-27 16:22:02,276 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
(Retry: 2)
org.apache.hadoop.hbase.Leases$LeaseStillHeldException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
at java.lang.Thread.run(Thread.java:619)
2009-09-27 16:22:02,281 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
(Retry: 3)
org.apache.hadoop.hbase.Leases$LeaseStillHeldException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
at java.lang.Thread.run(Thread.java:619)
2009-09-27 16:22:02,286 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
(Retry: 4)
org.apache.hadoop.hbase.Leases$LeaseStillHeldException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
at java.lang.Thread.run(Thread.java:619)
2009-09-27 16:22:02,290 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
(Retry: 5)
org.apache.hadoop.hbase.Leases$LeaseStillHeldException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
at java.lang.Thread.run(Thread.java:619)
2009-09-27 16:22:02,294 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu10/192.168.33.10:2222
2009-09-27 16:22:02,295 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.33.6:60343 remote=ubuntu10/192.168.33.10:2222]
2009-09-27 16:22:02,295 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-09-27 16:22:02,296 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
(Retry: 6)
org.apache.hadoop.hbase.Leases$LeaseStillHeldException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
at java.lang.Thread.run(Thread.java:619)
2009-09-27 16:22:02,298 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x123fa71ce2e0000 to sun.nio.ch.SelectionKeyImpl@23bf8cd5
java.io.IOException: Session Expired
at
org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:548)
at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:661)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
2009-09-27 16:22:02,298 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: Expired, type: None, path: null
2009-09-27 16:22:02,299 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session
expired
2009-09-27 16:22:02,301 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
(Retry: 7)
org.apache.hadoop.hbase.Leases$LeaseStillHeldException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
at java.lang.Thread.run(Thread.java:619)
2009-09-27 16:22:02,303 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
request=0.0, regions=1, stores=2, storefiles=0, storefileIndexSize=0,
memstoreSize=0, usedHeap=29, maxHeap=2993, blockCacheSize=5151568,
blockCacheFree=622696432, blockCacheCount=0, blockCacheHitRatio=0
2009-09-27 16:22:02,307 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
(Retry: 8)
org.apache.hadoop.hbase.Leases$LeaseStillHeldException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
at java.lang.Thread.run(Thread.java:619)
2009-09-27 16:22:02,307 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Stop was requested,
clearing the toDo despite of the exception
2009-09-27 16:22:02,308 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
server on 60020
2009-09-27 16:22:02,308 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 0 on 60020: exiting
2009-09-27 16:22:02,309 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 1 on 60020: exiting
2009-09-27 16:22:02,309 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 3 on 60020: exiting
2009-09-27 16:22:02,310 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC
Server listener on 60020
2009-09-27 16:22:02,310 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC
Server Responder
2009-09-27 16:22:02,310 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
2009-09-27 16:22:02,325 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 4 on 60020: exiting
2009-09-27 16:22:02,325 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 5 on 60020: exiting
2009-09-27 16:22:02,328 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 6 on 60020: exiting
2009-09-27 16:22:02,329 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 7 on 60020: exiting
2009-09-27 16:22:02,329 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 8 on 60020: exiting
2009-09-27 16:22:02,330 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 2 on 60020: exiting
2009-09-27 16:22:02,330 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 9 on 60020: exiting
2009-09-27 16:22:02,465 INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher:
regionserver/192.168.33.6:60020.cacheFlusher exiting
2009-09-27 16:22:02,465 INFO
org.apache.hadoop.hbase.regionserver.LogFlusher:
regionserver/192.168.33.6:60020.logFlusher exiting
2009-09-27 16:22:02,465 INFO org.apache.hadoop.hbase.regionserver.LogRoller:
LogRoller exiting.
2009-09-27 16:22:02,466 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread:
regionserver/192.168.33.6:60020.compactor exiting
2009-09-27 16:22:02,466 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer$MajorCompactionChecker:
regionserver/192.168.33.6:60020.majorCompactionChecker exiting
2009-09-27 16:22:02,478 INFO org.apache.hadoop.hdfs.DFSClient:
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
/hbase/.logs/ubuntu7,60020,1254037263642/hlog.dat.1254037264267 File does
not exist. Holder DFSClient_679922832 does not have any open files.
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1317)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1308)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1236)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:739)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy1.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy1.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2875)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2757)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2048)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2234)
2009-09-27 16:22:02,478 WARN org.apache.hadoop.hdfs.DFSClient:
NotReplicatedYetException sleeping
/hbase/.logs/ubuntu7,60020,1254037263642/hlog.dat.1254037264267 retries left
4
2009-09-27 16:22:02,883 INFO org.apache.hadoop.hdfs.DFSClient:
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
/hbase/.logs/ubuntu7,60020,1254037263642/hlog.dat.1254037264267 File does
not exist. Holder DFSClient_679922832 does not have any open files.
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1317)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1308)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1236)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:739)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy1.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy1.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2875)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2757)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2048)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2234)
2009-09-27 16:22:02,883 WARN org.apache.hadoop.hdfs.DFSClient:
NotReplicatedYetException sleeping
/hbase/.logs/ubuntu7,60020,1254037263642/hlog.dat.1254037264267 retries left
3
2009-09-27 16:22:03,687 INFO org.apache.hadoop.hdfs.DFSClient:
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
/hbase/.logs/ubuntu7,60020,1254037263642/hlog.dat.1254037264267 File does
not exist. Holder DFSClient_679922832 does not have any open files.
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1317)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1308)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1236)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:739)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy1.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy1.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2875)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2757)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2048)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2234)
2009-09-27 16:22:03,687 WARN org.apache.hadoop.hdfs.DFSClient:
NotReplicatedYetException sleeping
/hbase/.logs/ubuntu7,60020,1254037263642/hlog.dat.1254037264267 retries left
2
2009-09-27 16:22:04,520 INFO org.apache.hadoop.hbase.Leases:
regionserver/192.168.33.6:60020.leaseChecker closing leases
2009-09-27 16:22:04,521 INFO org.apache.hadoop.hbase.Leases:
regionserver/192.168.33.6:60020.leaseChecker closed leases
2009-09-27 16:22:05,292 INFO org.apache.hadoop.hdfs.DFSClient:
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
/hbase/.logs/ubuntu7,60020,1254037263642/hlog.dat.1254037264267 File does
not exist. Holder DFSClient_679922832 does not have any open files.
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1317)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1308)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1236)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:739)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy1.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy1.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2875)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2757)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2048)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2234)
2009-09-27 16:22:05,292 WARN org.apache.hadoop.hdfs.DFSClient:
NotReplicatedYetException sleeping
/hbase/.logs/ubuntu7,60020,1254037263642/hlog.dat.1254037264267 retries left
1
2009-09-27 16:22:08,370 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: worker thread exiting
2009-09-27 16:22:08,497 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer
Exception: org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
/hbase/.logs/ubuntu7,60020,1254037263642/hlog.dat.1254037264267 File does
not exist. Holder DFSClient_679922832 does not have any open files.
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1317)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1308)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1236)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:739)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy1.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy1.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2875)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2757)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2048)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2234)
2009-09-27 16:22:08,498 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block null bad datanode[0] nodes == null
2009-09-27 16:22:08,498 WARN org.apache.hadoop.hdfs.DFSClient: Could not get
block locations. Source file
"/hbase/.logs/ubuntu7,60020,1254037263642/hlog.dat.1254037264267" -
Aborting...
2009-09-27 16:22:08,499 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to close log in
abort
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
/hbase/.logs/ubuntu7,60020,1254037263642/hlog.dat.1254037264267 File does
not exist. Holder DFSClient_679922832 does not have any open files.
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1317)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1308)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1236)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:637)
at java.lang.Thread.run(Thread.java:619)
2009-09-27 16:22:08,501 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed .META.,,1
2009-09-27 16:22:08,502 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: aborting server at:
192.168.33.6:60020
2009-09-27 16:22:08,502 INFO org.apache.zookeeper.ZooKeeper: Closing
session: 0x123fa71ce2e0000
2009-09-27 16:22:08,502 INFO org.apache.zookeeper.ClientCnxn: Closing
ClientCnxn for session: 0x123fa71ce2e0000
2009-09-27 16:22:08,502 INFO org.apache.zookeeper.ClientCnxn: Disconnecting
ClientCnxn for session: 0x123fa71ce2e0000
2009-09-27 16:22:08,502 INFO org.apache.zookeeper.ZooKeeper: Session:
0x123fa71ce2e0000 closed
2009-09-27 16:22:08,503 INFO org.apache.zookeeper.ClientCnxn: EventThread
shut down
2009-09-27 16:22:08,505 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver/
192.168.33.6:60020 exiting
2009-09-27 16:22:08,507 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Starting shutdown
thread.
2009-09-27 16:22:08,507 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Shutdown thread complete
Re: is there any problem with our environment?
Posted by Zheng Lv <lv...@gmail.com>.
following is the logs in one of shut down regionservers:
Sun Sep 27 15:40:35 CST 2009 Starting regionserver on ubuntu9
ulimit -n 1024
2009-09-27 15:40:35,998 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer:
vmInputArguments=[-Xmx3000m, -Djava.net.preferIPv4Stack=true,
-XX:+HeapDumpOnOutOfMemoryError, -XX:+UseConcMarkSweepGC,
-XX:+CMSIncrementalMode, -Dhbase.log.dir=/opt/hadoop/hbase/bin/../logs,
-Dhbase.log.file=hbase-cyd-regionserver-ubuntu9.log,
-Dhbase.home.dir=/opt/hadoop/hbase/bin/.., -Dhbase.id.str=cyd,
-Dhbase.root.logger=INFO,DRFA,
-Djava.library.path=/opt/hadoop/hbase/bin/../lib/native/Linux-amd64-64]
2009-09-27 15:40:36,009 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: My address is
ubuntu9:60020
2009-09-27 15:40:36,059 INFO org.apache.hadoop.hbase.ipc.HBaseRpcMetrics:
Initializing RPC Metrics with hostName=HRegionServer, port=60020
2009-09-27 15:40:36,067 INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher:
globalMemStoreLimit=1.2g, globalMemStoreLimitLowMark=748.5m, maxHeap=2.9g
2009-09-27 15:40:36,069 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Runs every 10000000ms
2009-09-27 15:40:36,087 INFO org.apache.zookeeper.ZooKeeper: Client
environment:zookeeper.version=3.2.0--1, built on 05/15/2009 06:05 GMT
2009-09-27 15:40:36,087 INFO org.apache.zookeeper.ZooKeeper: Client
environment:host.name=ubuntu9
2009-09-27 15:40:36,087 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.version=1.6.0_14
2009-09-27 15:40:36,087 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.vendor=Sun Microsystems Inc.
2009-09-27 15:40:36,087 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.home=/usr/lib/jvm/java-6-sun-1.6.0.14/jre
2009-09-27 15:40:36,087 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.class.path=/home/cyd/conf/hbase:/usr/lib/jvm/java-6-sun/lib/tools.jar:/opt/hadoop/hbase/bin/..:/opt/hadoop/hbase/bin/../hbase-0.20.0.jar:/opt/hadoop/hbase/bin/../lib/AgileJSON-2009-03-30.jar:/opt/hadoop/hbase/bin/../lib/commons-cli-2.0-SNAPSHOT.jar:/opt/hadoop/hbase/bin/../lib/commons-el-from-jetty-5.1.4.jar:/opt/hadoop/hbase/bin/../lib/commons-httpclient-3.0.1.jar:/opt/hadoop/hbase/bin/../lib/commons-logging-1.0.4.jar:/opt/hadoop/hbase/bin/../lib/commons-logging-api-1.0.4.jar:/opt/hadoop/hbase/bin/../lib/commons-math-1.1.jar:/opt/hadoop/hbase/bin/../lib/hadoop-0.20.0-plus4681-core.jar:/opt/hadoop/hbase/bin/../lib/hadoop-0.20.0-test.jar:/opt/hadoop/hbase/bin/../lib/jasper-compiler-5.5.12.jar:/opt/hadoop/hbase/bin/../lib/jasper-runtime-5.5.12.jar:/opt/hadoop/hbase/bin/../lib/jetty-6.1.14.jar:/opt/hadoop/hbase/bin/../lib/jetty-util-6.1.14.jar:/opt/hadoop/hbase/bin/../lib/jruby-complete-1.2.0.jar:/opt/hadoop/hbase/bin/../lib/json.jar:/opt/hadoop/hbase/bin/../lib/junit-3.8.1.jar:/opt/hadoop/hbase/bin/../lib/libthrift-r771587.jar:/opt/hadoop/hbase/bin/../lib/log4j-1.2.15.jar:/opt/hadoop/hbase/bin/../lib/lucene-core-2.2.0.jar:/opt/hadoop/hbase/bin/../lib/servlet-api-2.5-6.1.14.jar:/opt/hadoop/hbase/bin/../lib/xmlenc-0.52.jar:/opt/hadoop/hbase/bin/../lib/zookeeper-r785019-hbase-1329.jar:/opt/hadoop/hbase/bin/../lib/jsp-2.1/jsp-2.1.jar:/opt/hadoop/hbase/bin/../lib/jsp-2.1/jsp-api-2.1.jar
2009-09-27 15:40:36,087 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.library.path=/opt/hadoop/hbase/bin/../lib/native/Linux-amd64-64
2009-09-27 15:40:36,087 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=/tmp
2009-09-27 15:40:36,087 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.compiler=<NA>
2009-09-27 15:40:36,087 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.name=Linux
2009-09-27 15:40:36,087 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.arch=amd64
2009-09-27 15:40:36,087 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.version=2.6.28-11-server
2009-09-27 15:40:36,087 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.name=cyd
2009-09-27 15:40:36,087 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.home=/home/cyd
2009-09-27 15:40:36,087 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.dir=/opt/hadoop/hbase-0.20.0
2009-09-27 15:40:36,088 INFO org.apache.zookeeper.ZooKeeper: Initiating
client connection, host=ubuntu12:2222,ubuntu10:2222,ubuntu2:2222
sessionTimeout=240000
watcher=org.apache.hadoop.hbase.regionserver.HRegionServer@21a80a69
2009-09-27 15:40:36,090 INFO org.apache.zookeeper.ClientCnxn:
zookeeper.disableAutoWatchReset is false
2009-09-27 15:40:36,107 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu12/192.168.33.12:2222
2009-09-27 15:40:36,119 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.33.5:52277 remote=ubuntu12/192.168.33.12:2222]
2009-09-27 15:40:36,124 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-09-27 15:40:36,128 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x0 to sun.nio.ch.SelectionKeyImpl@214a55f2
java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104)
at sun.nio.ch.IOUtil.write(IOUtil.java:75)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:681)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
2009-09-27 15:40:36,132 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown input
java.net.SocketException: Transport endpoint is not connected
at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-09-27 15:40:36,133 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.net.SocketException: Transport endpoint is not connected
at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-09-27 15:40:36,248 WARN
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to set watcher on
ZNode /hbase/master
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/master
at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:649)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.watchMasterAddress(ZooKeeperWrapper.java:304)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.watchMasterAddress(HRegionServer.java:386)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.reinitializeZooKeeper(HRegionServer.java:316)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.reinitialize(HRegionServer.java:307)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:277)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.doMain(HRegionServer.java:2426)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2493)
2009-09-27 15:40:36,248 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to set watcher on
ZooKeeper master address. Retrying.
2009-09-27 15:40:36,807 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu10/192.168.33.10:2222
2009-09-27 15:40:36,807 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.33.5:47531 remote=ubuntu10/192.168.33.10:2222]
2009-09-27 15:40:36,808 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-09-27 15:40:36,819 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: SyncConnected, type: None, path: null
2009-09-27 15:40:39,452 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Telling master at
192.168.33.7:60000 that we are up
2009-09-27 15:41:18,922 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Master passed us address
to use. Was=192.168.33.5:60020, Now=192.168.33.5
2009-09-27 15:41:19,446 INFO org.apache.hadoop.hbase.regionserver.HLog: HLog
configuration: blocksize=67108864, rollsize=63753420, enabled=true,
flushlogentries=100, optionallogflushinternal=10000ms
2009-09-27 15:41:19,589 INFO org.apache.hadoop.hbase.regionserver.HLog: New
hlog /hbase/.logs/ubuntu9,60020,1254037278829/hlog.dat.1254037279447
2009-09-27 15:41:19,597 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=RegionServer,
sessionId=regionserver/192.168.33.5:60020
2009-09-27 15:41:19,597 INFO
org.apache.hadoop.hbase.regionserver.metrics.RegionServerMetrics:
Initialized
2009-09-27 15:41:19,803 INFO org.apache.hadoop.http.HttpServer: Jetty bound
to port 60030
2009-09-27 15:41:20,119 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
Responder: starting
2009-09-27 15:41:20,119 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
listener on 60020: starting
2009-09-27 15:41:20,120 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 0 on 60020: starting
2009-09-27 15:41:20,120 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 1 on 60020: starting
2009-09-27 15:41:20,121 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 2 on 60020: starting
2009-09-27 15:41:20,121 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 3 on 60020: starting
2009-09-27 15:41:20,121 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 4 on 60020: starting
2009-09-27 15:41:20,121 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 5 on 60020: starting
2009-09-27 15:41:20,121 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 6 on 60020: starting
2009-09-27 15:41:20,121 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 7 on 60020: starting
2009-09-27 15:41:20,122 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 8 on 60020: starting
2009-09-27 15:41:20,122 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 9 on 60020: starting
2009-09-27 15:41:20,139 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: HRegionServer started
at: 192.168.33.5:60020
2009-09-27 15:41:20,154 INFO org.apache.hadoop.hbase.regionserver.StoreFile:
Allocating LruBlockCache with maximum size 598.8m
2009-09-27 15:41:20,186 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
-ROOT-,,0
2009-09-27 15:41:20,191 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN:
-ROOT-,,0
2009-09-27 15:41:20,311 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region -ROOT-,,0/70236052 available; sequence id is 3
2009-09-27 16:24:10,813 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039598290_2497,1254040068962
2009-09-27 16:24:10,818 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039498012_112,1254040068962
2009-09-27 16:24:10,818 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039598290_2497,1254040068962
2009-09-27 16:24:10,845 INFO org.apache.zookeeper.ZooKeeper: Initiating
client connection, host=ubuntu12:2222,ubuntu10:2222,ubuntu2:2222
sessionTimeout=240000
watcher=org.apache.hadoop.hbase.client.HConnectionManager$ClientZKWatcher@5da631c2
2009-09-27 16:24:10,857 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu10/192.168.33.10:2222
2009-09-27 16:24:10,859 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.33.5:45230 remote=ubuntu10/192.168.33.10:2222]
2009-09-27 16:24:10,859 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-09-27 16:24:11,293 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039598290_2497,1254040068962/716276977
available; sequence id is 42721
2009-09-27 16:24:11,293 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039498012_112,1254040068962
2009-09-27 16:24:11,409 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039498012_112,1254040068962/537713758
available; sequence id is 42720
2009-09-27 16:24:19,615 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039598290_2497,1254040068962
2009-09-27 16:24:30,944 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039598290_2497,1254040068962
in 11sec
2009-09-27 16:24:36,842 INFO org.apache.hadoop.hbase.regionserver.HLog: Roll
/hbase/.logs/ubuntu9,60020,1254037278829/hlog.dat.1254037279447,
entries=6202, calcsize=63809254, filesize=63152219. New hlog
/hbase/.logs/ubuntu9,60020,1254037278829/hlog.dat.1254039876831
2009-09-27 16:24:50,945 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039498012_112,1254040068962
2009-09-27 16:24:52,656 INFO org.apache.hadoop.hbase.regionserver.HLog: Roll
/hbase/.logs/ubuntu9,60020,1254037278829/hlog.dat.1254039876831,
entries=6199, calcsize=63808410, filesize=63151916. New hlog
/hbase/.logs/ubuntu9,60020,1254037278829/hlog.dat.1254039892644
2009-09-27 16:24:59,247 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039498012_112,1254040068962
in 8sec
2009-09-27 16:24:59,247 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039598290_2497,1254040068962
2009-09-27 16:25:00,919 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC
Server handler 8 on 60020 took 1050ms appending an edit to hlog;
editcount=2575
2009-09-27 16:25:05,508 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039598290_2497,1254040068962
in 6sec
2009-09-27 16:25:05,508 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting split of region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039598290_2497,1254040068962
2009-09-27 16:25:05,526 WARN org.apache.hadoop.hbase.regionserver.HRegion:
Clock skew; parent regions id is 1254040068962 but current time here is
1254039905526
2009-09-27 16:25:06,517 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039598290_2497,1254040068962
2009-09-27 16:25:07,400 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039598290_2497,1254040068963/609760395
available; sequence id is 59591
2009-09-27 16:25:07,400 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039598290_2497,1254040068963
2009-09-27 16:25:07,669 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039664595_3865,1254040068963/1740143430
available; sequence id is 59592
2009-09-27 16:25:07,670 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039664595_3865,1254040068963
2009-09-27 16:25:07,708 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META
updated, and report to master all successful. Old region=REGION => {NAME =>
'webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039598290_2497,1254040068962',
STARTKEY => 'http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039598290_2497',
ENDKEY => '', ENCODED => 716276977, OFFLINE => true, SPLIT => true, TABLE =>
{{NAME => 'webpage', FAMILIES => [{NAME => 'CF_CONTENT', COMPRESSION =>
'NONE', VERSIONS => '2', TTL => '2147483647', BLOCKSIZE => '65536',
IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'CF_INFORMATION',
COMPRESSION => 'NONE', VERSIONS => '1', TTL => '2147483647', BLOCKSIZE =>
'65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, new regions:
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039598290_2497,1254040068963,
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039664595_3865,1254040068963.
Split took 2sec
2009-09-27 16:25:07,960 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039598290_2497,1254040068963
2009-09-27 16:25:07,960 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039664595_3865,1254040068963
2009-09-27 16:25:07,960 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039598290_2497,1254040068963
2009-09-27 16:25:08,105 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039598290_2497,1254040068963/609760395
available; sequence id is 59591
2009-09-27 16:25:08,106 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039664595_3865,1254040068963
2009-09-27 16:25:08,252 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039664595_3865,1254040068963/1740143430
available; sequence id is 59592
2009-09-27 16:25:16,942 INFO org.apache.hadoop.hbase.regionserver.HLog: Roll
/hbase/.logs/ubuntu9,60020,1254037278829/hlog.dat.1254039892644,
entries=6195, calcsize=63806619, filesize=63150833. New hlog
/hbase/.logs/ubuntu9,60020,1254037278829/hlog.dat.1254039916931
2009-09-27 16:25:27,708 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039598290_2497,1254040068963
2009-09-27 16:25:31,816 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC
Server handler 3 on 60020 took 1025ms appending an edit to hlog;
editcount=5693
2009-09-27 16:26:40,530 WARN org.apache.hadoop.hdfs.DFSClient:
DFSOutputStream ResponseProcessor exception for block
blk_1706358752536933321_1161java.net.SocketTimeoutException: 69000 millis
timeout while waiting for channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/192.168.33.5:54159 remote=/
192.168.33.5:50010]
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.DataInputStream.readFully(DataInputStream.java:178)
at java.io.DataInputStream.readLong(DataInputStream.java:399)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2369)
2009-09-27 16:26:44,579 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1706358752536933321_1161 bad datanode[0]
192.168.33.5:50010
2009-09-27 16:26:44,581 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_1706358752536933321_1161 in pipeline
192.168.33.5:50010, 192.168.33.6:50010, 192.168.33.8:50010: bad datanode
192.168.33.5:50010
2009-09-27 16:26:44,625 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC
Server handler 5 on 60020 took 72782ms appending an edit to hlog;
editcount=5705
2009-09-27 16:26:44,628 WARN org.apache.hadoop.hbase.regionserver.HLog:
regionserver/192.168.33.5:60020.logFlusher took 65000ms optional sync'ing
hlog; editcount=5711
2009-09-27 16:26:45,656 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039598290_2497,1254040068963
in 1mins, 17sec
2009-09-27 16:26:45,657 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039664595_3865,1254040068963
2009-09-27 16:26:46,035 INFO org.apache.hadoop.hbase.regionserver.HLog: Roll
/hbase/.logs/ubuntu9,60020,1254037278829/hlog.dat.1254039916931,
entries=6204, calcsize=63810201, filesize=63153162. New hlog
/hbase/.logs/ubuntu9,60020,1254037278829/hlog.dat.1254040005995
2009-09-27 16:27:17,571 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x123fa71ce2e0002 to sun.nio.ch.SelectionKeyImpl@2e530cf2
java.io.IOException: TIMED OUT
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-09-27 16:27:17,638 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x123fa71ce2e0006 to sun.nio.ch.SelectionKeyImpl@23bf8cd5
java.io.IOException: TIMED OUT
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-09-27 16:27:17,672 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: Disconnected, type: None, path: null
2009-09-27 16:27:17,713 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu2/192.168.33.9:2222
2009-09-27 16:27:18,209 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu2/192.168.33.9:2222
2009-09-27 16:27:20,721 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x123fa71ce2e0002 to sun.nio.ch.SelectionKeyImpl@42a014ef
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
2009-09-27 16:27:20,722 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown input
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-09-27 16:27:20,722 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-09-27 16:27:20,723 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x123fa71ce2e0006 to sun.nio.ch.SelectionKeyImpl@1bbe7915
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
2009-09-27 16:27:20,723 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown input
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-09-27 16:27:20,723 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-09-27 16:27:21,149 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu12/192.168.33.12:2222
2009-09-27 16:27:21,435 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu12/192.168.33.12:2222
2009-09-27 16:27:24,161 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x123fa71ce2e0002 to sun.nio.ch.SelectionKeyImpl@c86965
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
2009-09-27 16:27:24,161 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown input
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-09-27 16:27:24,162 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-09-27 16:27:24,162 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x123fa71ce2e0006 to sun.nio.ch.SelectionKeyImpl@657f6649
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
2009-09-27 16:27:24,163 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown input
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-09-27 16:27:24,163 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-09-27 16:27:25,367 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu10/192.168.33.10:2222
2009-09-27 16:27:25,411 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu10/192.168.33.10:2222
2009-09-27 16:27:31,368 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x123fa71ce2e0006 to sun.nio.ch.SelectionKeyImpl@f1476ae
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
2009-09-27 16:27:31,369 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown input
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-09-27 16:27:31,369 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-09-27 16:27:31,370 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x123fa71ce2e0002 to sun.nio.ch.SelectionKeyImpl@87ba13e
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
2009-09-27 16:27:31,370 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown input
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-09-27 16:27:31,371 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-09-27 16:27:31,758 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu2/192.168.33.9:2222
2009-09-27 16:27:31,876 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu2/192.168.33.9:2222
2009-09-27 16:27:34,508 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.33.5:40133 remote=ubuntu2/192.168.33.9:2222]
2009-09-27 16:27:34,508 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-09-27 16:27:34,509 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC
Server handler 1 on 60020 took 35745ms appending an edit to hlog;
editcount=4530
2009-09-27 16:27:34,510 WARN org.apache.hadoop.hbase.regionserver.HLog:
regionserver/192.168.33.5:60020.logFlusher took 29882ms optional sync'ing
hlog; editcount=4536
2009-09-27 16:27:34,510 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.33.5:40134 remote=ubuntu2/192.168.33.9:2222]
2009-09-27 16:27:34,511 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-09-27 16:27:34,527 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: SyncConnected, type: None, path: null
2009-09-27 16:27:39,002 INFO org.apache.hadoop.hbase.regionserver.HLog: Roll
/hbase/.logs/ubuntu9,60020,1254037278829/hlog.dat.1254040005995,
entries=6199, calcsize=63811594, filesize=63154324. New hlog
/hbase/.logs/ubuntu9,60020,1254037278829/hlog.dat.1254040058990
2009-09-27 16:27:52,662 INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Too many store files
for region webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039664595_3865,1254040068963:
5, requesting compaction and waiting
2009-09-27 16:27:53,968 INFO org.apache.hadoop.hbase.regionserver.HLog: Roll
/hbase/.logs/ubuntu9,60020,1254037278829/hlog.dat.1254040058990,
entries=6192, calcsize=63754896, filesize=63096792. New hlog
/hbase/.logs/ubuntu9,60020,1254037278829/hlog.dat.1254040073957
2009-09-27 16:28:08,128 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Blocking updates for 'IPC Server handler 6 on 60020' on region
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039664595_3865,1254040068963:
memstore size 128.1m is >= than blocking 128.0m size
2009-09-27 16:28:16,518 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039664595_3865,1254040068963
in 1mins, 30sec
2009-09-27 16:28:16,518 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting split of region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039664595_3865,1254040068963
2009-09-27 16:28:16,668 INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Compaction has
completed, we waited 12500ms, finishing flush of region webpage,http:\x2F\
x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039664595_3865,1254040068963
2009-09-27 16:28:21,064 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039664595_3865,1254040068963
2009-09-27 16:28:21,064 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Unblocking updates for region
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039664595_3865,1254040068963
'IPC Server handler 6 on 60020'
2009-09-27 16:28:21,796 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039664595_3865,1254040096527/308407564
available; sequence id is 85972
2009-09-27 16:28:21,796 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039664595_3865,1254040096527
2009-09-27 16:28:21,968 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039916262_6503,1254040096527/1278943722
available; sequence id is 85973
2009-09-27 16:28:21,969 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039916262_6503,1254040096527
2009-09-27 16:28:21,984 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META
updated, and report to master all successful. Old region=REGION => {NAME =>
'webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039664595_3865,1254040068963',
STARTKEY => 'http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039664595_3865',
ENDKEY => '', ENCODED => 1740143430, OFFLINE => true, SPLIT => true, TABLE
=> {{NAME => 'webpage', FAMILIES => [{NAME => 'CF_CONTENT', COMPRESSION =>
'NONE', VERSIONS => '2', TTL => '2147483647', BLOCKSIZE => '65536',
IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'CF_INFORMATION',
COMPRESSION => 'NONE', VERSIONS => '1', TTL => '2147483647', BLOCKSIZE =>
'65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, new regions:
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039664595_3865,1254040096527,
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039916262_6503,1254040096527.
Split took 5sec
2009-09-27 16:31:06,980 WARN org.apache.hadoop.hdfs.DFSClient:
DFSOutputStream ResponseProcessor exception for block
blk_-2072248271029020079_1170java.net.SocketTimeoutException: 69000 millis
timeout while waiting for channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/192.168.33.5:54224 remote=/
192.168.33.5:50010]
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.DataInputStream.readFully(DataInputStream.java:178)
at java.io.DataInputStream.readLong(DataInputStream.java:399)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2369)
2009-09-27 16:31:06,980 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_-2072248271029020079_1170 bad datanode[0]
192.168.33.5:50010
2009-09-27 16:31:06,981 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_-2072248271029020079_1170 in pipeline
192.168.33.5:50010, 192.168.33.6:50010, 192.168.33.8:50010: bad datanode
192.168.33.5:50010
2009-09-27 16:31:27,990 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: /192.168.33.6:50020. Already tried 0 time(s).
2009-09-27 16:33:21,350 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040043019_9471,1254040612499
2009-09-27 16:33:21,351 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040106544_11251,1254040612499
2009-09-27 16:33:21,351 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040043019_9471,1254040612499
2009-09-27 16:33:21,537 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040043019_9471,1254040612499/867215477
available; sequence id is 129132
2009-09-27 16:33:21,537 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040106544_11251,1254040612499
2009-09-27 16:33:21,537 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040043019_9471,1254040612499
2009-09-27 16:33:21,859 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040106544_11251,1254040612499/2135003529
available; sequence id is 129133
2009-09-27 16:33:50,822 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: compactions no longer
limited
2009-09-27 16:34:10,671 WARN org.apache.hadoop.hdfs.DFSClient:
DFSOutputStream ResponseProcessor exception for block
blk_-2072248271029020079_1234java.net.SocketTimeoutException: 66000 millis
timeout while waiting for channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/192.168.33.5:43480 remote=/
192.168.33.6:50010]
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.DataInputStream.readFully(DataInputStream.java:178)
at java.io.DataInputStream.readLong(DataInputStream.java:399)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2369)
2009-09-27 16:34:31,248 WARN org.apache.hadoop.hdfs.DFSClient:
DFSOutputStream ResponseProcessor exception for block
blk_-2143462669730028869_1282java.net.SocketTimeoutException: 69000 millis
timeout while waiting for channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/192.168.33.5:46689 remote=/
192.168.33.5:50010]
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.DataInputStream.readFully(DataInputStream.java:178)
at java.io.DataInputStream.readLong(DataInputStream.java:399)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2369)
2009-09-27 16:34:32,647 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_-2072248271029020079_1234 bad datanode[0]
192.168.33.6:50010
2009-09-27 16:34:32,648 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_-2072248271029020079_1234 in pipeline
192.168.33.6:50010, 192.168.33.8:50010: bad datanode 192.168.33.6:50010
2009-09-27 16:34:32,685 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC
Server handler 7 on 60020 took 67075ms appending an edit to hlog;
editcount=6073
2009-09-27 16:34:32,686 WARN org.apache.hadoop.hbase.regionserver.HLog:
regionserver/192.168.33.5:60020.logFlusher took 58173ms optional sync'ing
hlog; editcount=6074
2009-09-27 16:34:32,998 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_-2143462669730028869_1282 bad datanode[0]
192.168.33.5:50010
2009-09-27 16:34:32,998 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_-2143462669730028869_1282 in pipeline
192.168.33.5:50010, 192.168.33.6:50010, 192.168.33.8:50010: bad datanode
192.168.33.5:50010
2009-09-27 16:34:33,007 INFO org.apache.hadoop.hbase.regionserver.HLog: Roll
/hbase/.logs/ubuntu9,60020,1254037278829/hlog.dat.1254040073957,
entries=6193, calcsize=63755028, filesize=63097127. New hlog
/hbase/.logs/ubuntu9,60020,1254037278829/hlog.dat.1254040472966
2009-09-27 16:34:45,212 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC
Server handler 3 on 60020 took 9528ms appending an edit to hlog;
editcount=1152
2009-09-27 16:34:45,213 WARN org.apache.hadoop.hbase.regionserver.HLog:
regionserver/192.168.33.5:60020.logFlusher took 2527ms optional sync'ing
hlog; editcount=1158
2009-09-27 16:34:46,593 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040043019_9471,1254040612499
in 1mins, 25sec
2009-09-27 16:34:46,593 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040106544_11251,1254040612499
2009-09-27 16:34:58,504 INFO org.apache.hadoop.hbase.regionserver.HLog: Roll
/hbase/.logs/ubuntu9,60020,1254037278829/hlog.dat.1254040472966,
entries=6187, calcsize=63759284, filesize=63101288. New hlog
/hbase/.logs/ubuntu9,60020,1254037278829/hlog.dat.1254040498496
2009-09-27 16:34:59,153 INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Too many store files
for region webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040106544_11251,1254040612499:
5, requesting compaction and waiting
2009-09-27 16:35:08,662 INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Compaction has
completed, we waited 10000ms, finishing flush of region webpage,http:\x2F\
x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040106544_11251,1254040612499
2009-09-27 16:35:09,828 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040106544_11251,1254040612499
in 23sec
2009-09-27 16:35:09,828 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting split of region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040106544_11251,1254040612499
2009-09-27 16:35:09,836 WARN org.apache.hadoop.hbase.regionserver.HRegion:
Clock skew; parent regions id is 1254040612499 but current time here is
1254040509836
2009-09-27 16:35:13,132 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC
Server handler 9 on 60020 took 1322ms appending an edit to hlog;
editcount=4679
2009-09-27 16:35:13,634 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040106544_11251,1254040612499
2009-09-27 16:35:14,437 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040106544_11251,1254040612500/512381466
available; sequence id is 140212
2009-09-27 16:35:14,438 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040106544_11251,1254040612500
2009-09-27 16:35:14,638 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040237301_15439,1254040612500/95607190
available; sequence id is 140213
2009-09-27 16:35:14,639 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040237301_15439,1254040612500
2009-09-27 16:35:14,652 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META
updated, and report to master all successful. Old region=REGION => {NAME =>
'webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040106544_11251,1254040612499',
STARTKEY => 'http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040106544_11251',
ENDKEY => '', ENCODED => 2135003529, OFFLINE => true, SPLIT => true, TABLE
=> {{NAME => 'webpage', FAMILIES => [{NAME => 'CF_CONTENT', COMPRESSION =>
'NONE', VERSIONS => '2', TTL => '2147483647', BLOCKSIZE => '65536',
IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'CF_INFORMATION',
COMPRESSION => 'NONE', VERSIONS => '1', TTL => '2147483647', BLOCKSIZE =>
'65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, new regions:
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040106544_11251,1254040612500,
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040237301_15439,1254040612500.
Split took 4sec
2009-09-27 16:35:15,613 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040106544_11251,1254040612500
2009-09-27 16:35:15,613 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040237301_15439,1254040612500
2009-09-27 16:35:15,613 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040106544_11251,1254040612500
2009-09-27 16:35:15,769 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040106544_11251,1254040612500/512381466
available; sequence id is 140212
2009-09-27 16:35:15,770 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040237301_15439,1254040612500
2009-09-27 16:35:15,770 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040106544_11251,1254040612500
2009-09-27 16:35:15,945 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040237301_15439,1254040612500/95607190
available; sequence id is 140213
2009-09-27 16:35:25,402 INFO org.apache.hadoop.hbase.regionserver.HLog: Roll
/hbase/.logs/ubuntu9,60020,1254037278829/hlog.dat.1254040498496,
entries=6194, calcsize=63761088, filesize=63102912. New hlog
/hbase/.logs/ubuntu9,60020,1254037278829/hlog.dat.1254040525388
2009-09-27 16:35:25,967 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040106544_11251,1254040612500
in 10sec
2009-09-27 16:35:25,968 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040237301_15439,1254040612500
2009-09-27 16:36:09,228 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC
Server handler 3 on 60020 took 39592ms appending an edit to hlog;
editcount=1415
2009-09-27 16:36:09,228 WARN org.apache.hadoop.hbase.regionserver.HLog:
regionserver/192.168.33.5:60020.logFlusher took 36542ms optional sync'ing
hlog; editcount=1421
2009-09-27 16:36:15,344 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC
Server handler 3 on 60020 took 1053ms appending an edit to hlog;
editcount=3575
2009-09-27 16:36:19,598 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040237301_15439,1254040612500
in 53sec
2009-09-27 16:36:19,598 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting split of region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040237301_15439,1254040612500
2009-09-27 16:36:19,606 WARN org.apache.hadoop.hbase.regionserver.HRegion:
Clock skew; parent regions id is 1254040612500 but current time here is
1254040579606
2009-09-27 16:36:22,416 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040237301_15439,1254040612500
2009-09-27 16:36:22,966 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040237301_15439,1254040612501/2088743043
available; sequence id is 146389
2009-09-27 16:36:22,966 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040237301_15439,1254040612501
2009-09-27 16:36:23,092 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040368751_18455,1254040612501/1245778272
available; sequence id is 146390
2009-09-27 16:36:23,093 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040368751_18455,1254040612501
2009-09-27 16:36:23,109 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META
updated, and report to master all successful. Old region=REGION => {NAME =>
'webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040237301_15439,1254040612500',
STARTKEY => 'http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040237301_15439',
ENDKEY => '', ENCODED => 95607190, OFFLINE => true, SPLIT => true, TABLE =>
{{NAME => 'webpage', FAMILIES => [{NAME => 'CF_CONTENT', COMPRESSION =>
'NONE', VERSIONS => '2', TTL => '2147483647', BLOCKSIZE => '65536',
IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'CF_INFORMATION',
COMPRESSION => 'NONE', VERSIONS => '1', TTL => '2147483647', BLOCKSIZE =>
'65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, new regions:
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040237301_15439,1254040612501,
webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040368751_18455,1254040612501.
Split took 3sec
2009-09-27 16:37:34,798 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x123fa71ce2e0002 to sun.nio.ch.SelectionKeyImpl@29dc7e1f
java.io.IOException: TIMED OUT
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-09-27 16:37:34,820 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x123fa71ce2e0006 to sun.nio.ch.SelectionKeyImpl@6c96f1cb
java.io.IOException: TIMED OUT
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-09-27 16:37:34,899 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: Disconnected, type: None, path: null
2009-09-27 16:37:35,275 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu12/192.168.33.12:2222
2009-09-27 16:37:35,880 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu12/192.168.33.12:2222
2009-09-27 16:37:38,281 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x123fa71ce2e0002 to sun.nio.ch.SelectionKeyImpl@41d77c9f
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
2009-09-27 16:37:38,281 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown input
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-09-27 16:37:38,282 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-09-27 16:37:38,282 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x123fa71ce2e0006 to sun.nio.ch.SelectionKeyImpl@2a36cc7e
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
2009-09-27 16:37:38,282 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown input
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-09-27 16:37:38,283 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-09-27 16:37:38,769 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu10/192.168.33.10:2222
2009-09-27 16:37:39,111 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: ubuntu6/192.168.33.7:9000. Already tried 0 time(s).
2009-09-27 16:37:39,158 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu10/192.168.33.10:2222
2009-09-27 16:37:52,108 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x123fa71ce2e0006 to sun.nio.ch.SelectionKeyImpl@73ab79d8
java.io.IOException: TIMED OUT
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-09-27 16:37:52,109 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.net.SocketException: Transport endpoint is not connected
at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-09-27 16:37:52,498 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x123fa71ce2e0002 to sun.nio.ch.SelectionKeyImpl@3305fed5
java.io.IOException: TIMED OUT
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-09-27 16:37:52,498 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.net.SocketException: Transport endpoint is not connected
at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-09-27 16:37:53,857 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu2/192.168.33.9:2222
2009-09-27 16:37:53,938 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu2/192.168.33.9:2222
2009-09-27 16:38:00,121 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: ubuntu6/192.168.33.7:9000. Already tried 1 time(s).
2009-09-27 16:38:04,502 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.33.5:37122 remote=ubuntu2/192.168.33.9:2222]
2009-09-27 16:38:04,502 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-09-27 16:38:04,503 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x123fa71ce2e0002 to sun.nio.ch.SelectionKeyImpl@41254de4
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
at sun.nio.ch.IOUtil.read(IOUtil.java:200)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:651)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
2009-09-27 16:38:04,504 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown input
java.net.SocketException: Transport endpoint is not connected
at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-09-27 16:38:04,505 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.net.SocketException: Transport endpoint is not connected
at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-09-27 16:38:04,513 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x123fa71ce2e0006 to sun.nio.ch.SelectionKeyImpl@4e9d6c1e
java.net.SocketException: Connection reset by peer
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
2009-09-27 16:38:04,513 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown input
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-09-27 16:38:04,514 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-09-27 16:38:04,693 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu12/192.168.33.12:2222
2009-09-27 16:38:04,693 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.33.5:45155 remote=ubuntu12/192.168.33.12:2222]
2009-09-27 16:38:04,694 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-09-27 16:38:04,695 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: Expired, type: None, path: null
2009-09-27 16:38:04,695 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x123fa71ce2e0002 to sun.nio.ch.SelectionKeyImpl@3eac3b6
java.io.IOException: Session Expired
at
org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:548)
at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:661)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
2009-09-27 16:38:04,695 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session
expired
2009-09-27 16:38:04,699 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
request=0.0, regions=5, stores=9, storefiles=9, storefileIndexSize=0,
memstoreSize=0, usedHeap=104, maxHeap=2993, blockCacheSize=5152344,
blockCacheFree=622695656, blockCacheCount=1, blockCacheHitRatio=0
2009-09-27 16:38:05,092 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu12/192.168.33.12:2222
2009-09-27 16:38:05,093 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.33.5:45156 remote=ubuntu12/192.168.33.12:2222]
2009-09-27 16:38:05,093 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-09-27 16:38:05,094 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x123fa71ce2e0006 to sun.nio.ch.SelectionKeyImpl@1e8516be
java.io.IOException: Session Expired
at
org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:548)
at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:661)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
2009-09-27 16:38:05,094 INFO org.apache.zookeeper.ZooKeeper: Closing
session: 0x123fa71ce2e0006
2009-09-27 16:38:05,095 INFO org.apache.zookeeper.ClientCnxn: Closing
ClientCnxn for session: 0x123fa71ce2e0006
2009-09-27 16:38:05,095 INFO org.apache.zookeeper.ClientCnxn: Disconnecting
ClientCnxn for session: 0x123fa71ce2e0006
2009-09-27 16:38:05,095 INFO org.apache.zookeeper.ZooKeeper: Session:
0x123fa71ce2e0006 closed
2009-09-27 16:38:05,096 INFO org.apache.zookeeper.ClientCnxn: EventThread
shut down
2009-09-27 16:38:05,297 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 5 on 60020, call openScanner([B@6d3c1c92, startRow=, stopRow=,
maxVersions=1, caching=1, timeRange=[0,9223372036854775807),
families={(family=info, columns={}}) from 192.168.33.7:60950: error:
java.io.IOException: Server not running, aborting
java.io.IOException: Server not running, aborting
at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1865)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-09-27 16:38:05,405 INFO org.apache.hadoop.hbase.regionserver.LogRoller:
LogRoller exiting.
2009-09-27 16:38:05,952 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: worker thread exiting
2009-09-27 16:38:07,249 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 4 on 60020, call getClosestRowBefore([B@47a4fd1, [B@7a0e24a2,
[B@36874648) from 192.168.33.7:60956: error: java.io.IOException: Server not
running, aborting
java.io.IOException: Server not running, aborting
at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1748)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-09-27 16:38:07,511 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
server on 60020
2009-09-27 16:38:07,512 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 0 on 60020: exiting
2009-09-27 16:38:07,512 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 5 on 60020: exiting
2009-09-27 16:38:07,520 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 7 on 60020: exiting
2009-09-27 16:38:07,520 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 2 on 60020: exiting
2009-09-27 16:38:07,521 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
2009-09-27 16:38:07,521 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC
Server Responder
2009-09-27 16:38:07,522 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 9 on 60020: exiting
2009-09-27 16:38:07,523 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 8 on 60020: exiting
2009-09-27 16:38:07,523 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 6 on 60020: exiting
2009-09-27 16:38:07,524 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 3 on 60020: exiting
2009-09-27 16:38:07,524 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 1 on 60020: exiting
2009-09-27 16:38:07,539 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 4 on 60020: exiting
2009-09-27 16:38:07,562 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC
Server listener on 60020
2009-09-27 16:38:07,683 INFO
org.apache.hadoop.hbase.regionserver.LogFlusher:
regionserver/192.168.33.5:60020.logFlusher exiting
2009-09-27 16:38:07,683 INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher:
regionserver/192.168.33.5:60020.cacheFlusher exiting
2009-09-27 16:38:07,684 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer$MajorCompactionChecker:
regionserver/192.168.33.5:60020.majorCompactionChecker exiting
2009-09-27 16:38:07,687 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread:
regionserver/192.168.33.5:60020.compactor exiting
2009-09-27 16:38:07,726 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: On abort, closed hlog
2009-09-27 16:38:07,733 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040106544_11251,1254040612500
2009-09-27 16:38:07,733 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039598290_2497,1254040068963
2009-09-27 16:38:07,733 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed -ROOT-,,0
2009-09-27 16:38:07,734 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254039498012_112,1254040068962
2009-09-27 16:38:07,734 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1254040043019_9471,1254040612499
2009-09-27 16:38:07,734 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: aborting server at:
192.168.33.5:60020
2009-09-27 16:38:07,734 INFO org.apache.zookeeper.ZooKeeper: Closing
session: 0x123fa71ce2e0002
2009-09-27 16:38:07,734 INFO org.apache.zookeeper.ClientCnxn: Closing
ClientCnxn for session: 0x123fa71ce2e0002
2009-09-27 16:38:07,735 INFO org.apache.zookeeper.ClientCnxn: Disconnecting
ClientCnxn for session: 0x123fa71ce2e0002
2009-09-27 16:38:07,735 INFO org.apache.zookeeper.ZooKeeper: Session:
0x123fa71ce2e0002 closed
2009-09-27 16:38:07,735 INFO org.apache.zookeeper.ClientCnxn: EventThread
shut down
2009-09-27 16:38:07,837 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver/
192.168.33.5:60020 exiting
2009-09-27 16:38:09,670 INFO org.apache.hadoop.hbase.Leases:
regionserver/192.168.33.5:60020.leaseChecker closing leases
2009-09-27 16:38:09,670 INFO org.apache.hadoop.hbase.Leases:
regionserver/192.168.33.5:60020.leaseChecker closed leases
2009-09-27 16:38:09,672 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Starting shutdown
thread.
2009-09-27 16:38:09,673 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Shutdown thread complete