You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Brian Bockelman <bb...@cse.unl.edu> on 2010/05/15 02:22:05 UTC

NameNode deadlocked (help?)

Hey guys,

I know it's 5PM on a Friday, but we just saw one of our big cluster's namenode's deadlock.

This is 0.19.1; does this ring a bell for anyone?  I haven't had any time to start going through source code, but I figured I'd send out a SOS in case if this looked familiar.

We had restarted this cluster a few hours ago and made the following changes:

1) Increased the number of datanode handlers from 10 to 40.
2) Increased ipc.server.listen.queue.size from 128 to 256.

If nothing else, I figure a deadlocked NN might be interesting to devs...

Brian

2010-05-14 17:11:30
Full thread dump Java HotSpot(TM) 64-Bit Server VM (11.2-b01 mixed mode):

"IPC Server handler 39 on 9000" daemon prio=10 tid=0x00002aaafc181400 nid=0x4cd waiting for monitor entry [0x0000000045962000..0x0000000045962d90]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
	- waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)

"IPC Server handler 38 on 9000" daemon prio=10 tid=0x00002aaafc17f800 nid=0x4cc waiting for monitor entry [0x0000000045861000..0x0000000045861d10]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getStats(FSNamesystem.java:3326)
	- waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.getStats(NameNode.java:505)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)

"IPC Server handler 37 on 9000" daemon prio=10 tid=0x00002aaafc17e000 nid=0x4cb waiting for monitor entry [0x000000004575f000..0x0000000045760a90]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
	- waiting to lock <0x00002aaab3843e40> (a org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)

"IPC Server handler 36 on 9000" daemon prio=10 tid=0x00002aaafc17c400 nid=0x4ca waiting for monitor entry [0x000000004565e000..0x000000004565fa10]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:3281)
	- waiting to lock <0x00002aaab3843e40> (a org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:649)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)

"IPC Server handler 35 on 9000" daemon prio=10 tid=0x00002aaafc17a800 nid=0x4c9 waiting for monitor entry [0x000000004555e000..0x000000004555eb90]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
	- waiting to lock <0x00002aaab3843e40> (a org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
	at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)

"IPC Server handler 34 on 9000" daemon prio=10 tid=0x00002aaafc178c00 nid=0x4c8 waiting for monitor entry [0x000000004545d000..0x000000004545db10]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
	- waiting to lock <0x00002aaab3843e40> (a org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
	at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)

"IPC Server handler 33 on 9000" daemon prio=10 tid=0x00002aaafc177400 nid=0x4c7 waiting for monitor entry [0x000000004535c000..0x000000004535cc90]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
	- waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)

"IPC Server handler 32 on 9000" daemon prio=10 tid=0x00002aaafc175800 nid=0x4c6 waiting for monitor entry [0x000000004525a000..0x000000004525bc10]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
	- waiting to lock <0x00002aaab3843e40> (a org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)

"IPC Server handler 31 on 9000" daemon prio=10 tid=0x00002aaafc173c00 nid=0x4c5 waiting for monitor entry [0x000000004515a000..0x000000004515ad90]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:3281)
	- waiting to lock <0x00002aaab3843e40> (a org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:649)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)

"IPC Server handler 30 on 9000" daemon prio=10 tid=0x00002aaafc172400 nid=0x4c4 waiting for monitor entry [0x0000000045059000..0x0000000045059d10]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
	- waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)

"IPC Server handler 29 on 9000" daemon prio=10 tid=0x00002aaafc171000 nid=0x4c3 waiting for monitor entry [0x0000000044f57000..0x0000000044f58a90]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
	- waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)

"IPC Server handler 28 on 9000" daemon prio=10 tid=0x00002aaafc16f400 nid=0x4c2 waiting for monitor entry [0x0000000044e56000..0x0000000044e57a10]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
	- waiting to lock <0x00002aaab3843e40> (a org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)

"IPC Server handler 27 on 9000" daemon prio=10 tid=0x00002aaafc16d800 nid=0x4c1 waiting for monitor entry [0x0000000044d56000..0x0000000044d56b90]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
	- waiting to lock <0x00002aaab3843e40> (a org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)

"IPC Server handler 26 on 9000" daemon prio=10 tid=0x00002aaafc16bc00 nid=0x4c0 waiting for monitor entry [0x0000000044c55000..0x0000000044c55b10]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
	- waiting to lock <0x00002aaab3843e40> (a org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)

"IPC Server handler 25 on 9000" daemon prio=10 tid=0x00002aaafc16a400 nid=0x4bf waiting for monitor entry [0x0000000044b54000..0x0000000044b54c90]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
	- waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)

"IPC Server handler 24 on 9000" daemon prio=10 tid=0x00002aaafc168800 nid=0x4be waiting for monitor entry [0x0000000044a53000..0x0000000044a53c10]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
	- waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
	at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)

"IPC Server handler 23 on 9000" daemon prio=10 tid=0x00002aaafc166c00 nid=0x4bd waiting for monitor entry [0x0000000044952000..0x0000000044952d90]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
	- waiting to lock <0x00002aaab3843e40> (a org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)

"IPC Server handler 22 on 9000" daemon prio=10 tid=0x00002aaafc165400 nid=0x4bc waiting for monitor entry [0x0000000044851000..0x0000000044851d10]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
	- waiting to lock <0x00002aaab3843e40> (a org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)

"IPC Server handler 21 on 9000" daemon prio=10 tid=0x00002aaafc164000 nid=0x4bb waiting for monitor entry [0x0000000044750000..0x0000000044750a90]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
	- waiting to lock <0x00002aaab3843e40> (a org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)

"IPC Server handler 20 on 9000" daemon prio=10 tid=0x00002aaafc162400 nid=0x4ba waiting for monitor entry [0x000000004464f000..0x000000004464fa10]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
	- waiting to lock <0x00002aaab3843e40> (a org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
	at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)

"IPC Server handler 19 on 9000" daemon prio=10 tid=0x00002aaafc488400 nid=0x4b9 waiting for monitor entry [0x000000004454e000..0x000000004454eb90]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
	- waiting to lock <0x00002aaab3843e40> (a org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
	at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)

"IPC Server handler 18 on 9000" daemon prio=10 tid=0x00002aaafc486800 nid=0x4b8 waiting for monitor entry [0x000000004444d000..0x000000004444db10]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.nextGenerationStampForBlock(FSNamesystem.java:4548)
	- waiting to lock <0x00002aaab3843e40> (a org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.nextGenerationStamp(NameNode.java:402)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)

"IPC Server handler 17 on 9000" daemon prio=10 tid=0x00002aaafc484c00 nid=0x4b7 waiting for monitor entry [0x000000004434c000..0x000000004434cc90]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.nextGenerationStampForBlock(FSNamesystem.java:4548)
	- waiting to lock <0x00002aaab3843e40> (a org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.nextGenerationStamp(NameNode.java:402)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)

"IPC Server handler 16 on 9000" daemon prio=10 tid=0x00002aaafc483000 nid=0x4b6 waiting for monitor entry [0x000000004424a000..0x000000004424bc10]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
	- waiting to lock <0x00002aaab3843e40> (a org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)

"IPC Server handler 15 on 9000" daemon prio=10 tid=0x00002aaafc481800 nid=0x4b5 waiting for monitor entry [0x000000004414a000..0x000000004414ad90]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
	- waiting to lock <0x00002aaab3843e40> (a org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)

"IPC Server handler 14 on 9000" daemon prio=10 tid=0x00002aaafc47fc00 nid=0x4b4 waiting for monitor entry [0x0000000044049000..0x0000000044049d10]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
	- waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)

"IPC Server handler 13 on 9000" daemon prio=10 tid=0x00002aaafc47e000 nid=0x4b3 waiting for monitor entry [0x0000000043f48000..0x0000000043f48a90]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
	- waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
	at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)

"IPC Server handler 12 on 9000" daemon prio=10 tid=0x00002aaafc47c400 nid=0x4b2 waiting for monitor entry [0x0000000043e46000..0x0000000043e47a10]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
	- waiting to lock <0x00002aaab3843e40> (a org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)

"IPC Server handler 11 on 9000" daemon prio=10 tid=0x00002aaafc47b400 nid=0x4b1 waiting for monitor entry [0x0000000043d45000..0x0000000043d46b90]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
	- waiting to lock <0x00002aaab3843e40> (a org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)

"IPC Server handler 10 on 9000" daemon prio=10 tid=0x00002aaafc479800 nid=0x4b0 waiting for monitor entry [0x0000000043c45000..0x0000000043c45b10]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
	- waiting to lock <0x00002aaab3843e40> (a org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
	at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)

"IPC Server handler 9 on 9000" daemon prio=10 tid=0x00002aaafc477c00 nid=0x4af waiting for monitor entry [0x0000000043b43000..0x0000000043b44c90]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
	- waiting to lock <0x00002aaab3843e40> (a org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)

"IPC Server handler 8 on 9000" daemon prio=10 tid=0x00002aaafc476400 nid=0x4ae waiting for monitor entry [0x0000000043a42000..0x0000000043a43c10]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
	- waiting to lock <0x00002aaab3843e40> (a org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)

"IPC Server handler 7 on 9000" daemon prio=10 tid=0x00002aaafc474c00 nid=0x4ad waiting for monitor entry [0x0000000043942000..0x0000000043942d90]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
	- waiting to lock <0x00002aaab3843e40> (a org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)

"IPC Server handler 6 on 9000" daemon prio=10 tid=0x00002aaafc473000 nid=0x4ac waiting for monitor entry [0x0000000043841000..0x0000000043841d10]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
	- waiting to lock <0x00002aaab3843e40> (a org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)

"IPC Server handler 5 on 9000" daemon prio=10 tid=0x00002aaafc471400 nid=0x4ab waiting for monitor entry [0x0000000043740000..0x0000000043740a90]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
	- waiting to lock <0x00002aaab3843e40> (a org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)

"IPC Server handler 4 on 9000" daemon prio=10 tid=0x00002aaafc121800 nid=0x4aa waiting for monitor entry [0x000000004363f000..0x000000004363fa10]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
	- waiting to lock <0x00002aaab3843e40> (a org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)

"IPC Server handler 3 on 9000" daemon prio=10 tid=0x00002aaafc11fc00 nid=0x4a9 waiting for monitor entry [0x000000004353d000..0x000000004353eb90]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
	- waiting to lock <0x00002aaab3843e40> (a org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)

"IPC Server handler 2 on 9000" daemon prio=10 tid=0x00002aaafc11e400 nid=0x4a8 waiting for monitor entry [0x000000004343c000..0x000000004343db10]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
	- waiting to lock <0x00002aaab3843e40> (a org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)

"IPC Server handler 1 on 9000" daemon prio=10 tid=0x00002aaafc467800 nid=0x4a7 waiting for monitor entry [0x000000004333b000..0x000000004333cc90]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
	- waiting to lock <0x00002aaab3843e40> (a org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)

"IPC Server handler 0 on 9000" daemon prio=10 tid=0x00002aaafc466400 nid=0x4a6 waiting for monitor entry [0x000000004323b000..0x000000004323bc10]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
	- waiting to lock <0x00002aaab3843e40> (a org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)

"IPC Server listener on 9000" daemon prio=10 tid=0x00002aaafc12fc00 nid=0x4a5 waiting on condition [0x000000004313a000..0x000000004313ad90]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00002aaab3b8fd70> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
	at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:254)
	at org.apache.hadoop.ipc.Server$Connection.processData(Server.java:850)
	at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:814)
	at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:400)
	at org.apache.hadoop.ipc.Server$Listener.run(Server.java:309)

"IPC Server Responder" daemon prio=10 tid=0x00002aaafc0bbc00 nid=0x4a4 runnable [0x0000000043039000..0x0000000043039d10]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x00002aaab3b90e80> (a sun.nio.ch.Util$1)
	- locked <0x00002aaab3b90e68> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00002aaab3b90b08> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.apache.hadoop.ipc.Server$Responder.run(Server.java:458)

"SocketListener0-1" prio=10 tid=0x00002aaafc0ba400 nid=0x4a3 in Object.wait() [0x0000000042f38000..0x0000000042f38a90]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00002aaadf36cb00> (a org.mortbay.util.ThreadPool$PoolThread)
	at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:522)
	- locked <0x00002aaadf36cb00> (a org.mortbay.util.ThreadPool$PoolThread)

"SocketListener0-0" prio=10 tid=0x00002aaafc0b4c00 nid=0x4a2 in Object.wait() [0x0000000042e37000..0x0000000042e37a10]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00002aaadf36c820> (a org.mortbay.util.ThreadPool$PoolThread)
	at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:522)
	- locked <0x00002aaadf36c820> (a org.mortbay.util.ThreadPool$PoolThread)

"Acceptor ServerSocket[addr=proxy-1/169.228.130.63,port=0,localport=50070]" prio=10 tid=0x00002aaafc0c8400 nid=0x4a1 runnable [0x0000000042d36000..0x0000000042d36b90]
   java.lang.Thread.State: RUNNABLE
	at java.lang.Object.<init>(Object.java:20)
	at java.net.SocketImpl.<init>(SocketImpl.java:27)
	at java.net.PlainSocketImpl.<init>(PlainSocketImpl.java:90)
	at java.net.SocksSocketImpl.<init>(SocksSocketImpl.java:33)
	at java.net.Socket.setImpl(Socket.java:434)
	at java.net.ServerSocket.implAccept(ServerSocket.java:445)
	at java.net.ServerSocket.accept(ServerSocket.java:421)
	at org.mortbay.util.ThreadedServer.acceptSocket(ThreadedServer.java:432)
	at org.mortbay.util.ThreadedServer$Acceptor.run(ThreadedServer.java:631)

"SessionScavenger" daemon prio=10 tid=0x00002aaafc0c1000 nid=0x4a0 waiting on condition [0x0000000042c35000..0x0000000042c35b10]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(AbstractSessionManager.java:587)

"SessionScavenger" daemon prio=10 tid=0x00002aaafc0fa000 nid=0x49f waiting on condition [0x0000000040b2a000..0x0000000040b2ac90]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(AbstractSessionManager.java:587)

"SessionScavenger" daemon prio=10 tid=0x00002aaafc139800 nid=0x49a waiting on condition [0x00000000408b7000..0x00000000408b7c10]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(AbstractSessionManager.java:587)

"org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@1b0deb5f" daemon prio=10 tid=0x00002aaafc0a2800 nid=0x496 waiting for monitor entry [0x0000000042b34000..0x0000000042b34d90]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:61)
	- waiting to lock <0x00002aaab3843e40> (a org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
	at java.lang.Thread.run(Thread.java:619)

"org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@784a7df6" daemon prio=10 tid=0x00002aaafc090c00 nid=0x495 waiting for monitor entry [0x00000000411f8000..0x00000000411f8d10]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.computeReplicationWork(FSNamesystem.java:2410)
	- waiting to lock <0x00002aaab3843e40> (a org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.computeDatanodeWork(FSNamesystem.java:2376)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2333)
	at java.lang.Thread.run(Thread.java:619)

"org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@d24111a" daemon prio=10 tid=0x00002aaafc013400 nid=0x494 waiting for monitor entry [0x00000000410f7000..0x00000000410f7a90]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:349)
	- waiting to lock <0x00002aaab3843e40> (a org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
	at java.lang.Thread.run(Thread.java:619)

"org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@1c778255" daemon prio=10 tid=0x00002aaafc012c00 nid=0x493 runnable [0x00000000413da000..0x00000000413daa10]
   java.lang.Thread.State: RUNNABLE
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeStoredBlock(FSNamesystem.java:3236)
	- locked <0x00002aaab3843e40> (a org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeDatanode(FSNamesystem.java:2695)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.heartbeatCheck(FSNamesystem.java:2785)
	- locked <0x00002aaab3659dd8> (a java.util.TreeMap)
	- locked <0x00002aaab3653848> (a java.util.ArrayList)
	- locked <0x00002aaab3843e40> (a org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2312)
	at java.lang.Thread.run(Thread.java:619)

"org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@44676e3f" daemon prio=10 tid=0x00002aaafc012800 nid=0x492 waiting on condition [0x0000000040d16000..0x0000000040d16b90]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
	at java.lang.Thread.run(Thread.java:619)

"Timer thread for monitoring dfs" daemon prio=10 tid=0x00002aaafc03d800 nid=0x428 waiting for monitor entry [0x0000000040e7e000..0x0000000040e7ed10]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getStats(FSNamesystem.java:3326)
	- waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getCapacityTotal(FSNamesystem.java:3335)
	at org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics.doUpdates(FSNamesystemMetrics.java:90)
	- locked <0x00002aaab3b3c580> (a org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics)
	at org.apache.hadoop.metrics.spi.AbstractMetricsContext.timerEvent(AbstractMetricsContext.java:284)
	at org.apache.hadoop.metrics.spi.AbstractMetricsContext.access$000(AbstractMetricsContext.java:50)
	at org.apache.hadoop.metrics.spi.AbstractMetricsContext$1.run(AbstractMetricsContext.java:249)
	at java.util.TimerThread.mainLoop(Timer.java:512)
	at java.util.TimerThread.run(Timer.java:462)

"Low Memory Detector" daemon prio=10 tid=0x000000005c306c00 nid=0x426 runnable [0x0000000000000000..0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread1" daemon prio=10 tid=0x000000005c304000 nid=0x425 waiting on condition [0x0000000000000000..0x0000000042830420]
   java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=10 tid=0x000000005c300000 nid=0x424 waiting on condition [0x0000000000000000..0x000000004272f3a0]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x000000005c2fe400 nid=0x423 waiting on condition [0x0000000000000000..0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x000000005c2da800 nid=0x422 in Object.wait() [0x000000004252e000..0x000000004252ec10]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00002aaab38ec320> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
	- locked <0x00002aaab38ec320> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x000000005c2d8c00 nid=0x421 in Object.wait() [0x000000004177e000..0x000000004177ed90]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00002aaab38e9ec0> (a java.lang.ref.Reference$Lock)
	at java.lang.Object.wait(Object.java:485)
	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
	- locked <0x00002aaab38e9ec0> (a java.lang.ref.Reference$Lock)

"main" prio=10 tid=0x000000005c268400 nid=0x417 in Object.wait() [0x0000000040494000..0x0000000040494ed0]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00002aaab3b3db50> (a org.apache.hadoop.ipc.RPC$Server)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.hadoop.ipc.Server.join(Server.java:1023)
	- locked <0x00002aaab3b3db50> (a org.apache.hadoop.ipc.RPC$Server)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.join(NameNode.java:221)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:870)

"VM Thread" prio=10 tid=0x000000005c2d3400 nid=0x420 runnable 

"GC task thread#0 (ParallelGC)" prio=10 tid=0x000000005c273000 nid=0x418 runnable 

"GC task thread#1 (ParallelGC)" prio=10 tid=0x000000005c274800 nid=0x419 runnable 

"GC task thread#2 (ParallelGC)" prio=10 tid=0x000000005c276400 nid=0x41a runnable 

"GC task thread#3 (ParallelGC)" prio=10 tid=0x000000005c277c00 nid=0x41b runnable 

"GC task thread#4 (ParallelGC)" prio=10 tid=0x000000005c279400 nid=0x41c runnable 

"GC task thread#5 (ParallelGC)" prio=10 tid=0x000000005c27b000 nid=0x41d runnable 

"GC task thread#6 (ParallelGC)" prio=10 tid=0x000000005c27c800 nid=0x41e runnable 

"GC task thread#7 (ParallelGC)" prio=10 tid=0x000000005c27e000 nid=0x41f runnable 

"VM Periodic Task Thread" prio=10 tid=0x000000005c308c00 nid=0x427 waiting on condition 

JNI global references: 984

Heap
 PSYoungGen      total 203712K, used 107631K [0x00002aaadd0b0000, 0x00002aaaf1e00000, 0x00002aaaf1e00000)
  eden space 124096K, 86% used [0x00002aaadd0b0000,0x00002aaae39cbe58,0x00002aaae49e0000)
  from space 79616K, 0% used [0x00002aaae49e0000,0x00002aaae49e0000,0x00002aaae97a0000)
  to   space 113728K, 0% used [0x00002aaaeaef0000,0x00002aaaeaef0000,0x00002aaaf1e00000)
 PSOldGen        total 682688K, used 682687K [0x00002aaab3600000, 0x00002aaadd0b0000, 0x00002aaadd0b0000)
  object space 682688K, 99% used [0x00002aaab3600000,0x00002aaadd0affe8,0x00002aaadd0b0000)
 PSPermGen       total 21248K, used 16689K [0x00002aaaae200000, 0x00002aaaaf6c0000, 0x00002aaab3600000)
  object space 21248K, 78% used [0x00002aaaae200000,0x00002aaaaf24c660,0x00002aaaaf6c0000)

Re: NameNode deadlocked (help?)

Posted by Steve Loughran <st...@apache.org>.
Brian Bockelman wrote:
> On May 17, 2010, at 5:25 AM, Steve Loughran wrote:
> 
>> Brian Bockelman wrote:
>>> On May 14, 2010, at 8:27 PM, Todd Lipcon wrote:
>>>> Hey Brian,
>>>>
>>>> Yep, excessive GC definitely sounds like a likely culprit. I'm surprised you
>>>> didn't see OOMEs in the log, though.
>>>>
>>> We didn't until the third restart today.  I have no clue why we haven't seen this in the past 9 months of this cluster though...
>>> Anyhow, it looks like this might have done the trick... the sysadmin is heading over to kick over a few errant datanodes, and we should be able to get out of safemode soon.  Luckily, it's a 4-day weekend in Europe and otherwise a Friday evening in the US, so there's only a few folks using it.
>> good thing we europeans have long weekends.
>>
> 
> :) Indeed
> 
>>>> If you want to monitor GC, I'd recommend adding -verbose:gc
>>>> -XX:+PrintGCDetails -XX:+PrintGCDateStamps to your java options -
>>>> occasionally useful for times like this.
>>>>
>> What are your current GC options? Played with compressed object pointers yet?
> 
> I've been eyeballing them, but haven't had any chance yet.  We probably won't mess with them until we start to run out of RAM on the machine themselves.
> 
> This particular instance was a simple oversight - there was no need to try and fit the NN into a 1GB heap on a dedicated machine.  I tell folks 1GB RAM per 1M objects.  It's almost always an over-estimate but, better safe than deadlocked on a Friday evening...
> 

I've been using compressed pointers on JRockit for a long time, a very 
nice JVM that doesn't ever seem to run out of stack when you 
accidentally tail recurse without end. The Sun JVM pointers are newer, 
not had any problems with that part of the JVM, and the benefits in both 
memory consumption and possibly in cache hits make it very appealing.

Re: NameNode deadlocked (help?)

Posted by Brian Bockelman <bb...@cse.unl.edu>.
On May 17, 2010, at 5:25 AM, Steve Loughran wrote:

> Brian Bockelman wrote:
>> On May 14, 2010, at 8:27 PM, Todd Lipcon wrote:
>>> Hey Brian,
>>> 
>>> Yep, excessive GC definitely sounds like a likely culprit. I'm surprised you
>>> didn't see OOMEs in the log, though.
>>> 
>> We didn't until the third restart today.  I have no clue why we haven't seen this in the past 9 months of this cluster though...
>> Anyhow, it looks like this might have done the trick... the sysadmin is heading over to kick over a few errant datanodes, and we should be able to get out of safemode soon.  Luckily, it's a 4-day weekend in Europe and otherwise a Friday evening in the US, so there's only a few folks using it.
> 
> good thing we europeans have long weekends.
> 

:) Indeed

>>> If you want to monitor GC, I'd recommend adding -verbose:gc
>>> -XX:+PrintGCDetails -XX:+PrintGCDateStamps to your java options -
>>> occasionally useful for times like this.
>>> 
> 
> What are your current GC options? Played with compressed object pointers yet?

I've been eyeballing them, but haven't had any chance yet.  We probably won't mess with them until we start to run out of RAM on the machine themselves.

This particular instance was a simple oversight - there was no need to try and fit the NN into a 1GB heap on a dedicated machine.  I tell folks 1GB RAM per 1M objects.  It's almost always an over-estimate but, better safe than deadlocked on a Friday evening...

Brian

Re: NameNode deadlocked (help?)

Posted by Steve Loughran <st...@apache.org>.
Brian Bockelman wrote:
> On May 14, 2010, at 8:27 PM, Todd Lipcon wrote:
> 
>> Hey Brian,
>>
>> Yep, excessive GC definitely sounds like a likely culprit. I'm surprised you
>> didn't see OOMEs in the log, though.
>>
> 
> We didn't until the third restart today.  I have no clue why we haven't seen this in the past 9 months of this cluster though...
> 
> Anyhow, it looks like this might have done the trick... the sysadmin is heading over to kick over a few errant datanodes, and we should be able to get out of safemode soon.  Luckily, it's a 4-day weekend in Europe and otherwise a Friday evening in the US, so there's only a few folks using it.

good thing we europeans have long weekends.

> 
>> If you want to monitor GC, I'd recommend adding -verbose:gc
>> -XX:+PrintGCDetails -XX:+PrintGCDateStamps to your java options -
>> occasionally useful for times like this.
>>

What are your current GC options? Played with compressed object pointers 
yet?

Re: NameNode deadlocked (help?)

Posted by Brian Bockelman <bb...@cse.unl.edu>.
On May 14, 2010, at 8:27 PM, Todd Lipcon wrote:

> Hey Brian,
> 
> Yep, excessive GC definitely sounds like a likely culprit. I'm surprised you
> didn't see OOMEs in the log, though.
> 

We didn't until the third restart today.  I have no clue why we haven't seen this in the past 9 months of this cluster though...

Anyhow, it looks like this might have done the trick... the sysadmin is heading over to kick over a few errant datanodes, and we should be able to get out of safemode soon.  Luckily, it's a 4-day weekend in Europe and otherwise a Friday evening in the US, so there's only a few folks using it.

> If you want to monitor GC, I'd recommend adding -verbose:gc
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps to your java options -
> occasionally useful for times like this.
> 
> Glad to hear you're back up and running.

Me too...

Brian

> 
> -Todd
> 
> On Fri, May 14, 2010 at 5:53 PM, Brian Bockelman <bb...@cse.unl.edu>wrote:
> 
>> Hey Todd,
>> 
>> The thread remains in that trace after a few attempts.
>> 
>> Being that no one was going to get anything done quickly, we decided to
>> restart the NN.  Upon restart, I found something *really* interesting - the
>> server had been running for quite awhile with 1GB of heap!
>> 
>> Being that these are the statistics for the cluster:
>> 
>> 720246 files and directories, 3096099 blocks = 3816345 total
>> 
>> I have no clue how this system ran before!  Hopefully this is really the
>> issue and not some red herring.
>> 
>> My theory is that we somehow managed to scrape below the memory limits ...
>> the same sysadmin has been complaining for a few months that the NN would
>> lock up for a few seconds at 100% CPU.  This could have been signs that I
>> (stupidly) missed of excessive garbage collection.
>> 
>> Sigh.  Need a beer.
>> 
>> Brian
>> 
>> On May 14, 2010, at 7:46 PM, Todd Lipcon wrote:
>> 
>>> Hey Brian,
>>> 
>>> I don't see a way in which this could infinite loop unless some
>>> datastructure got corrupted.
>>> 
>>> If you jstack the NN a few times, does that thread remain in that trace
>> or
>>> does it move around?
>>> 
>>> -Todd
>>> 
>>> On Fri, May 14, 2010 at 5:41 PM, Todd Lipcon <to...@cloudera.com> wrote:
>>> 
>>>> Hey Brian,
>>>> 
>>>> Looks like it's not deadlocked, but rather just busy doing a lot of
>> work:
>>>> 
>>>> 
>>>> 
>> "org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@1c778255
>> "
>>>> daemon prio=10 tid=0x00002aaafc012c00 nid=0x493 runnable
>>>> [0x00000000413da000..0x00000000413daa10]
>>>> java.lang.Thread.State: RUNNABLE
>>>>      at
>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeStoredBlock(FSNamesystem.java:3236)
>>>>      - locked <0x00002aaab3843e40> (a
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>>      at
>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeDatanode(FSNamesystem.java:2695)
>>>>      at
>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.heartbeatCheck(FSNamesystem.java:2785)
>>>>      - locked <0x00002aaab3659dd8> (a java.util.TreeMap)
>>>>      - locked <0x00002aaab3653848> (a java.util.ArrayList)
>>>>      - locked <0x00002aaab3843e40> (a
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>>      at
>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2312)
>>>>      at java.lang.Thread.run(Thread.java:619)
>>>> 
>>>> Let me dig up the 19.1 source code and see if this looks like an
>> infinite
>>>> loop or just one that's tying it up for some number of seconds.
>>>> 
>>>> Anything being written into the logs?
>>>> 
>>>> -Todd
>>>> 
>>>> 
>>>> On Fri, May 14, 2010 at 5:22 PM, Brian Bockelman <bbockelm@cse.unl.edu
>>> wrote:
>>>> 
>>>>> Hey guys,
>>>>> 
>>>>> I know it's 5PM on a Friday, but we just saw one of our big cluster's
>>>>> namenode's deadlock.
>>>>> 
>>>>> This is 0.19.1; does this ring a bell for anyone?  I haven't had any
>> time
>>>>> to start going through source code, but I figured I'd send out a SOS in
>> case
>>>>> if this looked familiar.
>>>>> 
>>>>> We had restarted this cluster a few hours ago and made the following
>>>>> changes:
>>>>> 
>>>>> 1) Increased the number of datanode handlers from 10 to 40.
>>>>> 2) Increased ipc.server.listen.queue.size from 128 to 256.
>>>>> 
>>>>> If nothing else, I figure a deadlocked NN might be interesting to
>> devs...
>>>>> 
>>>>> Brian
>>>>> 
>>>>> 2010-05-14 17:11:30
>>>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (11.2-b01 mixed
>> mode):
>>>>> 
>>>>> "IPC Server handler 39 on 9000" daemon prio=10 tid=0x00002aaafc181400
>>>>> nid=0x4cd waiting for monitor entry
>> [0x0000000045962000..0x0000000045962d90]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
>>>>>      - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
>>>>>      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>      at
>>>>> 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>      at
>>>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>>>      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>>>> 
>>>>> "IPC Server handler 38 on 9000" daemon prio=10 tid=0x00002aaafc17f800
>>>>> nid=0x4cc waiting for monitor entry
>> [0x0000000045861000..0x0000000045861d10]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getStats(FSNamesystem.java:3326)
>>>>>      - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getStats(NameNode.java:505)
>>>>>      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>      at
>>>>> 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>      at
>>>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>>>      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>>>> 
>>>>> "IPC Server handler 37 on 9000" daemon prio=10 tid=0x00002aaafc17e000
>>>>> nid=0x4cb waiting for monitor entry
>> [0x000000004575f000..0x0000000045760a90]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>>>      - waiting to lock <0x00002aaab3843e40> (a
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>>>      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>      at
>>>>> 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>      at
>>>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>>>      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>>>> 
>>>>> "IPC Server handler 36 on 9000" daemon prio=10 tid=0x00002aaafc17c400
>>>>> nid=0x4ca waiting for monitor entry
>> [0x000000004565e000..0x000000004565fa10]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:3281)
>>>>>      - waiting to lock <0x00002aaab3843e40> (a
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:649)
>>>>>      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>      at
>>>>> 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>      at
>>>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>>>      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>>>> 
>>>>> "IPC Server handler 35 on 9000" daemon prio=10 tid=0x00002aaafc17a800
>>>>> nid=0x4c9 waiting for monitor entry
>> [0x000000004555e000..0x000000004555eb90]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
>>>>>      - waiting to lock <0x00002aaab3843e40> (a
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
>>>>>      at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
>>>>>      at
>>>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>>>      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>>>> 
>>>>> "IPC Server handler 34 on 9000" daemon prio=10 tid=0x00002aaafc178c00
>>>>> nid=0x4c8 waiting for monitor entry
>> [0x000000004545d000..0x000000004545db10]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
>>>>>      - waiting to lock <0x00002aaab3843e40> (a
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
>>>>>      at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
>>>>>      at
>>>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>>>      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>>>> 
>>>>> "IPC Server handler 33 on 9000" daemon prio=10 tid=0x00002aaafc177400
>>>>> nid=0x4c7 waiting for monitor entry
>> [0x000000004535c000..0x000000004535cc90]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
>>>>>      - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
>>>>>      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>      at
>>>>> 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>      at
>>>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>>>      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>>>> 
>>>>> "IPC Server handler 32 on 9000" daemon prio=10 tid=0x00002aaafc175800
>>>>> nid=0x4c6 waiting for monitor entry
>> [0x000000004525a000..0x000000004525bc10]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>>>      - waiting to lock <0x00002aaab3843e40> (a
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>>>      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>      at
>>>>> 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>      at
>>>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>>>      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>>>> 
>>>>> "IPC Server handler 31 on 9000" daemon prio=10 tid=0x00002aaafc173c00
>>>>> nid=0x4c5 waiting for monitor entry
>> [0x000000004515a000..0x000000004515ad90]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:3281)
>>>>>      - waiting to lock <0x00002aaab3843e40> (a
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:649)
>>>>>      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>      at
>>>>> 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>      at
>>>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>>>      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>>>> 
>>>>> "IPC Server handler 30 on 9000" daemon prio=10 tid=0x00002aaafc172400
>>>>> nid=0x4c4 waiting for monitor entry
>> [0x0000000045059000..0x0000000045059d10]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
>>>>>      - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
>>>>>      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>      at
>>>>> 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>      at
>>>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>>>      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>>>> 
>>>>> "IPC Server handler 29 on 9000" daemon prio=10 tid=0x00002aaafc171000
>>>>> nid=0x4c3 waiting for monitor entry
>> [0x0000000044f57000..0x0000000044f58a90]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
>>>>>      - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
>>>>>      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>      at
>>>>> 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>      at
>>>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>>>      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>>>> 
>>>>> "IPC Server handler 28 on 9000" daemon prio=10 tid=0x00002aaafc16f400
>>>>> nid=0x4c2 waiting for monitor entry
>> [0x0000000044e56000..0x0000000044e57a10]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>>>      - waiting to lock <0x00002aaab3843e40> (a
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>>>      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>      at
>>>>> 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>      at
>>>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>>>      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>>>> 
>>>>> "IPC Server handler 27 on 9000" daemon prio=10 tid=0x00002aaafc16d800
>>>>> nid=0x4c1 waiting for monitor entry
>> [0x0000000044d56000..0x0000000044d56b90]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>>>      - waiting to lock <0x00002aaab3843e40> (a
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>>>      at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>>>>>      at
>>>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>>>      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>>>> 
>>>>> "IPC Server handler 26 on 9000" daemon prio=10 tid=0x00002aaafc16bc00
>>>>> nid=0x4c0 waiting for monitor entry
>> [0x0000000044c55000..0x0000000044c55b10]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>>>      - waiting to lock <0x00002aaab3843e40> (a
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>>>      at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>>>>>      at
>>>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>>>      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>>>> 
>>>>> "IPC Server handler 25 on 9000" daemon prio=10 tid=0x00002aaafc16a400
>>>>> nid=0x4bf waiting for monitor entry
>> [0x0000000044b54000..0x0000000044b54c90]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
>>>>>      - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
>>>>>      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>      at
>>>>> 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>      at
>>>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>>>      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>>>> 
>>>>> "IPC Server handler 24 on 9000" daemon prio=10 tid=0x00002aaafc168800
>>>>> nid=0x4be waiting for monitor entry
>> [0x0000000044a53000..0x0000000044a53c10]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
>>>>>      - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
>>>>>      at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
>>>>>      at
>>>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>>>      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>>>> 
>>>>> "IPC Server handler 23 on 9000" daemon prio=10 tid=0x00002aaafc166c00
>>>>> nid=0x4bd waiting for monitor entry
>> [0x0000000044952000..0x0000000044952d90]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>>>      - waiting to lock <0x00002aaab3843e40> (a
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>>>      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>      at
>>>>> 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>      at
>>>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>>>      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>>>> 
>>>>> "IPC Server handler 22 on 9000" daemon prio=10 tid=0x00002aaafc165400
>>>>> nid=0x4bc waiting for monitor entry
>> [0x0000000044851000..0x0000000044851d10]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>>>      - waiting to lock <0x00002aaab3843e40> (a
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>>>      at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>>>>>      at
>>>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>>>      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>>>> 
>>>>> "IPC Server handler 21 on 9000" daemon prio=10 tid=0x00002aaafc164000
>>>>> nid=0x4bb waiting for monitor entry
>> [0x0000000044750000..0x0000000044750a90]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>>>      - waiting to lock <0x00002aaab3843e40> (a
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>>>      at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>>>>>      at
>>>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>>>      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>>>> 
>>>>> "IPC Server handler 20 on 9000" daemon prio=10 tid=0x00002aaafc162400
>>>>> nid=0x4ba waiting for monitor entry
>> [0x000000004464f000..0x000000004464fa10]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
>>>>>      - waiting to lock <0x00002aaab3843e40> (a
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
>>>>>      at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
>>>>>      at
>>>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>>>      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>>>> 
>>>>> "IPC Server handler 19 on 9000" daemon prio=10 tid=0x00002aaafc488400
>>>>> nid=0x4b9 waiting for monitor entry
>> [0x000000004454e000..0x000000004454eb90]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
>>>>>      - waiting to lock <0x00002aaab3843e40> (a
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
>>>>>      at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
>>>>>      at
>>>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>>>      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>>>> 
>>>>> "IPC Server handler 18 on 9000" daemon prio=10 tid=0x00002aaafc486800
>>>>> nid=0x4b8 waiting for monitor entry
>> [0x000000004444d000..0x000000004444db10]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.nextGenerationStampForBlock(FSNamesystem.java:4548)
>>>>>      - waiting to lock <0x00002aaab3843e40> (a
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.NameNode.nextGenerationStamp(NameNode.java:402)
>>>>>      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>      at
>>>>> 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>      at
>>>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>>>      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>>>> 
>>>>> "IPC Server handler 17 on 9000" daemon prio=10 tid=0x00002aaafc484c00
>>>>> nid=0x4b7 waiting for monitor entry
>> [0x000000004434c000..0x000000004434cc90]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.nextGenerationStampForBlock(FSNamesystem.java:4548)
>>>>>      - waiting to lock <0x00002aaab3843e40> (a
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.NameNode.nextGenerationStamp(NameNode.java:402)
>>>>>      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>      at
>>>>> 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>      at
>>>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>>>      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>>>> 
>>>>> "IPC Server handler 16 on 9000" daemon prio=10 tid=0x00002aaafc483000
>>>>> nid=0x4b6 waiting for monitor entry
>> [0x000000004424a000..0x000000004424bc10]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>>>      - waiting to lock <0x00002aaab3843e40> (a
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>>>      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>      at
>>>>> 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>      at
>>>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>>>      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>>>> 
>>>>> "IPC Server handler 15 on 9000" daemon prio=10 tid=0x00002aaafc481800
>>>>> nid=0x4b5 waiting for monitor entry
>> [0x000000004414a000..0x000000004414ad90]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>>>      - waiting to lock <0x00002aaab3843e40> (a
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>>>      at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>>>>>      at
>>>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>>>      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>>>> 
>>>>> "IPC Server handler 14 on 9000" daemon prio=10 tid=0x00002aaafc47fc00
>>>>> nid=0x4b4 waiting for monitor entry
>> [0x0000000044049000..0x0000000044049d10]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
>>>>>      - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
>>>>>      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>      at
>>>>> 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>      at
>>>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>>>      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>>>> 
>>>>> "IPC Server handler 13 on 9000" daemon prio=10 tid=0x00002aaafc47e000
>>>>> nid=0x4b3 waiting for monitor entry
>> [0x0000000043f48000..0x0000000043f48a90]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
>>>>>      - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
>>>>>      at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
>>>>>      at
>>>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>>>      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>>>> 
>>>>> "IPC Server handler 12 on 9000" daemon prio=10 tid=0x00002aaafc47c400
>>>>> nid=0x4b2 waiting for monitor entry
>> [0x0000000043e46000..0x0000000043e47a10]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>>>      - waiting to lock <0x00002aaab3843e40> (a
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>>>      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>      at
>>>>> 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>      at
>>>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>>>      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>>>> 
>>>>> "IPC Server handler 11 on 9000" daemon prio=10 tid=0x00002aaafc47b400
>>>>> nid=0x4b1 waiting for monitor entry
>> [0x0000000043d45000..0x0000000043d46b90]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>>>      - waiting to lock <0x00002aaab3843e40> (a
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>>>      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>      at
>>>>> 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>      at
>>>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>>>      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>>>> 
>>>>> "IPC Server handler 10 on 9000" daemon prio=10 tid=0x00002aaafc479800
>>>>> nid=0x4b0 waiting for monitor entry
>> [0x0000000043c45000..0x0000000043c45b10]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
>>>>>      - waiting to lock <0x00002aaab3843e40> (a
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
>>>>>      at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
>>>>>      at
>>>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>>>      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>>>> 
>>>>> "IPC Server handler 9 on 9000" daemon prio=10 tid=0x00002aaafc477c00
>>>>> nid=0x4af waiting for monitor entry
>> [0x0000000043b43000..0x0000000043b44c90]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>>>      - waiting to lock <0x00002aaab3843e40> (a
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>>>      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>      at
>>>>> 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>      at
>>>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>>>      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>>>> 
>>>>> "IPC Server handler 8 on 9000" daemon prio=10 tid=0x00002aaafc476400
>>>>> nid=0x4ae waiting for monitor entry
>> [0x0000000043a42000..0x0000000043a43c10]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>>>      - waiting to lock <0x00002aaab3843e40> (a
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>>>      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>      at
>>>>> 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>      at
>>>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>>>      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>>>> 
>>>>> "IPC Server handler 7 on 9000" daemon prio=10 tid=0x00002aaafc474c00
>>>>> nid=0x4ad waiting for monitor entry
>> [0x0000000043942000..0x0000000043942d90]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>>>      - waiting to lock <0x00002aaab3843e40> (a
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>>>      at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>>>>>      at
>>>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>>>      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>>>> 
>>>>> "IPC Server handler 6 on 9000" daemon prio=10 tid=0x00002aaafc473000
>>>>> nid=0x4ac waiting for monitor entry
>> [0x0000000043841000..0x0000000043841d10]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>>>      - waiting to lock <0x00002aaab3843e40> (a
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>>>      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>      at
>>>>> 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>      at
>>>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>>>      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>>>> 
>>>>> "IPC Server handler 5 on 9000" daemon prio=10 tid=0x00002aaafc471400
>>>>> nid=0x4ab waiting for monitor entry
>> [0x0000000043740000..0x0000000043740a90]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>>>      - waiting to lock <0x00002aaab3843e40> (a
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>>>      at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>>>>>      at
>>>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>>>      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>>>> 
>>>>> "IPC Server handler 4 on 9000" daemon prio=10 tid=0x00002aaafc121800
>>>>> nid=0x4aa waiting for monitor entry
>> [0x000000004363f000..0x000000004363fa10]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>>>      - waiting to lock <0x00002aaab3843e40> (a
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>>>      at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>>>>>      at
>>>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>>>      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>>>> 
>>>>> "IPC Server handler 3 on 9000" daemon prio=10 tid=0x00002aaafc11fc00
>>>>> nid=0x4a9 waiting for monitor entry
>> [0x000000004353d000..0x000000004353eb90]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>>>      - waiting to lock <0x00002aaab3843e40> (a
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>>>      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>      at
>>>>> 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>      at
>>>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>>>      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>>>> 
>>>>> "IPC Server handler 2 on 9000" daemon prio=10 tid=0x00002aaafc11e400
>>>>> nid=0x4a8 waiting for monitor entry
>> [0x000000004343c000..0x000000004343db10]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>>>      - waiting to lock <0x00002aaab3843e40> (a
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>>>      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>      at
>>>>> 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>      at
>>>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>>>      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>>>> 
>>>>> "IPC Server handler 1 on 9000" daemon prio=10 tid=0x00002aaafc467800
>>>>> nid=0x4a7 waiting for monitor entry
>> [0x000000004333b000..0x000000004333cc90]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>>>      - waiting to lock <0x00002aaab3843e40> (a
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>>>      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>      at
>>>>> 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>      at
>>>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>>>      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>>>> 
>>>>> "IPC Server handler 0 on 9000" daemon prio=10 tid=0x00002aaafc466400
>>>>> nid=0x4a6 waiting for monitor entry
>> [0x000000004323b000..0x000000004323bc10]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>>>      - waiting to lock <0x00002aaab3843e40> (a
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>>>      at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>>>>>      at
>>>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>>>      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>>>> 
>>>>> "IPC Server listener on 9000" daemon prio=10 tid=0x00002aaafc12fc00
>>>>> nid=0x4a5 waiting on condition [0x000000004313a000..0x000000004313ad90]
>>>>> java.lang.Thread.State: WAITING (parking)
>>>>>      at sun.misc.Unsafe.park(Native Method)
>>>>>      - parking to wait for  <0x00002aaab3b8fd70> (a
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>>      at
>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>>>      at
>>>>> 
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
>>>>>      at
>>>>> 
>> java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:254)
>>>>>      at
>>>>> org.apache.hadoop.ipc.Server$Connection.processData(Server.java:850)
>>>>>      at
>>>>> org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:814)
>>>>>      at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:400)
>>>>>      at org.apache.hadoop.ipc.Server$Listener.run(Server.java:309)
>>>>> 
>>>>> "IPC Server Responder" daemon prio=10 tid=0x00002aaafc0bbc00 nid=0x4a4
>>>>> runnable [0x0000000043039000..0x0000000043039d10]
>>>>> java.lang.Thread.State: RUNNABLE
>>>>>      at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>      at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>>>>      at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>      at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>      - locked <0x00002aaab3b90e80> (a sun.nio.ch.Util$1)
>>>>>      - locked <0x00002aaab3b90e68> (a
>>>>> java.util.Collections$UnmodifiableSet)
>>>>>      - locked <0x00002aaab3b90b08> (a sun.nio.ch.EPollSelectorImpl)
>>>>>      at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>      at org.apache.hadoop.ipc.Server$Responder.run(Server.java:458)
>>>>> 
>>>>> "SocketListener0-1" prio=10 tid=0x00002aaafc0ba400 nid=0x4a3 in
>>>>> Object.wait() [0x0000000042f38000..0x0000000042f38a90]
>>>>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>>>>      at java.lang.Object.wait(Native Method)
>>>>>      - waiting on <0x00002aaadf36cb00> (a
>>>>> org.mortbay.util.ThreadPool$PoolThread)
>>>>>      at
>> org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:522)
>>>>>      - locked <0x00002aaadf36cb00> (a
>>>>> org.mortbay.util.ThreadPool$PoolThread)
>>>>> 
>>>>> "SocketListener0-0" prio=10 tid=0x00002aaafc0b4c00 nid=0x4a2 in
>>>>> Object.wait() [0x0000000042e37000..0x0000000042e37a10]
>>>>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>>>>      at java.lang.Object.wait(Native Method)
>>>>>      - waiting on <0x00002aaadf36c820> (a
>>>>> org.mortbay.util.ThreadPool$PoolThread)
>>>>>      at
>> org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:522)
>>>>>      - locked <0x00002aaadf36c820> (a
>>>>> org.mortbay.util.ThreadPool$PoolThread)
>>>>> 
>>>>> "Acceptor ServerSocket[addr=proxy-1/169.228.130.63
>> ,port=0,localport=50070]"
>>>>> prio=10 tid=0x00002aaafc0c8400 nid=0x4a1 runnable
>>>>> [0x0000000042d36000..0x0000000042d36b90]
>>>>> java.lang.Thread.State: RUNNABLE
>>>>>      at java.lang.Object.<init>(Object.java:20)
>>>>>      at java.net.SocketImpl.<init>(SocketImpl.java:27)
>>>>>      at java.net.PlainSocketImpl.<init>(PlainSocketImpl.java:90)
>>>>>      at java.net.SocksSocketImpl.<init>(SocksSocketImpl.java:33)
>>>>>      at java.net.Socket.setImpl(Socket.java:434)
>>>>>      at java.net.ServerSocket.implAccept(ServerSocket.java:445)
>>>>>      at java.net.ServerSocket.accept(ServerSocket.java:421)
>>>>>      at
>>>>> org.mortbay.util.ThreadedServer.acceptSocket(ThreadedServer.java:432)
>>>>>      at
>>>>> org.mortbay.util.ThreadedServer$Acceptor.run(ThreadedServer.java:631)
>>>>> 
>>>>> "SessionScavenger" daemon prio=10 tid=0x00002aaafc0c1000 nid=0x4a0
>> waiting
>>>>> on condition [0x0000000042c35000..0x0000000042c35b10]
>>>>> java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>>      at java.lang.Thread.sleep(Native Method)
>>>>>      at
>>>>> 
>> org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(AbstractSessionManager.java:587)
>>>>> 
>>>>> "SessionScavenger" daemon prio=10 tid=0x00002aaafc0fa000 nid=0x49f
>> waiting
>>>>> on condition [0x0000000040b2a000..0x0000000040b2ac90]
>>>>> java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>>      at java.lang.Thread.sleep(Native Method)
>>>>>      at
>>>>> 
>> org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(AbstractSessionManager.java:587)
>>>>> 
>>>>> "SessionScavenger" daemon prio=10 tid=0x00002aaafc139800 nid=0x49a
>> waiting
>>>>> on condition [0x00000000408b7000..0x00000000408b7c10]
>>>>> java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>>      at java.lang.Thread.sleep(Native Method)
>>>>>      at
>>>>> 
>> org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(AbstractSessionManager.java:587)
>>>>> 
>>>>> 
>>>>> 
>> "org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@1b0deb5f
>> "
>>>>> daemon prio=10 tid=0x00002aaafc0a2800 nid=0x496 waiting for monitor
>> entry
>>>>> [0x0000000042b34000..0x0000000042b34d90]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:61)
>>>>>      - waiting to lock <0x00002aaab3843e40> (a
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>>>      at java.lang.Thread.run(Thread.java:619)
>>>>> 
>>>>> 
>>>>> 
>> "org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@784a7df6
>> "
>>>>> daemon prio=10 tid=0x00002aaafc090c00 nid=0x495 waiting for monitor
>> entry
>>>>> [0x00000000411f8000..0x00000000411f8d10]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.computeReplicationWork(FSNamesystem.java:2410)
>>>>>      - waiting to lock <0x00002aaab3843e40> (a
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.computeDatanodeWork(FSNamesystem.java:2376)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2333)
>>>>>      at java.lang.Thread.run(Thread.java:619)
>>>>> 
>>>>> "org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@d24111a"
>>>>> daemon prio=10 tid=0x00002aaafc013400 nid=0x494 waiting for monitor
>> entry
>>>>> [0x00000000410f7000..0x00000000410f7a90]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:349)
>>>>>      - waiting to lock <0x00002aaab3843e40> (a
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>>>      at java.lang.Thread.run(Thread.java:619)
>>>>> 
>>>>> 
>>>>> 
>> "org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@1c778255
>> "
>>>>> daemon prio=10 tid=0x00002aaafc012c00 nid=0x493 runnable
>>>>> [0x00000000413da000..0x00000000413daa10]
>>>>> java.lang.Thread.State: RUNNABLE
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeStoredBlock(FSNamesystem.java:3236)
>>>>>      - locked <0x00002aaab3843e40> (a
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeDatanode(FSNamesystem.java:2695)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.heartbeatCheck(FSNamesystem.java:2785)
>>>>>      - locked <0x00002aaab3659dd8> (a java.util.TreeMap)
>>>>>      - locked <0x00002aaab3653848> (a java.util.ArrayList)
>>>>>      - locked <0x00002aaab3843e40> (a
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2312)
>>>>>      at java.lang.Thread.run(Thread.java:619)
>>>>> 
>>>>> 
>>>>> 
>> "org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@44676e3f
>> "
>>>>> daemon prio=10 tid=0x00002aaafc012800 nid=0x492 waiting on condition
>>>>> [0x0000000040d16000..0x0000000040d16b90]
>>>>> java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>>>      at java.lang.Thread.sleep(Native Method)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
>>>>>      at java.lang.Thread.run(Thread.java:619)
>>>>> 
>>>>> "Timer thread for monitoring dfs" daemon prio=10 tid=0x00002aaafc03d800
>>>>> nid=0x428 waiting for monitor entry
>> [0x0000000040e7e000..0x0000000040e7ed10]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getStats(FSNamesystem.java:3326)
>>>>>      - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getCapacityTotal(FSNamesystem.java:3335)
>>>>>      at
>>>>> 
>> org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics.doUpdates(FSNamesystemMetrics.java:90)
>>>>>      - locked <0x00002aaab3b3c580> (a
>>>>> org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics)
>>>>>      at
>>>>> 
>> org.apache.hadoop.metrics.spi.AbstractMetricsContext.timerEvent(AbstractMetricsContext.java:284)
>>>>>      at
>>>>> 
>> org.apache.hadoop.metrics.spi.AbstractMetricsContext.access$000(AbstractMetricsContext.java:50)
>>>>>      at
>>>>> 
>> org.apache.hadoop.metrics.spi.AbstractMetricsContext$1.run(AbstractMetricsContext.java:249)
>>>>>      at java.util.TimerThread.mainLoop(Timer.java:512)
>>>>>      at java.util.TimerThread.run(Timer.java:462)
>>>>> 
>>>>> "Low Memory Detector" daemon prio=10 tid=0x000000005c306c00 nid=0x426
>>>>> runnable [0x0000000000000000..0x0000000000000000]
>>>>> java.lang.Thread.State: RUNNABLE
>>>>> 
>>>>> "CompilerThread1" daemon prio=10 tid=0x000000005c304000 nid=0x425
>> waiting
>>>>> on condition [0x0000000000000000..0x0000000042830420]
>>>>> java.lang.Thread.State: RUNNABLE
>>>>> 
>>>>> "CompilerThread0" daemon prio=10 tid=0x000000005c300000 nid=0x424
>> waiting
>>>>> on condition [0x0000000000000000..0x000000004272f3a0]
>>>>> java.lang.Thread.State: RUNNABLE
>>>>> 
>>>>> "Signal Dispatcher" daemon prio=10 tid=0x000000005c2fe400 nid=0x423
>>>>> waiting on condition [0x0000000000000000..0x0000000000000000]
>>>>> java.lang.Thread.State: RUNNABLE
>>>>> 
>>>>> "Finalizer" daemon prio=10 tid=0x000000005c2da800 nid=0x422 in
>>>>> Object.wait() [0x000000004252e000..0x000000004252ec10]
>>>>> java.lang.Thread.State: WAITING (on object monitor)
>>>>>      at java.lang.Object.wait(Native Method)
>>>>>      - waiting on <0x00002aaab38ec320> (a
>>>>> java.lang.ref.ReferenceQueue$Lock)
>>>>>      at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
>>>>>      - locked <0x00002aaab38ec320> (a
>> java.lang.ref.ReferenceQueue$Lock)
>>>>>      at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
>>>>>      at
>> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>>>>> 
>>>>> "Reference Handler" daemon prio=10 tid=0x000000005c2d8c00 nid=0x421 in
>>>>> Object.wait() [0x000000004177e000..0x000000004177ed90]
>>>>> java.lang.Thread.State: WAITING (on object monitor)
>>>>>      at java.lang.Object.wait(Native Method)
>>>>>      - waiting on <0x00002aaab38e9ec0> (a
>> java.lang.ref.Reference$Lock)
>>>>>      at java.lang.Object.wait(Object.java:485)
>>>>>      at
>> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>>>>>      - locked <0x00002aaab38e9ec0> (a java.lang.ref.Reference$Lock)
>>>>> 
>>>>> "main" prio=10 tid=0x000000005c268400 nid=0x417 in Object.wait()
>>>>> [0x0000000040494000..0x0000000040494ed0]
>>>>> java.lang.Thread.State: WAITING (on object monitor)
>>>>>      at java.lang.Object.wait(Native Method)
>>>>>      - waiting on <0x00002aaab3b3db50> (a
>>>>> org.apache.hadoop.ipc.RPC$Server)
>>>>>      at java.lang.Object.wait(Object.java:485)
>>>>>      at org.apache.hadoop.ipc.Server.join(Server.java:1023)
>>>>>      - locked <0x00002aaab3b3db50> (a
>> org.apache.hadoop.ipc.RPC$Server)
>>>>>      at
>>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.join(NameNode.java:221)
>>>>>      at
>>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:870)
>>>>> 
>>>>> "VM Thread" prio=10 tid=0x000000005c2d3400 nid=0x420 runnable
>>>>> 
>>>>> "GC task thread#0 (ParallelGC)" prio=10 tid=0x000000005c273000
>> nid=0x418
>>>>> runnable
>>>>> 
>>>>> "GC task thread#1 (ParallelGC)" prio=10 tid=0x000000005c274800
>> nid=0x419
>>>>> runnable
>>>>> 
>>>>> "GC task thread#2 (ParallelGC)" prio=10 tid=0x000000005c276400
>> nid=0x41a
>>>>> runnable
>>>>> 
>>>>> "GC task thread#3 (ParallelGC)" prio=10 tid=0x000000005c277c00
>> nid=0x41b
>>>>> runnable
>>>>> 
>>>>> "GC task thread#4 (ParallelGC)" prio=10 tid=0x000000005c279400
>> nid=0x41c
>>>>> runnable
>>>>> 
>>>>> "GC task thread#5 (ParallelGC)" prio=10 tid=0x000000005c27b000
>> nid=0x41d
>>>>> runnable
>>>>> 
>>>>> "GC task thread#6 (ParallelGC)" prio=10 tid=0x000000005c27c800
>> nid=0x41e
>>>>> runnable
>>>>> 
>>>>> "GC task thread#7 (ParallelGC)" prio=10 tid=0x000000005c27e000
>> nid=0x41f
>>>>> runnable
>>>>> 
>>>>> "VM Periodic Task Thread" prio=10 tid=0x000000005c308c00 nid=0x427
>> waiting
>>>>> on condition
>>>>> 
>>>>> JNI global references: 984
>>>>> 
>>>>> Heap
>>>>> PSYoungGen      total 203712K, used 107631K [0x00002aaadd0b0000,
>>>>> 0x00002aaaf1e00000, 0x00002aaaf1e00000)
>>>>> eden space 124096K, 86% used
>>>>> [0x00002aaadd0b0000,0x00002aaae39cbe58,0x00002aaae49e0000)
>>>>> from space 79616K, 0% used
>>>>> [0x00002aaae49e0000,0x00002aaae49e0000,0x00002aaae97a0000)
>>>>> to   space 113728K, 0% used
>>>>> [0x00002aaaeaef0000,0x00002aaaeaef0000,0x00002aaaf1e00000)
>>>>> PSOldGen        total 682688K, used 682687K [0x00002aaab3600000,
>>>>> 0x00002aaadd0b0000, 0x00002aaadd0b0000)
>>>>> object space 682688K, 99% used
>>>>> [0x00002aaab3600000,0x00002aaadd0affe8,0x00002aaadd0b0000)
>>>>> PSPermGen       total 21248K, used 16689K [0x00002aaaae200000,
>>>>> 0x00002aaaaf6c0000, 0x00002aaab3600000)
>>>>> object space 21248K, 78% used
>>>>> [0x00002aaaae200000,0x00002aaaaf24c660,0x00002aaaaf6c0000)
>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> --
>>>> Todd Lipcon
>>>> Software Engineer, Cloudera
>>>> 
>>> 
>>> 
>>> 
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>> 
>> 
> 
> 
> -- 
> Todd Lipcon
> Software Engineer, Cloudera

Re: NameNode deadlocked (help?)

Posted by Todd Lipcon <to...@cloudera.com>.
Hey Brian,

Yep, excessive GC definitely sounds like a likely culprit. I'm surprised you
didn't see OOMEs in the log, though.

If you want to monitor GC, I'd recommend adding -verbose:gc
-XX:+PrintGCDetails -XX:+PrintGCDateStamps to your java options -
occasionally useful for times like this.

Glad to hear you're back up and running.

-Todd

On Fri, May 14, 2010 at 5:53 PM, Brian Bockelman <bb...@cse.unl.edu>wrote:

> Hey Todd,
>
> The thread remains in that trace after a few attempts.
>
> Being that no one was going to get anything done quickly, we decided to
> restart the NN.  Upon restart, I found something *really* interesting - the
> server had been running for quite awhile with 1GB of heap!
>
> Being that these are the statistics for the cluster:
>
> 720246 files and directories, 3096099 blocks = 3816345 total
>
> I have no clue how this system ran before!  Hopefully this is really the
> issue and not some red herring.
>
> My theory is that we somehow managed to scrape below the memory limits ...
> the same sysadmin has been complaining for a few months that the NN would
> lock up for a few seconds at 100% CPU.  This could have been signs that I
> (stupidly) missed of excessive garbage collection.
>
> Sigh.  Need a beer.
>
> Brian
>
> On May 14, 2010, at 7:46 PM, Todd Lipcon wrote:
>
> > Hey Brian,
> >
> > I don't see a way in which this could infinite loop unless some
> > datastructure got corrupted.
> >
> > If you jstack the NN a few times, does that thread remain in that trace
> or
> > does it move around?
> >
> > -Todd
> >
> > On Fri, May 14, 2010 at 5:41 PM, Todd Lipcon <to...@cloudera.com> wrote:
> >
> >> Hey Brian,
> >>
> >> Looks like it's not deadlocked, but rather just busy doing a lot of
> work:
> >>
> >>
> >>
> "org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@1c778255
> "
> >> daemon prio=10 tid=0x00002aaafc012c00 nid=0x493 runnable
> >> [0x00000000413da000..0x00000000413daa10]
> >>  java.lang.Thread.State: RUNNABLE
> >>       at
> >>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeStoredBlock(FSNamesystem.java:3236)
> >>       - locked <0x00002aaab3843e40> (a
> >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>       at
> >>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeDatanode(FSNamesystem.java:2695)
> >>       at
> >>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.heartbeatCheck(FSNamesystem.java:2785)
> >>       - locked <0x00002aaab3659dd8> (a java.util.TreeMap)
> >>       - locked <0x00002aaab3653848> (a java.util.ArrayList)
> >>       - locked <0x00002aaab3843e40> (a
> >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>       at
> >>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2312)
> >>       at java.lang.Thread.run(Thread.java:619)
> >>
> >> Let me dig up the 19.1 source code and see if this looks like an
> infinite
> >> loop or just one that's tying it up for some number of seconds.
> >>
> >> Anything being written into the logs?
> >>
> >> -Todd
> >>
> >>
> >> On Fri, May 14, 2010 at 5:22 PM, Brian Bockelman <bbockelm@cse.unl.edu
> >wrote:
> >>
> >>> Hey guys,
> >>>
> >>> I know it's 5PM on a Friday, but we just saw one of our big cluster's
> >>> namenode's deadlock.
> >>>
> >>> This is 0.19.1; does this ring a bell for anyone?  I haven't had any
> time
> >>> to start going through source code, but I figured I'd send out a SOS in
> case
> >>> if this looked familiar.
> >>>
> >>> We had restarted this cluster a few hours ago and made the following
> >>> changes:
> >>>
> >>> 1) Increased the number of datanode handlers from 10 to 40.
> >>> 2) Increased ipc.server.listen.queue.size from 128 to 256.
> >>>
> >>> If nothing else, I figure a deadlocked NN might be interesting to
> devs...
> >>>
> >>> Brian
> >>>
> >>> 2010-05-14 17:11:30
> >>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (11.2-b01 mixed
> mode):
> >>>
> >>> "IPC Server handler 39 on 9000" daemon prio=10 tid=0x00002aaafc181400
> >>> nid=0x4cd waiting for monitor entry
> [0x0000000045962000..0x0000000045962d90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
> >>>       - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 38 on 9000" daemon prio=10 tid=0x00002aaafc17f800
> >>> nid=0x4cc waiting for monitor entry
> [0x0000000045861000..0x0000000045861d10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getStats(FSNamesystem.java:3326)
> >>>       - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getStats(NameNode.java:505)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 37 on 9000" daemon prio=10 tid=0x00002aaafc17e000
> >>> nid=0x4cb waiting for monitor entry
> [0x000000004575f000..0x0000000045760a90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 36 on 9000" daemon prio=10 tid=0x00002aaafc17c400
> >>> nid=0x4ca waiting for monitor entry
> [0x000000004565e000..0x000000004565fa10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:3281)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:649)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 35 on 9000" daemon prio=10 tid=0x00002aaafc17a800
> >>> nid=0x4c9 waiting for monitor entry
> [0x000000004555e000..0x000000004555eb90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
> >>>       at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 34 on 9000" daemon prio=10 tid=0x00002aaafc178c00
> >>> nid=0x4c8 waiting for monitor entry
> [0x000000004545d000..0x000000004545db10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
> >>>       at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 33 on 9000" daemon prio=10 tid=0x00002aaafc177400
> >>> nid=0x4c7 waiting for monitor entry
> [0x000000004535c000..0x000000004535cc90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
> >>>       - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 32 on 9000" daemon prio=10 tid=0x00002aaafc175800
> >>> nid=0x4c6 waiting for monitor entry
> [0x000000004525a000..0x000000004525bc10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 31 on 9000" daemon prio=10 tid=0x00002aaafc173c00
> >>> nid=0x4c5 waiting for monitor entry
> [0x000000004515a000..0x000000004515ad90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:3281)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:649)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 30 on 9000" daemon prio=10 tid=0x00002aaafc172400
> >>> nid=0x4c4 waiting for monitor entry
> [0x0000000045059000..0x0000000045059d10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
> >>>       - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 29 on 9000" daemon prio=10 tid=0x00002aaafc171000
> >>> nid=0x4c3 waiting for monitor entry
> [0x0000000044f57000..0x0000000044f58a90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
> >>>       - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 28 on 9000" daemon prio=10 tid=0x00002aaafc16f400
> >>> nid=0x4c2 waiting for monitor entry
> [0x0000000044e56000..0x0000000044e57a10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 27 on 9000" daemon prio=10 tid=0x00002aaafc16d800
> >>> nid=0x4c1 waiting for monitor entry
> [0x0000000044d56000..0x0000000044d56b90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 26 on 9000" daemon prio=10 tid=0x00002aaafc16bc00
> >>> nid=0x4c0 waiting for monitor entry
> [0x0000000044c55000..0x0000000044c55b10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 25 on 9000" daemon prio=10 tid=0x00002aaafc16a400
> >>> nid=0x4bf waiting for monitor entry
> [0x0000000044b54000..0x0000000044b54c90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
> >>>       - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 24 on 9000" daemon prio=10 tid=0x00002aaafc168800
> >>> nid=0x4be waiting for monitor entry
> [0x0000000044a53000..0x0000000044a53c10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
> >>>       - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
> >>>       at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 23 on 9000" daemon prio=10 tid=0x00002aaafc166c00
> >>> nid=0x4bd waiting for monitor entry
> [0x0000000044952000..0x0000000044952d90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 22 on 9000" daemon prio=10 tid=0x00002aaafc165400
> >>> nid=0x4bc waiting for monitor entry
> [0x0000000044851000..0x0000000044851d10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 21 on 9000" daemon prio=10 tid=0x00002aaafc164000
> >>> nid=0x4bb waiting for monitor entry
> [0x0000000044750000..0x0000000044750a90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 20 on 9000" daemon prio=10 tid=0x00002aaafc162400
> >>> nid=0x4ba waiting for monitor entry
> [0x000000004464f000..0x000000004464fa10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
> >>>       at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 19 on 9000" daemon prio=10 tid=0x00002aaafc488400
> >>> nid=0x4b9 waiting for monitor entry
> [0x000000004454e000..0x000000004454eb90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
> >>>       at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 18 on 9000" daemon prio=10 tid=0x00002aaafc486800
> >>> nid=0x4b8 waiting for monitor entry
> [0x000000004444d000..0x000000004444db10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.nextGenerationStampForBlock(FSNamesystem.java:4548)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.nextGenerationStamp(NameNode.java:402)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 17 on 9000" daemon prio=10 tid=0x00002aaafc484c00
> >>> nid=0x4b7 waiting for monitor entry
> [0x000000004434c000..0x000000004434cc90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.nextGenerationStampForBlock(FSNamesystem.java:4548)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.nextGenerationStamp(NameNode.java:402)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 16 on 9000" daemon prio=10 tid=0x00002aaafc483000
> >>> nid=0x4b6 waiting for monitor entry
> [0x000000004424a000..0x000000004424bc10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 15 on 9000" daemon prio=10 tid=0x00002aaafc481800
> >>> nid=0x4b5 waiting for monitor entry
> [0x000000004414a000..0x000000004414ad90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 14 on 9000" daemon prio=10 tid=0x00002aaafc47fc00
> >>> nid=0x4b4 waiting for monitor entry
> [0x0000000044049000..0x0000000044049d10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
> >>>       - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 13 on 9000" daemon prio=10 tid=0x00002aaafc47e000
> >>> nid=0x4b3 waiting for monitor entry
> [0x0000000043f48000..0x0000000043f48a90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
> >>>       - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
> >>>       at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 12 on 9000" daemon prio=10 tid=0x00002aaafc47c400
> >>> nid=0x4b2 waiting for monitor entry
> [0x0000000043e46000..0x0000000043e47a10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 11 on 9000" daemon prio=10 tid=0x00002aaafc47b400
> >>> nid=0x4b1 waiting for monitor entry
> [0x0000000043d45000..0x0000000043d46b90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 10 on 9000" daemon prio=10 tid=0x00002aaafc479800
> >>> nid=0x4b0 waiting for monitor entry
> [0x0000000043c45000..0x0000000043c45b10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
> >>>       at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 9 on 9000" daemon prio=10 tid=0x00002aaafc477c00
> >>> nid=0x4af waiting for monitor entry
> [0x0000000043b43000..0x0000000043b44c90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 8 on 9000" daemon prio=10 tid=0x00002aaafc476400
> >>> nid=0x4ae waiting for monitor entry
> [0x0000000043a42000..0x0000000043a43c10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 7 on 9000" daemon prio=10 tid=0x00002aaafc474c00
> >>> nid=0x4ad waiting for monitor entry
> [0x0000000043942000..0x0000000043942d90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 6 on 9000" daemon prio=10 tid=0x00002aaafc473000
> >>> nid=0x4ac waiting for monitor entry
> [0x0000000043841000..0x0000000043841d10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 5 on 9000" daemon prio=10 tid=0x00002aaafc471400
> >>> nid=0x4ab waiting for monitor entry
> [0x0000000043740000..0x0000000043740a90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 4 on 9000" daemon prio=10 tid=0x00002aaafc121800
> >>> nid=0x4aa waiting for monitor entry
> [0x000000004363f000..0x000000004363fa10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 3 on 9000" daemon prio=10 tid=0x00002aaafc11fc00
> >>> nid=0x4a9 waiting for monitor entry
> [0x000000004353d000..0x000000004353eb90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 2 on 9000" daemon prio=10 tid=0x00002aaafc11e400
> >>> nid=0x4a8 waiting for monitor entry
> [0x000000004343c000..0x000000004343db10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 1 on 9000" daemon prio=10 tid=0x00002aaafc467800
> >>> nid=0x4a7 waiting for monitor entry
> [0x000000004333b000..0x000000004333cc90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 0 on 9000" daemon prio=10 tid=0x00002aaafc466400
> >>> nid=0x4a6 waiting for monitor entry
> [0x000000004323b000..0x000000004323bc10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server listener on 9000" daemon prio=10 tid=0x00002aaafc12fc00
> >>> nid=0x4a5 waiting on condition [0x000000004313a000..0x000000004313ad90]
> >>>  java.lang.Thread.State: WAITING (parking)
> >>>       at sun.misc.Unsafe.park(Native Method)
> >>>       - parking to wait for  <0x00002aaab3b8fd70> (a
> >>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>       at
> >>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> >>>       at
> >>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> >>>       at
> >>>
> java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:254)
> >>>       at
> >>> org.apache.hadoop.ipc.Server$Connection.processData(Server.java:850)
> >>>       at
> >>> org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:814)
> >>>       at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:400)
> >>>       at org.apache.hadoop.ipc.Server$Listener.run(Server.java:309)
> >>>
> >>> "IPC Server Responder" daemon prio=10 tid=0x00002aaafc0bbc00 nid=0x4a4
> >>> runnable [0x0000000043039000..0x0000000043039d10]
> >>>  java.lang.Thread.State: RUNNABLE
> >>>       at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>       at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> >>>       at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> >>>       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> >>>       - locked <0x00002aaab3b90e80> (a sun.nio.ch.Util$1)
> >>>       - locked <0x00002aaab3b90e68> (a
> >>> java.util.Collections$UnmodifiableSet)
> >>>       - locked <0x00002aaab3b90b08> (a sun.nio.ch.EPollSelectorImpl)
> >>>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> >>>       at org.apache.hadoop.ipc.Server$Responder.run(Server.java:458)
> >>>
> >>> "SocketListener0-1" prio=10 tid=0x00002aaafc0ba400 nid=0x4a3 in
> >>> Object.wait() [0x0000000042f38000..0x0000000042f38a90]
> >>>  java.lang.Thread.State: TIMED_WAITING (on object monitor)
> >>>       at java.lang.Object.wait(Native Method)
> >>>       - waiting on <0x00002aaadf36cb00> (a
> >>> org.mortbay.util.ThreadPool$PoolThread)
> >>>       at
> org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:522)
> >>>       - locked <0x00002aaadf36cb00> (a
> >>> org.mortbay.util.ThreadPool$PoolThread)
> >>>
> >>> "SocketListener0-0" prio=10 tid=0x00002aaafc0b4c00 nid=0x4a2 in
> >>> Object.wait() [0x0000000042e37000..0x0000000042e37a10]
> >>>  java.lang.Thread.State: TIMED_WAITING (on object monitor)
> >>>       at java.lang.Object.wait(Native Method)
> >>>       - waiting on <0x00002aaadf36c820> (a
> >>> org.mortbay.util.ThreadPool$PoolThread)
> >>>       at
> org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:522)
> >>>       - locked <0x00002aaadf36c820> (a
> >>> org.mortbay.util.ThreadPool$PoolThread)
> >>>
> >>> "Acceptor ServerSocket[addr=proxy-1/169.228.130.63
> ,port=0,localport=50070]"
> >>> prio=10 tid=0x00002aaafc0c8400 nid=0x4a1 runnable
> >>> [0x0000000042d36000..0x0000000042d36b90]
> >>>  java.lang.Thread.State: RUNNABLE
> >>>       at java.lang.Object.<init>(Object.java:20)
> >>>       at java.net.SocketImpl.<init>(SocketImpl.java:27)
> >>>       at java.net.PlainSocketImpl.<init>(PlainSocketImpl.java:90)
> >>>       at java.net.SocksSocketImpl.<init>(SocksSocketImpl.java:33)
> >>>       at java.net.Socket.setImpl(Socket.java:434)
> >>>       at java.net.ServerSocket.implAccept(ServerSocket.java:445)
> >>>       at java.net.ServerSocket.accept(ServerSocket.java:421)
> >>>       at
> >>> org.mortbay.util.ThreadedServer.acceptSocket(ThreadedServer.java:432)
> >>>       at
> >>> org.mortbay.util.ThreadedServer$Acceptor.run(ThreadedServer.java:631)
> >>>
> >>> "SessionScavenger" daemon prio=10 tid=0x00002aaafc0c1000 nid=0x4a0
> waiting
> >>> on condition [0x0000000042c35000..0x0000000042c35b10]
> >>>  java.lang.Thread.State: TIMED_WAITING (sleeping)
> >>>       at java.lang.Thread.sleep(Native Method)
> >>>       at
> >>>
> org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(AbstractSessionManager.java:587)
> >>>
> >>> "SessionScavenger" daemon prio=10 tid=0x00002aaafc0fa000 nid=0x49f
> waiting
> >>> on condition [0x0000000040b2a000..0x0000000040b2ac90]
> >>>  java.lang.Thread.State: TIMED_WAITING (sleeping)
> >>>       at java.lang.Thread.sleep(Native Method)
> >>>       at
> >>>
> org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(AbstractSessionManager.java:587)
> >>>
> >>> "SessionScavenger" daemon prio=10 tid=0x00002aaafc139800 nid=0x49a
> waiting
> >>> on condition [0x00000000408b7000..0x00000000408b7c10]
> >>>  java.lang.Thread.State: TIMED_WAITING (sleeping)
> >>>       at java.lang.Thread.sleep(Native Method)
> >>>       at
> >>>
> org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(AbstractSessionManager.java:587)
> >>>
> >>>
> >>>
> "org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@1b0deb5f
> "
> >>> daemon prio=10 tid=0x00002aaafc0a2800 nid=0x496 waiting for monitor
> entry
> >>> [0x0000000042b34000..0x0000000042b34d90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:61)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at java.lang.Thread.run(Thread.java:619)
> >>>
> >>>
> >>>
> "org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@784a7df6
> "
> >>> daemon prio=10 tid=0x00002aaafc090c00 nid=0x495 waiting for monitor
> entry
> >>> [0x00000000411f8000..0x00000000411f8d10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.computeReplicationWork(FSNamesystem.java:2410)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.computeDatanodeWork(FSNamesystem.java:2376)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2333)
> >>>       at java.lang.Thread.run(Thread.java:619)
> >>>
> >>> "org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@d24111a"
> >>> daemon prio=10 tid=0x00002aaafc013400 nid=0x494 waiting for monitor
> entry
> >>> [0x00000000410f7000..0x00000000410f7a90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:349)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at java.lang.Thread.run(Thread.java:619)
> >>>
> >>>
> >>>
> "org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@1c778255
> "
> >>> daemon prio=10 tid=0x00002aaafc012c00 nid=0x493 runnable
> >>> [0x00000000413da000..0x00000000413daa10]
> >>>  java.lang.Thread.State: RUNNABLE
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeStoredBlock(FSNamesystem.java:3236)
> >>>       - locked <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeDatanode(FSNamesystem.java:2695)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.heartbeatCheck(FSNamesystem.java:2785)
> >>>       - locked <0x00002aaab3659dd8> (a java.util.TreeMap)
> >>>       - locked <0x00002aaab3653848> (a java.util.ArrayList)
> >>>       - locked <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2312)
> >>>       at java.lang.Thread.run(Thread.java:619)
> >>>
> >>>
> >>>
> "org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@44676e3f
> "
> >>> daemon prio=10 tid=0x00002aaafc012800 nid=0x492 waiting on condition
> >>> [0x0000000040d16000..0x0000000040d16b90]
> >>>  java.lang.Thread.State: TIMED_WAITING (sleeping)
> >>>       at java.lang.Thread.sleep(Native Method)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
> >>>       at java.lang.Thread.run(Thread.java:619)
> >>>
> >>> "Timer thread for monitoring dfs" daemon prio=10 tid=0x00002aaafc03d800
> >>> nid=0x428 waiting for monitor entry
> [0x0000000040e7e000..0x0000000040e7ed10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getStats(FSNamesystem.java:3326)
> >>>       - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getCapacityTotal(FSNamesystem.java:3335)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics.doUpdates(FSNamesystemMetrics.java:90)
> >>>       - locked <0x00002aaab3b3c580> (a
> >>> org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics)
> >>>       at
> >>>
> org.apache.hadoop.metrics.spi.AbstractMetricsContext.timerEvent(AbstractMetricsContext.java:284)
> >>>       at
> >>>
> org.apache.hadoop.metrics.spi.AbstractMetricsContext.access$000(AbstractMetricsContext.java:50)
> >>>       at
> >>>
> org.apache.hadoop.metrics.spi.AbstractMetricsContext$1.run(AbstractMetricsContext.java:249)
> >>>       at java.util.TimerThread.mainLoop(Timer.java:512)
> >>>       at java.util.TimerThread.run(Timer.java:462)
> >>>
> >>> "Low Memory Detector" daemon prio=10 tid=0x000000005c306c00 nid=0x426
> >>> runnable [0x0000000000000000..0x0000000000000000]
> >>>  java.lang.Thread.State: RUNNABLE
> >>>
> >>> "CompilerThread1" daemon prio=10 tid=0x000000005c304000 nid=0x425
> waiting
> >>> on condition [0x0000000000000000..0x0000000042830420]
> >>>  java.lang.Thread.State: RUNNABLE
> >>>
> >>> "CompilerThread0" daemon prio=10 tid=0x000000005c300000 nid=0x424
> waiting
> >>> on condition [0x0000000000000000..0x000000004272f3a0]
> >>>  java.lang.Thread.State: RUNNABLE
> >>>
> >>> "Signal Dispatcher" daemon prio=10 tid=0x000000005c2fe400 nid=0x423
> >>> waiting on condition [0x0000000000000000..0x0000000000000000]
> >>>  java.lang.Thread.State: RUNNABLE
> >>>
> >>> "Finalizer" daemon prio=10 tid=0x000000005c2da800 nid=0x422 in
> >>> Object.wait() [0x000000004252e000..0x000000004252ec10]
> >>>  java.lang.Thread.State: WAITING (on object monitor)
> >>>       at java.lang.Object.wait(Native Method)
> >>>       - waiting on <0x00002aaab38ec320> (a
> >>> java.lang.ref.ReferenceQueue$Lock)
> >>>       at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
> >>>       - locked <0x00002aaab38ec320> (a
> java.lang.ref.ReferenceQueue$Lock)
> >>>       at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
> >>>       at
> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> >>>
> >>> "Reference Handler" daemon prio=10 tid=0x000000005c2d8c00 nid=0x421 in
> >>> Object.wait() [0x000000004177e000..0x000000004177ed90]
> >>>  java.lang.Thread.State: WAITING (on object monitor)
> >>>       at java.lang.Object.wait(Native Method)
> >>>       - waiting on <0x00002aaab38e9ec0> (a
> java.lang.ref.Reference$Lock)
> >>>       at java.lang.Object.wait(Object.java:485)
> >>>       at
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> >>>       - locked <0x00002aaab38e9ec0> (a java.lang.ref.Reference$Lock)
> >>>
> >>> "main" prio=10 tid=0x000000005c268400 nid=0x417 in Object.wait()
> >>> [0x0000000040494000..0x0000000040494ed0]
> >>>  java.lang.Thread.State: WAITING (on object monitor)
> >>>       at java.lang.Object.wait(Native Method)
> >>>       - waiting on <0x00002aaab3b3db50> (a
> >>> org.apache.hadoop.ipc.RPC$Server)
> >>>       at java.lang.Object.wait(Object.java:485)
> >>>       at org.apache.hadoop.ipc.Server.join(Server.java:1023)
> >>>       - locked <0x00002aaab3b3db50> (a
> org.apache.hadoop.ipc.RPC$Server)
> >>>       at
> >>> org.apache.hadoop.hdfs.server.namenode.NameNode.join(NameNode.java:221)
> >>>       at
> >>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:870)
> >>>
> >>> "VM Thread" prio=10 tid=0x000000005c2d3400 nid=0x420 runnable
> >>>
> >>> "GC task thread#0 (ParallelGC)" prio=10 tid=0x000000005c273000
> nid=0x418
> >>> runnable
> >>>
> >>> "GC task thread#1 (ParallelGC)" prio=10 tid=0x000000005c274800
> nid=0x419
> >>> runnable
> >>>
> >>> "GC task thread#2 (ParallelGC)" prio=10 tid=0x000000005c276400
> nid=0x41a
> >>> runnable
> >>>
> >>> "GC task thread#3 (ParallelGC)" prio=10 tid=0x000000005c277c00
> nid=0x41b
> >>> runnable
> >>>
> >>> "GC task thread#4 (ParallelGC)" prio=10 tid=0x000000005c279400
> nid=0x41c
> >>> runnable
> >>>
> >>> "GC task thread#5 (ParallelGC)" prio=10 tid=0x000000005c27b000
> nid=0x41d
> >>> runnable
> >>>
> >>> "GC task thread#6 (ParallelGC)" prio=10 tid=0x000000005c27c800
> nid=0x41e
> >>> runnable
> >>>
> >>> "GC task thread#7 (ParallelGC)" prio=10 tid=0x000000005c27e000
> nid=0x41f
> >>> runnable
> >>>
> >>> "VM Periodic Task Thread" prio=10 tid=0x000000005c308c00 nid=0x427
> waiting
> >>> on condition
> >>>
> >>> JNI global references: 984
> >>>
> >>> Heap
> >>> PSYoungGen      total 203712K, used 107631K [0x00002aaadd0b0000,
> >>> 0x00002aaaf1e00000, 0x00002aaaf1e00000)
> >>> eden space 124096K, 86% used
> >>> [0x00002aaadd0b0000,0x00002aaae39cbe58,0x00002aaae49e0000)
> >>> from space 79616K, 0% used
> >>> [0x00002aaae49e0000,0x00002aaae49e0000,0x00002aaae97a0000)
> >>> to   space 113728K, 0% used
> >>> [0x00002aaaeaef0000,0x00002aaaeaef0000,0x00002aaaf1e00000)
> >>> PSOldGen        total 682688K, used 682687K [0x00002aaab3600000,
> >>> 0x00002aaadd0b0000, 0x00002aaadd0b0000)
> >>> object space 682688K, 99% used
> >>> [0x00002aaab3600000,0x00002aaadd0affe8,0x00002aaadd0b0000)
> >>> PSPermGen       total 21248K, used 16689K [0x00002aaaae200000,
> >>> 0x00002aaaaf6c0000, 0x00002aaab3600000)
> >>> object space 21248K, 78% used
> >>> [0x00002aaaae200000,0x00002aaaaf24c660,0x00002aaaaf6c0000)
> >>>
> >>>
> >>
> >>
> >> --
> >> Todd Lipcon
> >> Software Engineer, Cloudera
> >>
> >
> >
> >
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
>
>


-- 
Todd Lipcon
Software Engineer, Cloudera

Re: NameNode deadlocked (help?)

Posted by Brian Bockelman <bb...@cse.unl.edu>.
Hey Todd,

The thread remains in that trace after a few attempts.

Being that no one was going to get anything done quickly, we decided to restart the NN.  Upon restart, I found something *really* interesting - the server had been running for quite awhile with 1GB of heap!

Being that these are the statistics for the cluster:

720246 files and directories, 3096099 blocks = 3816345 total

I have no clue how this system ran before!  Hopefully this is really the issue and not some red herring.

My theory is that we somehow managed to scrape below the memory limits ... the same sysadmin has been complaining for a few months that the NN would lock up for a few seconds at 100% CPU.  This could have been signs that I (stupidly) missed of excessive garbage collection.

Sigh.  Need a beer.

Brian

On May 14, 2010, at 7:46 PM, Todd Lipcon wrote:

> Hey Brian,
> 
> I don't see a way in which this could infinite loop unless some
> datastructure got corrupted.
> 
> If you jstack the NN a few times, does that thread remain in that trace or
> does it move around?
> 
> -Todd
> 
> On Fri, May 14, 2010 at 5:41 PM, Todd Lipcon <to...@cloudera.com> wrote:
> 
>> Hey Brian,
>> 
>> Looks like it's not deadlocked, but rather just busy doing a lot of work:
>> 
>> 
>> "org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@1c778255"
>> daemon prio=10 tid=0x00002aaafc012c00 nid=0x493 runnable
>> [0x00000000413da000..0x00000000413daa10]
>>  java.lang.Thread.State: RUNNABLE
>>       at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeStoredBlock(FSNamesystem.java:3236)
>>       - locked <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>       at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeDatanode(FSNamesystem.java:2695)
>>       at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.heartbeatCheck(FSNamesystem.java:2785)
>>       - locked <0x00002aaab3659dd8> (a java.util.TreeMap)
>>       - locked <0x00002aaab3653848> (a java.util.ArrayList)
>>       - locked <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>       at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2312)
>>       at java.lang.Thread.run(Thread.java:619)
>> 
>> Let me dig up the 19.1 source code and see if this looks like an infinite
>> loop or just one that's tying it up for some number of seconds.
>> 
>> Anything being written into the logs?
>> 
>> -Todd
>> 
>> 
>> On Fri, May 14, 2010 at 5:22 PM, Brian Bockelman <bb...@cse.unl.edu>wrote:
>> 
>>> Hey guys,
>>> 
>>> I know it's 5PM on a Friday, but we just saw one of our big cluster's
>>> namenode's deadlock.
>>> 
>>> This is 0.19.1; does this ring a bell for anyone?  I haven't had any time
>>> to start going through source code, but I figured I'd send out a SOS in case
>>> if this looked familiar.
>>> 
>>> We had restarted this cluster a few hours ago and made the following
>>> changes:
>>> 
>>> 1) Increased the number of datanode handlers from 10 to 40.
>>> 2) Increased ipc.server.listen.queue.size from 128 to 256.
>>> 
>>> If nothing else, I figure a deadlocked NN might be interesting to devs...
>>> 
>>> Brian
>>> 
>>> 2010-05-14 17:11:30
>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (11.2-b01 mixed mode):
>>> 
>>> "IPC Server handler 39 on 9000" daemon prio=10 tid=0x00002aaafc181400
>>> nid=0x4cd waiting for monitor entry [0x0000000045962000..0x0000000045962d90]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
>>>       - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
>>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>       at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>       at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>> 
>>> "IPC Server handler 38 on 9000" daemon prio=10 tid=0x00002aaafc17f800
>>> nid=0x4cc waiting for monitor entry [0x0000000045861000..0x0000000045861d10]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getStats(FSNamesystem.java:3326)
>>>       - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.getStats(NameNode.java:505)
>>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>       at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>       at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>> 
>>> "IPC Server handler 37 on 9000" daemon prio=10 tid=0x00002aaafc17e000
>>> nid=0x4cb waiting for monitor entry [0x000000004575f000..0x0000000045760a90]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>       - waiting to lock <0x00002aaab3843e40> (a
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>       at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>       at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>> 
>>> "IPC Server handler 36 on 9000" daemon prio=10 tid=0x00002aaafc17c400
>>> nid=0x4ca waiting for monitor entry [0x000000004565e000..0x000000004565fa10]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:3281)
>>>       - waiting to lock <0x00002aaab3843e40> (a
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:649)
>>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>       at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>       at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>> 
>>> "IPC Server handler 35 on 9000" daemon prio=10 tid=0x00002aaafc17a800
>>> nid=0x4c9 waiting for monitor entry [0x000000004555e000..0x000000004555eb90]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
>>>       - waiting to lock <0x00002aaab3843e40> (a
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
>>>       at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
>>>       at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>> 
>>> "IPC Server handler 34 on 9000" daemon prio=10 tid=0x00002aaafc178c00
>>> nid=0x4c8 waiting for monitor entry [0x000000004545d000..0x000000004545db10]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
>>>       - waiting to lock <0x00002aaab3843e40> (a
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
>>>       at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
>>>       at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>> 
>>> "IPC Server handler 33 on 9000" daemon prio=10 tid=0x00002aaafc177400
>>> nid=0x4c7 waiting for monitor entry [0x000000004535c000..0x000000004535cc90]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
>>>       - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
>>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>       at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>       at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>> 
>>> "IPC Server handler 32 on 9000" daemon prio=10 tid=0x00002aaafc175800
>>> nid=0x4c6 waiting for monitor entry [0x000000004525a000..0x000000004525bc10]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>       - waiting to lock <0x00002aaab3843e40> (a
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>       at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>       at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>> 
>>> "IPC Server handler 31 on 9000" daemon prio=10 tid=0x00002aaafc173c00
>>> nid=0x4c5 waiting for monitor entry [0x000000004515a000..0x000000004515ad90]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:3281)
>>>       - waiting to lock <0x00002aaab3843e40> (a
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:649)
>>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>       at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>       at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>> 
>>> "IPC Server handler 30 on 9000" daemon prio=10 tid=0x00002aaafc172400
>>> nid=0x4c4 waiting for monitor entry [0x0000000045059000..0x0000000045059d10]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
>>>       - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
>>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>       at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>       at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>> 
>>> "IPC Server handler 29 on 9000" daemon prio=10 tid=0x00002aaafc171000
>>> nid=0x4c3 waiting for monitor entry [0x0000000044f57000..0x0000000044f58a90]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
>>>       - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
>>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>       at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>       at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>> 
>>> "IPC Server handler 28 on 9000" daemon prio=10 tid=0x00002aaafc16f400
>>> nid=0x4c2 waiting for monitor entry [0x0000000044e56000..0x0000000044e57a10]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>       - waiting to lock <0x00002aaab3843e40> (a
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>       at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>       at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>> 
>>> "IPC Server handler 27 on 9000" daemon prio=10 tid=0x00002aaafc16d800
>>> nid=0x4c1 waiting for monitor entry [0x0000000044d56000..0x0000000044d56b90]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>       - waiting to lock <0x00002aaab3843e40> (a
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>       at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>>>       at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>> 
>>> "IPC Server handler 26 on 9000" daemon prio=10 tid=0x00002aaafc16bc00
>>> nid=0x4c0 waiting for monitor entry [0x0000000044c55000..0x0000000044c55b10]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>       - waiting to lock <0x00002aaab3843e40> (a
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>       at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>>>       at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>> 
>>> "IPC Server handler 25 on 9000" daemon prio=10 tid=0x00002aaafc16a400
>>> nid=0x4bf waiting for monitor entry [0x0000000044b54000..0x0000000044b54c90]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
>>>       - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
>>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>       at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>       at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>> 
>>> "IPC Server handler 24 on 9000" daemon prio=10 tid=0x00002aaafc168800
>>> nid=0x4be waiting for monitor entry [0x0000000044a53000..0x0000000044a53c10]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
>>>       - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
>>>       at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
>>>       at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>> 
>>> "IPC Server handler 23 on 9000" daemon prio=10 tid=0x00002aaafc166c00
>>> nid=0x4bd waiting for monitor entry [0x0000000044952000..0x0000000044952d90]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>       - waiting to lock <0x00002aaab3843e40> (a
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>       at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>       at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>> 
>>> "IPC Server handler 22 on 9000" daemon prio=10 tid=0x00002aaafc165400
>>> nid=0x4bc waiting for monitor entry [0x0000000044851000..0x0000000044851d10]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>       - waiting to lock <0x00002aaab3843e40> (a
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>       at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>>>       at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>> 
>>> "IPC Server handler 21 on 9000" daemon prio=10 tid=0x00002aaafc164000
>>> nid=0x4bb waiting for monitor entry [0x0000000044750000..0x0000000044750a90]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>       - waiting to lock <0x00002aaab3843e40> (a
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>       at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>>>       at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>> 
>>> "IPC Server handler 20 on 9000" daemon prio=10 tid=0x00002aaafc162400
>>> nid=0x4ba waiting for monitor entry [0x000000004464f000..0x000000004464fa10]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
>>>       - waiting to lock <0x00002aaab3843e40> (a
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
>>>       at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
>>>       at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>> 
>>> "IPC Server handler 19 on 9000" daemon prio=10 tid=0x00002aaafc488400
>>> nid=0x4b9 waiting for monitor entry [0x000000004454e000..0x000000004454eb90]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
>>>       - waiting to lock <0x00002aaab3843e40> (a
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
>>>       at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
>>>       at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>> 
>>> "IPC Server handler 18 on 9000" daemon prio=10 tid=0x00002aaafc486800
>>> nid=0x4b8 waiting for monitor entry [0x000000004444d000..0x000000004444db10]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.nextGenerationStampForBlock(FSNamesystem.java:4548)
>>>       - waiting to lock <0x00002aaab3843e40> (a
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.nextGenerationStamp(NameNode.java:402)
>>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>       at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>       at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>> 
>>> "IPC Server handler 17 on 9000" daemon prio=10 tid=0x00002aaafc484c00
>>> nid=0x4b7 waiting for monitor entry [0x000000004434c000..0x000000004434cc90]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.nextGenerationStampForBlock(FSNamesystem.java:4548)
>>>       - waiting to lock <0x00002aaab3843e40> (a
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.nextGenerationStamp(NameNode.java:402)
>>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>       at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>       at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>> 
>>> "IPC Server handler 16 on 9000" daemon prio=10 tid=0x00002aaafc483000
>>> nid=0x4b6 waiting for monitor entry [0x000000004424a000..0x000000004424bc10]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>       - waiting to lock <0x00002aaab3843e40> (a
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>       at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>       at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>> 
>>> "IPC Server handler 15 on 9000" daemon prio=10 tid=0x00002aaafc481800
>>> nid=0x4b5 waiting for monitor entry [0x000000004414a000..0x000000004414ad90]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>       - waiting to lock <0x00002aaab3843e40> (a
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>       at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>>>       at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>> 
>>> "IPC Server handler 14 on 9000" daemon prio=10 tid=0x00002aaafc47fc00
>>> nid=0x4b4 waiting for monitor entry [0x0000000044049000..0x0000000044049d10]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
>>>       - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
>>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>       at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>       at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>> 
>>> "IPC Server handler 13 on 9000" daemon prio=10 tid=0x00002aaafc47e000
>>> nid=0x4b3 waiting for monitor entry [0x0000000043f48000..0x0000000043f48a90]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
>>>       - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
>>>       at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
>>>       at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>> 
>>> "IPC Server handler 12 on 9000" daemon prio=10 tid=0x00002aaafc47c400
>>> nid=0x4b2 waiting for monitor entry [0x0000000043e46000..0x0000000043e47a10]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>       - waiting to lock <0x00002aaab3843e40> (a
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>       at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>       at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>> 
>>> "IPC Server handler 11 on 9000" daemon prio=10 tid=0x00002aaafc47b400
>>> nid=0x4b1 waiting for monitor entry [0x0000000043d45000..0x0000000043d46b90]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>       - waiting to lock <0x00002aaab3843e40> (a
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>       at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>       at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>> 
>>> "IPC Server handler 10 on 9000" daemon prio=10 tid=0x00002aaafc479800
>>> nid=0x4b0 waiting for monitor entry [0x0000000043c45000..0x0000000043c45b10]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
>>>       - waiting to lock <0x00002aaab3843e40> (a
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
>>>       at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
>>>       at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>> 
>>> "IPC Server handler 9 on 9000" daemon prio=10 tid=0x00002aaafc477c00
>>> nid=0x4af waiting for monitor entry [0x0000000043b43000..0x0000000043b44c90]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>       - waiting to lock <0x00002aaab3843e40> (a
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>       at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>       at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>> 
>>> "IPC Server handler 8 on 9000" daemon prio=10 tid=0x00002aaafc476400
>>> nid=0x4ae waiting for monitor entry [0x0000000043a42000..0x0000000043a43c10]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>       - waiting to lock <0x00002aaab3843e40> (a
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>       at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>       at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>> 
>>> "IPC Server handler 7 on 9000" daemon prio=10 tid=0x00002aaafc474c00
>>> nid=0x4ad waiting for monitor entry [0x0000000043942000..0x0000000043942d90]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>       - waiting to lock <0x00002aaab3843e40> (a
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>       at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>>>       at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>> 
>>> "IPC Server handler 6 on 9000" daemon prio=10 tid=0x00002aaafc473000
>>> nid=0x4ac waiting for monitor entry [0x0000000043841000..0x0000000043841d10]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>       - waiting to lock <0x00002aaab3843e40> (a
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>       at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>       at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>> 
>>> "IPC Server handler 5 on 9000" daemon prio=10 tid=0x00002aaafc471400
>>> nid=0x4ab waiting for monitor entry [0x0000000043740000..0x0000000043740a90]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>       - waiting to lock <0x00002aaab3843e40> (a
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>       at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>>>       at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>> 
>>> "IPC Server handler 4 on 9000" daemon prio=10 tid=0x00002aaafc121800
>>> nid=0x4aa waiting for monitor entry [0x000000004363f000..0x000000004363fa10]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>       - waiting to lock <0x00002aaab3843e40> (a
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>       at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>>>       at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>> 
>>> "IPC Server handler 3 on 9000" daemon prio=10 tid=0x00002aaafc11fc00
>>> nid=0x4a9 waiting for monitor entry [0x000000004353d000..0x000000004353eb90]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>       - waiting to lock <0x00002aaab3843e40> (a
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>       at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>       at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>> 
>>> "IPC Server handler 2 on 9000" daemon prio=10 tid=0x00002aaafc11e400
>>> nid=0x4a8 waiting for monitor entry [0x000000004343c000..0x000000004343db10]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>       - waiting to lock <0x00002aaab3843e40> (a
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>       at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>       at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>> 
>>> "IPC Server handler 1 on 9000" daemon prio=10 tid=0x00002aaafc467800
>>> nid=0x4a7 waiting for monitor entry [0x000000004333b000..0x000000004333cc90]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>       - waiting to lock <0x00002aaab3843e40> (a
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>       at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>       at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>> 
>>> "IPC Server handler 0 on 9000" daemon prio=10 tid=0x00002aaafc466400
>>> nid=0x4a6 waiting for monitor entry [0x000000004323b000..0x000000004323bc10]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>>       - waiting to lock <0x00002aaab3843e40> (a
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>>       at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>>>       at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>> 
>>> "IPC Server listener on 9000" daemon prio=10 tid=0x00002aaafc12fc00
>>> nid=0x4a5 waiting on condition [0x000000004313a000..0x000000004313ad90]
>>>  java.lang.Thread.State: WAITING (parking)
>>>       at sun.misc.Unsafe.park(Native Method)
>>>       - parking to wait for  <0x00002aaab3b8fd70> (a
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>       at
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>       at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
>>>       at
>>> java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:254)
>>>       at
>>> org.apache.hadoop.ipc.Server$Connection.processData(Server.java:850)
>>>       at
>>> org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:814)
>>>       at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:400)
>>>       at org.apache.hadoop.ipc.Server$Listener.run(Server.java:309)
>>> 
>>> "IPC Server Responder" daemon prio=10 tid=0x00002aaafc0bbc00 nid=0x4a4
>>> runnable [0x0000000043039000..0x0000000043039d10]
>>>  java.lang.Thread.State: RUNNABLE
>>>       at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>       at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>>       at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>       - locked <0x00002aaab3b90e80> (a sun.nio.ch.Util$1)
>>>       - locked <0x00002aaab3b90e68> (a
>>> java.util.Collections$UnmodifiableSet)
>>>       - locked <0x00002aaab3b90b08> (a sun.nio.ch.EPollSelectorImpl)
>>>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>       at org.apache.hadoop.ipc.Server$Responder.run(Server.java:458)
>>> 
>>> "SocketListener0-1" prio=10 tid=0x00002aaafc0ba400 nid=0x4a3 in
>>> Object.wait() [0x0000000042f38000..0x0000000042f38a90]
>>>  java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>>       at java.lang.Object.wait(Native Method)
>>>       - waiting on <0x00002aaadf36cb00> (a
>>> org.mortbay.util.ThreadPool$PoolThread)
>>>       at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:522)
>>>       - locked <0x00002aaadf36cb00> (a
>>> org.mortbay.util.ThreadPool$PoolThread)
>>> 
>>> "SocketListener0-0" prio=10 tid=0x00002aaafc0b4c00 nid=0x4a2 in
>>> Object.wait() [0x0000000042e37000..0x0000000042e37a10]
>>>  java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>>       at java.lang.Object.wait(Native Method)
>>>       - waiting on <0x00002aaadf36c820> (a
>>> org.mortbay.util.ThreadPool$PoolThread)
>>>       at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:522)
>>>       - locked <0x00002aaadf36c820> (a
>>> org.mortbay.util.ThreadPool$PoolThread)
>>> 
>>> "Acceptor ServerSocket[addr=proxy-1/169.228.130.63,port=0,localport=50070]"
>>> prio=10 tid=0x00002aaafc0c8400 nid=0x4a1 runnable
>>> [0x0000000042d36000..0x0000000042d36b90]
>>>  java.lang.Thread.State: RUNNABLE
>>>       at java.lang.Object.<init>(Object.java:20)
>>>       at java.net.SocketImpl.<init>(SocketImpl.java:27)
>>>       at java.net.PlainSocketImpl.<init>(PlainSocketImpl.java:90)
>>>       at java.net.SocksSocketImpl.<init>(SocksSocketImpl.java:33)
>>>       at java.net.Socket.setImpl(Socket.java:434)
>>>       at java.net.ServerSocket.implAccept(ServerSocket.java:445)
>>>       at java.net.ServerSocket.accept(ServerSocket.java:421)
>>>       at
>>> org.mortbay.util.ThreadedServer.acceptSocket(ThreadedServer.java:432)
>>>       at
>>> org.mortbay.util.ThreadedServer$Acceptor.run(ThreadedServer.java:631)
>>> 
>>> "SessionScavenger" daemon prio=10 tid=0x00002aaafc0c1000 nid=0x4a0 waiting
>>> on condition [0x0000000042c35000..0x0000000042c35b10]
>>>  java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>       at java.lang.Thread.sleep(Native Method)
>>>       at
>>> org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(AbstractSessionManager.java:587)
>>> 
>>> "SessionScavenger" daemon prio=10 tid=0x00002aaafc0fa000 nid=0x49f waiting
>>> on condition [0x0000000040b2a000..0x0000000040b2ac90]
>>>  java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>       at java.lang.Thread.sleep(Native Method)
>>>       at
>>> org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(AbstractSessionManager.java:587)
>>> 
>>> "SessionScavenger" daemon prio=10 tid=0x00002aaafc139800 nid=0x49a waiting
>>> on condition [0x00000000408b7000..0x00000000408b7c10]
>>>  java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>       at java.lang.Thread.sleep(Native Method)
>>>       at
>>> org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(AbstractSessionManager.java:587)
>>> 
>>> 
>>> "org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@1b0deb5f"
>>> daemon prio=10 tid=0x00002aaafc0a2800 nid=0x496 waiting for monitor entry
>>> [0x0000000042b34000..0x0000000042b34d90]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:61)
>>>       - waiting to lock <0x00002aaab3843e40> (a
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>       at java.lang.Thread.run(Thread.java:619)
>>> 
>>> 
>>> "org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@784a7df6"
>>> daemon prio=10 tid=0x00002aaafc090c00 nid=0x495 waiting for monitor entry
>>> [0x00000000411f8000..0x00000000411f8d10]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.computeReplicationWork(FSNamesystem.java:2410)
>>>       - waiting to lock <0x00002aaab3843e40> (a
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.computeDatanodeWork(FSNamesystem.java:2376)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2333)
>>>       at java.lang.Thread.run(Thread.java:619)
>>> 
>>> "org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@d24111a"
>>> daemon prio=10 tid=0x00002aaafc013400 nid=0x494 waiting for monitor entry
>>> [0x00000000410f7000..0x00000000410f7a90]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:349)
>>>       - waiting to lock <0x00002aaab3843e40> (a
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>       at java.lang.Thread.run(Thread.java:619)
>>> 
>>> 
>>> "org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@1c778255"
>>> daemon prio=10 tid=0x00002aaafc012c00 nid=0x493 runnable
>>> [0x00000000413da000..0x00000000413daa10]
>>>  java.lang.Thread.State: RUNNABLE
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeStoredBlock(FSNamesystem.java:3236)
>>>       - locked <0x00002aaab3843e40> (a
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeDatanode(FSNamesystem.java:2695)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.heartbeatCheck(FSNamesystem.java:2785)
>>>       - locked <0x00002aaab3659dd8> (a java.util.TreeMap)
>>>       - locked <0x00002aaab3653848> (a java.util.ArrayList)
>>>       - locked <0x00002aaab3843e40> (a
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2312)
>>>       at java.lang.Thread.run(Thread.java:619)
>>> 
>>> 
>>> "org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@44676e3f"
>>> daemon prio=10 tid=0x00002aaafc012800 nid=0x492 waiting on condition
>>> [0x0000000040d16000..0x0000000040d16b90]
>>>  java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>       at java.lang.Thread.sleep(Native Method)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
>>>       at java.lang.Thread.run(Thread.java:619)
>>> 
>>> "Timer thread for monitoring dfs" daemon prio=10 tid=0x00002aaafc03d800
>>> nid=0x428 waiting for monitor entry [0x0000000040e7e000..0x0000000040e7ed10]
>>>  java.lang.Thread.State: BLOCKED (on object monitor)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getStats(FSNamesystem.java:3326)
>>>       - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getCapacityTotal(FSNamesystem.java:3335)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics.doUpdates(FSNamesystemMetrics.java:90)
>>>       - locked <0x00002aaab3b3c580> (a
>>> org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics)
>>>       at
>>> org.apache.hadoop.metrics.spi.AbstractMetricsContext.timerEvent(AbstractMetricsContext.java:284)
>>>       at
>>> org.apache.hadoop.metrics.spi.AbstractMetricsContext.access$000(AbstractMetricsContext.java:50)
>>>       at
>>> org.apache.hadoop.metrics.spi.AbstractMetricsContext$1.run(AbstractMetricsContext.java:249)
>>>       at java.util.TimerThread.mainLoop(Timer.java:512)
>>>       at java.util.TimerThread.run(Timer.java:462)
>>> 
>>> "Low Memory Detector" daemon prio=10 tid=0x000000005c306c00 nid=0x426
>>> runnable [0x0000000000000000..0x0000000000000000]
>>>  java.lang.Thread.State: RUNNABLE
>>> 
>>> "CompilerThread1" daemon prio=10 tid=0x000000005c304000 nid=0x425 waiting
>>> on condition [0x0000000000000000..0x0000000042830420]
>>>  java.lang.Thread.State: RUNNABLE
>>> 
>>> "CompilerThread0" daemon prio=10 tid=0x000000005c300000 nid=0x424 waiting
>>> on condition [0x0000000000000000..0x000000004272f3a0]
>>>  java.lang.Thread.State: RUNNABLE
>>> 
>>> "Signal Dispatcher" daemon prio=10 tid=0x000000005c2fe400 nid=0x423
>>> waiting on condition [0x0000000000000000..0x0000000000000000]
>>>  java.lang.Thread.State: RUNNABLE
>>> 
>>> "Finalizer" daemon prio=10 tid=0x000000005c2da800 nid=0x422 in
>>> Object.wait() [0x000000004252e000..0x000000004252ec10]
>>>  java.lang.Thread.State: WAITING (on object monitor)
>>>       at java.lang.Object.wait(Native Method)
>>>       - waiting on <0x00002aaab38ec320> (a
>>> java.lang.ref.ReferenceQueue$Lock)
>>>       at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
>>>       - locked <0x00002aaab38ec320> (a java.lang.ref.ReferenceQueue$Lock)
>>>       at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
>>>       at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>>> 
>>> "Reference Handler" daemon prio=10 tid=0x000000005c2d8c00 nid=0x421 in
>>> Object.wait() [0x000000004177e000..0x000000004177ed90]
>>>  java.lang.Thread.State: WAITING (on object monitor)
>>>       at java.lang.Object.wait(Native Method)
>>>       - waiting on <0x00002aaab38e9ec0> (a java.lang.ref.Reference$Lock)
>>>       at java.lang.Object.wait(Object.java:485)
>>>       at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>>>       - locked <0x00002aaab38e9ec0> (a java.lang.ref.Reference$Lock)
>>> 
>>> "main" prio=10 tid=0x000000005c268400 nid=0x417 in Object.wait()
>>> [0x0000000040494000..0x0000000040494ed0]
>>>  java.lang.Thread.State: WAITING (on object monitor)
>>>       at java.lang.Object.wait(Native Method)
>>>       - waiting on <0x00002aaab3b3db50> (a
>>> org.apache.hadoop.ipc.RPC$Server)
>>>       at java.lang.Object.wait(Object.java:485)
>>>       at org.apache.hadoop.ipc.Server.join(Server.java:1023)
>>>       - locked <0x00002aaab3b3db50> (a org.apache.hadoop.ipc.RPC$Server)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.join(NameNode.java:221)
>>>       at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:870)
>>> 
>>> "VM Thread" prio=10 tid=0x000000005c2d3400 nid=0x420 runnable
>>> 
>>> "GC task thread#0 (ParallelGC)" prio=10 tid=0x000000005c273000 nid=0x418
>>> runnable
>>> 
>>> "GC task thread#1 (ParallelGC)" prio=10 tid=0x000000005c274800 nid=0x419
>>> runnable
>>> 
>>> "GC task thread#2 (ParallelGC)" prio=10 tid=0x000000005c276400 nid=0x41a
>>> runnable
>>> 
>>> "GC task thread#3 (ParallelGC)" prio=10 tid=0x000000005c277c00 nid=0x41b
>>> runnable
>>> 
>>> "GC task thread#4 (ParallelGC)" prio=10 tid=0x000000005c279400 nid=0x41c
>>> runnable
>>> 
>>> "GC task thread#5 (ParallelGC)" prio=10 tid=0x000000005c27b000 nid=0x41d
>>> runnable
>>> 
>>> "GC task thread#6 (ParallelGC)" prio=10 tid=0x000000005c27c800 nid=0x41e
>>> runnable
>>> 
>>> "GC task thread#7 (ParallelGC)" prio=10 tid=0x000000005c27e000 nid=0x41f
>>> runnable
>>> 
>>> "VM Periodic Task Thread" prio=10 tid=0x000000005c308c00 nid=0x427 waiting
>>> on condition
>>> 
>>> JNI global references: 984
>>> 
>>> Heap
>>> PSYoungGen      total 203712K, used 107631K [0x00002aaadd0b0000,
>>> 0x00002aaaf1e00000, 0x00002aaaf1e00000)
>>> eden space 124096K, 86% used
>>> [0x00002aaadd0b0000,0x00002aaae39cbe58,0x00002aaae49e0000)
>>> from space 79616K, 0% used
>>> [0x00002aaae49e0000,0x00002aaae49e0000,0x00002aaae97a0000)
>>> to   space 113728K, 0% used
>>> [0x00002aaaeaef0000,0x00002aaaeaef0000,0x00002aaaf1e00000)
>>> PSOldGen        total 682688K, used 682687K [0x00002aaab3600000,
>>> 0x00002aaadd0b0000, 0x00002aaadd0b0000)
>>> object space 682688K, 99% used
>>> [0x00002aaab3600000,0x00002aaadd0affe8,0x00002aaadd0b0000)
>>> PSPermGen       total 21248K, used 16689K [0x00002aaaae200000,
>>> 0x00002aaaaf6c0000, 0x00002aaab3600000)
>>> object space 21248K, 78% used
>>> [0x00002aaaae200000,0x00002aaaaf24c660,0x00002aaaaf6c0000)
>>> 
>>> 
>> 
>> 
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>> 
> 
> 
> 
> -- 
> Todd Lipcon
> Software Engineer, Cloudera

Re: NameNode deadlocked (help?)

Posted by Todd Lipcon <to...@cloudera.com>.
Hey Brian,

I don't see a way in which this could infinite loop unless some
datastructure got corrupted.

If you jstack the NN a few times, does that thread remain in that trace or
does it move around?

-Todd

On Fri, May 14, 2010 at 5:41 PM, Todd Lipcon <to...@cloudera.com> wrote:

> Hey Brian,
>
> Looks like it's not deadlocked, but rather just busy doing a lot of work:
>
>
> "org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@1c778255"
> daemon prio=10 tid=0x00002aaafc012c00 nid=0x493 runnable
> [0x00000000413da000..0x00000000413daa10]
>   java.lang.Thread.State: RUNNABLE
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeStoredBlock(FSNamesystem.java:3236)
>        - locked <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeDatanode(FSNamesystem.java:2695)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.heartbeatCheck(FSNamesystem.java:2785)
>        - locked <0x00002aaab3659dd8> (a java.util.TreeMap)
>        - locked <0x00002aaab3653848> (a java.util.ArrayList)
>        - locked <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2312)
>        at java.lang.Thread.run(Thread.java:619)
>
> Let me dig up the 19.1 source code and see if this looks like an infinite
> loop or just one that's tying it up for some number of seconds.
>
> Anything being written into the logs?
>
> -Todd
>
>
> On Fri, May 14, 2010 at 5:22 PM, Brian Bockelman <bb...@cse.unl.edu>wrote:
>
>> Hey guys,
>>
>> I know it's 5PM on a Friday, but we just saw one of our big cluster's
>> namenode's deadlock.
>>
>> This is 0.19.1; does this ring a bell for anyone?  I haven't had any time
>> to start going through source code, but I figured I'd send out a SOS in case
>> if this looked familiar.
>>
>> We had restarted this cluster a few hours ago and made the following
>> changes:
>>
>> 1) Increased the number of datanode handlers from 10 to 40.
>> 2) Increased ipc.server.listen.queue.size from 128 to 256.
>>
>> If nothing else, I figure a deadlocked NN might be interesting to devs...
>>
>> Brian
>>
>> 2010-05-14 17:11:30
>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (11.2-b01 mixed mode):
>>
>> "IPC Server handler 39 on 9000" daemon prio=10 tid=0x00002aaafc181400
>> nid=0x4cd waiting for monitor entry [0x0000000045962000..0x0000000045962d90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
>>        - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 38 on 9000" daemon prio=10 tid=0x00002aaafc17f800
>> nid=0x4cc waiting for monitor entry [0x0000000045861000..0x0000000045861d10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getStats(FSNamesystem.java:3326)
>>        - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getStats(NameNode.java:505)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 37 on 9000" daemon prio=10 tid=0x00002aaafc17e000
>> nid=0x4cb waiting for monitor entry [0x000000004575f000..0x0000000045760a90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 36 on 9000" daemon prio=10 tid=0x00002aaafc17c400
>> nid=0x4ca waiting for monitor entry [0x000000004565e000..0x000000004565fa10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:3281)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:649)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 35 on 9000" daemon prio=10 tid=0x00002aaafc17a800
>> nid=0x4c9 waiting for monitor entry [0x000000004555e000..0x000000004555eb90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
>>        at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 34 on 9000" daemon prio=10 tid=0x00002aaafc178c00
>> nid=0x4c8 waiting for monitor entry [0x000000004545d000..0x000000004545db10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
>>        at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 33 on 9000" daemon prio=10 tid=0x00002aaafc177400
>> nid=0x4c7 waiting for monitor entry [0x000000004535c000..0x000000004535cc90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
>>        - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 32 on 9000" daemon prio=10 tid=0x00002aaafc175800
>> nid=0x4c6 waiting for monitor entry [0x000000004525a000..0x000000004525bc10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 31 on 9000" daemon prio=10 tid=0x00002aaafc173c00
>> nid=0x4c5 waiting for monitor entry [0x000000004515a000..0x000000004515ad90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:3281)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:649)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 30 on 9000" daemon prio=10 tid=0x00002aaafc172400
>> nid=0x4c4 waiting for monitor entry [0x0000000045059000..0x0000000045059d10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
>>        - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 29 on 9000" daemon prio=10 tid=0x00002aaafc171000
>> nid=0x4c3 waiting for monitor entry [0x0000000044f57000..0x0000000044f58a90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
>>        - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 28 on 9000" daemon prio=10 tid=0x00002aaafc16f400
>> nid=0x4c2 waiting for monitor entry [0x0000000044e56000..0x0000000044e57a10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 27 on 9000" daemon prio=10 tid=0x00002aaafc16d800
>> nid=0x4c1 waiting for monitor entry [0x0000000044d56000..0x0000000044d56b90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 26 on 9000" daemon prio=10 tid=0x00002aaafc16bc00
>> nid=0x4c0 waiting for monitor entry [0x0000000044c55000..0x0000000044c55b10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 25 on 9000" daemon prio=10 tid=0x00002aaafc16a400
>> nid=0x4bf waiting for monitor entry [0x0000000044b54000..0x0000000044b54c90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
>>        - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 24 on 9000" daemon prio=10 tid=0x00002aaafc168800
>> nid=0x4be waiting for monitor entry [0x0000000044a53000..0x0000000044a53c10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
>>        - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
>>        at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 23 on 9000" daemon prio=10 tid=0x00002aaafc166c00
>> nid=0x4bd waiting for monitor entry [0x0000000044952000..0x0000000044952d90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 22 on 9000" daemon prio=10 tid=0x00002aaafc165400
>> nid=0x4bc waiting for monitor entry [0x0000000044851000..0x0000000044851d10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 21 on 9000" daemon prio=10 tid=0x00002aaafc164000
>> nid=0x4bb waiting for monitor entry [0x0000000044750000..0x0000000044750a90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 20 on 9000" daemon prio=10 tid=0x00002aaafc162400
>> nid=0x4ba waiting for monitor entry [0x000000004464f000..0x000000004464fa10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
>>        at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 19 on 9000" daemon prio=10 tid=0x00002aaafc488400
>> nid=0x4b9 waiting for monitor entry [0x000000004454e000..0x000000004454eb90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
>>        at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 18 on 9000" daemon prio=10 tid=0x00002aaafc486800
>> nid=0x4b8 waiting for monitor entry [0x000000004444d000..0x000000004444db10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.nextGenerationStampForBlock(FSNamesystem.java:4548)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.nextGenerationStamp(NameNode.java:402)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 17 on 9000" daemon prio=10 tid=0x00002aaafc484c00
>> nid=0x4b7 waiting for monitor entry [0x000000004434c000..0x000000004434cc90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.nextGenerationStampForBlock(FSNamesystem.java:4548)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.nextGenerationStamp(NameNode.java:402)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 16 on 9000" daemon prio=10 tid=0x00002aaafc483000
>> nid=0x4b6 waiting for monitor entry [0x000000004424a000..0x000000004424bc10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 15 on 9000" daemon prio=10 tid=0x00002aaafc481800
>> nid=0x4b5 waiting for monitor entry [0x000000004414a000..0x000000004414ad90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 14 on 9000" daemon prio=10 tid=0x00002aaafc47fc00
>> nid=0x4b4 waiting for monitor entry [0x0000000044049000..0x0000000044049d10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
>>        - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 13 on 9000" daemon prio=10 tid=0x00002aaafc47e000
>> nid=0x4b3 waiting for monitor entry [0x0000000043f48000..0x0000000043f48a90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
>>        - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
>>        at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 12 on 9000" daemon prio=10 tid=0x00002aaafc47c400
>> nid=0x4b2 waiting for monitor entry [0x0000000043e46000..0x0000000043e47a10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 11 on 9000" daemon prio=10 tid=0x00002aaafc47b400
>> nid=0x4b1 waiting for monitor entry [0x0000000043d45000..0x0000000043d46b90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 10 on 9000" daemon prio=10 tid=0x00002aaafc479800
>> nid=0x4b0 waiting for monitor entry [0x0000000043c45000..0x0000000043c45b10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
>>        at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 9 on 9000" daemon prio=10 tid=0x00002aaafc477c00
>> nid=0x4af waiting for monitor entry [0x0000000043b43000..0x0000000043b44c90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 8 on 9000" daemon prio=10 tid=0x00002aaafc476400
>> nid=0x4ae waiting for monitor entry [0x0000000043a42000..0x0000000043a43c10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 7 on 9000" daemon prio=10 tid=0x00002aaafc474c00
>> nid=0x4ad waiting for monitor entry [0x0000000043942000..0x0000000043942d90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 6 on 9000" daemon prio=10 tid=0x00002aaafc473000
>> nid=0x4ac waiting for monitor entry [0x0000000043841000..0x0000000043841d10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 5 on 9000" daemon prio=10 tid=0x00002aaafc471400
>> nid=0x4ab waiting for monitor entry [0x0000000043740000..0x0000000043740a90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 4 on 9000" daemon prio=10 tid=0x00002aaafc121800
>> nid=0x4aa waiting for monitor entry [0x000000004363f000..0x000000004363fa10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 3 on 9000" daemon prio=10 tid=0x00002aaafc11fc00
>> nid=0x4a9 waiting for monitor entry [0x000000004353d000..0x000000004353eb90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 2 on 9000" daemon prio=10 tid=0x00002aaafc11e400
>> nid=0x4a8 waiting for monitor entry [0x000000004343c000..0x000000004343db10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 1 on 9000" daemon prio=10 tid=0x00002aaafc467800
>> nid=0x4a7 waiting for monitor entry [0x000000004333b000..0x000000004333cc90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 0 on 9000" daemon prio=10 tid=0x00002aaafc466400
>> nid=0x4a6 waiting for monitor entry [0x000000004323b000..0x000000004323bc10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server listener on 9000" daemon prio=10 tid=0x00002aaafc12fc00
>> nid=0x4a5 waiting on condition [0x000000004313a000..0x000000004313ad90]
>>   java.lang.Thread.State: WAITING (parking)
>>        at sun.misc.Unsafe.park(Native Method)
>>        - parking to wait for  <0x00002aaab3b8fd70> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>        at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>        at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
>>        at
>> java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:254)
>>        at
>> org.apache.hadoop.ipc.Server$Connection.processData(Server.java:850)
>>        at
>> org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:814)
>>        at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:400)
>>        at org.apache.hadoop.ipc.Server$Listener.run(Server.java:309)
>>
>> "IPC Server Responder" daemon prio=10 tid=0x00002aaafc0bbc00 nid=0x4a4
>> runnable [0x0000000043039000..0x0000000043039d10]
>>   java.lang.Thread.State: RUNNABLE
>>        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>        - locked <0x00002aaab3b90e80> (a sun.nio.ch.Util$1)
>>        - locked <0x00002aaab3b90e68> (a
>> java.util.Collections$UnmodifiableSet)
>>        - locked <0x00002aaab3b90b08> (a sun.nio.ch.EPollSelectorImpl)
>>        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>        at org.apache.hadoop.ipc.Server$Responder.run(Server.java:458)
>>
>> "SocketListener0-1" prio=10 tid=0x00002aaafc0ba400 nid=0x4a3 in
>> Object.wait() [0x0000000042f38000..0x0000000042f38a90]
>>   java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>        at java.lang.Object.wait(Native Method)
>>        - waiting on <0x00002aaadf36cb00> (a
>> org.mortbay.util.ThreadPool$PoolThread)
>>        at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:522)
>>        - locked <0x00002aaadf36cb00> (a
>> org.mortbay.util.ThreadPool$PoolThread)
>>
>> "SocketListener0-0" prio=10 tid=0x00002aaafc0b4c00 nid=0x4a2 in
>> Object.wait() [0x0000000042e37000..0x0000000042e37a10]
>>   java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>        at java.lang.Object.wait(Native Method)
>>        - waiting on <0x00002aaadf36c820> (a
>> org.mortbay.util.ThreadPool$PoolThread)
>>        at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:522)
>>        - locked <0x00002aaadf36c820> (a
>> org.mortbay.util.ThreadPool$PoolThread)
>>
>> "Acceptor ServerSocket[addr=proxy-1/169.228.130.63,port=0,localport=50070]"
>> prio=10 tid=0x00002aaafc0c8400 nid=0x4a1 runnable
>> [0x0000000042d36000..0x0000000042d36b90]
>>   java.lang.Thread.State: RUNNABLE
>>        at java.lang.Object.<init>(Object.java:20)
>>        at java.net.SocketImpl.<init>(SocketImpl.java:27)
>>        at java.net.PlainSocketImpl.<init>(PlainSocketImpl.java:90)
>>        at java.net.SocksSocketImpl.<init>(SocksSocketImpl.java:33)
>>        at java.net.Socket.setImpl(Socket.java:434)
>>        at java.net.ServerSocket.implAccept(ServerSocket.java:445)
>>        at java.net.ServerSocket.accept(ServerSocket.java:421)
>>        at
>> org.mortbay.util.ThreadedServer.acceptSocket(ThreadedServer.java:432)
>>        at
>> org.mortbay.util.ThreadedServer$Acceptor.run(ThreadedServer.java:631)
>>
>> "SessionScavenger" daemon prio=10 tid=0x00002aaafc0c1000 nid=0x4a0 waiting
>> on condition [0x0000000042c35000..0x0000000042c35b10]
>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>        at java.lang.Thread.sleep(Native Method)
>>        at
>> org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(AbstractSessionManager.java:587)
>>
>> "SessionScavenger" daemon prio=10 tid=0x00002aaafc0fa000 nid=0x49f waiting
>> on condition [0x0000000040b2a000..0x0000000040b2ac90]
>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>        at java.lang.Thread.sleep(Native Method)
>>        at
>> org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(AbstractSessionManager.java:587)
>>
>> "SessionScavenger" daemon prio=10 tid=0x00002aaafc139800 nid=0x49a waiting
>> on condition [0x00000000408b7000..0x00000000408b7c10]
>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>        at java.lang.Thread.sleep(Native Method)
>>        at
>> org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(AbstractSessionManager.java:587)
>>
>>
>> "org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@1b0deb5f"
>> daemon prio=10 tid=0x00002aaafc0a2800 nid=0x496 waiting for monitor entry
>> [0x0000000042b34000..0x0000000042b34d90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:61)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at java.lang.Thread.run(Thread.java:619)
>>
>>
>> "org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@784a7df6"
>> daemon prio=10 tid=0x00002aaafc090c00 nid=0x495 waiting for monitor entry
>> [0x00000000411f8000..0x00000000411f8d10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.computeReplicationWork(FSNamesystem.java:2410)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.computeDatanodeWork(FSNamesystem.java:2376)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2333)
>>        at java.lang.Thread.run(Thread.java:619)
>>
>> "org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@d24111a"
>> daemon prio=10 tid=0x00002aaafc013400 nid=0x494 waiting for monitor entry
>> [0x00000000410f7000..0x00000000410f7a90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:349)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at java.lang.Thread.run(Thread.java:619)
>>
>>
>> "org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@1c778255"
>> daemon prio=10 tid=0x00002aaafc012c00 nid=0x493 runnable
>> [0x00000000413da000..0x00000000413daa10]
>>   java.lang.Thread.State: RUNNABLE
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeStoredBlock(FSNamesystem.java:3236)
>>        - locked <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeDatanode(FSNamesystem.java:2695)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.heartbeatCheck(FSNamesystem.java:2785)
>>        - locked <0x00002aaab3659dd8> (a java.util.TreeMap)
>>        - locked <0x00002aaab3653848> (a java.util.ArrayList)
>>        - locked <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2312)
>>        at java.lang.Thread.run(Thread.java:619)
>>
>>
>> "org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@44676e3f"
>> daemon prio=10 tid=0x00002aaafc012800 nid=0x492 waiting on condition
>> [0x0000000040d16000..0x0000000040d16b90]
>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>        at java.lang.Thread.sleep(Native Method)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
>>        at java.lang.Thread.run(Thread.java:619)
>>
>> "Timer thread for monitoring dfs" daemon prio=10 tid=0x00002aaafc03d800
>> nid=0x428 waiting for monitor entry [0x0000000040e7e000..0x0000000040e7ed10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getStats(FSNamesystem.java:3326)
>>        - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getCapacityTotal(FSNamesystem.java:3335)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics.doUpdates(FSNamesystemMetrics.java:90)
>>        - locked <0x00002aaab3b3c580> (a
>> org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics)
>>        at
>> org.apache.hadoop.metrics.spi.AbstractMetricsContext.timerEvent(AbstractMetricsContext.java:284)
>>        at
>> org.apache.hadoop.metrics.spi.AbstractMetricsContext.access$000(AbstractMetricsContext.java:50)
>>        at
>> org.apache.hadoop.metrics.spi.AbstractMetricsContext$1.run(AbstractMetricsContext.java:249)
>>        at java.util.TimerThread.mainLoop(Timer.java:512)
>>        at java.util.TimerThread.run(Timer.java:462)
>>
>> "Low Memory Detector" daemon prio=10 tid=0x000000005c306c00 nid=0x426
>> runnable [0x0000000000000000..0x0000000000000000]
>>   java.lang.Thread.State: RUNNABLE
>>
>> "CompilerThread1" daemon prio=10 tid=0x000000005c304000 nid=0x425 waiting
>> on condition [0x0000000000000000..0x0000000042830420]
>>   java.lang.Thread.State: RUNNABLE
>>
>> "CompilerThread0" daemon prio=10 tid=0x000000005c300000 nid=0x424 waiting
>> on condition [0x0000000000000000..0x000000004272f3a0]
>>   java.lang.Thread.State: RUNNABLE
>>
>> "Signal Dispatcher" daemon prio=10 tid=0x000000005c2fe400 nid=0x423
>> waiting on condition [0x0000000000000000..0x0000000000000000]
>>   java.lang.Thread.State: RUNNABLE
>>
>> "Finalizer" daemon prio=10 tid=0x000000005c2da800 nid=0x422 in
>> Object.wait() [0x000000004252e000..0x000000004252ec10]
>>   java.lang.Thread.State: WAITING (on object monitor)
>>        at java.lang.Object.wait(Native Method)
>>        - waiting on <0x00002aaab38ec320> (a
>> java.lang.ref.ReferenceQueue$Lock)
>>        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
>>        - locked <0x00002aaab38ec320> (a java.lang.ref.ReferenceQueue$Lock)
>>        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
>>        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>>
>> "Reference Handler" daemon prio=10 tid=0x000000005c2d8c00 nid=0x421 in
>> Object.wait() [0x000000004177e000..0x000000004177ed90]
>>   java.lang.Thread.State: WAITING (on object monitor)
>>        at java.lang.Object.wait(Native Method)
>>        - waiting on <0x00002aaab38e9ec0> (a java.lang.ref.Reference$Lock)
>>        at java.lang.Object.wait(Object.java:485)
>>        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>>        - locked <0x00002aaab38e9ec0> (a java.lang.ref.Reference$Lock)
>>
>> "main" prio=10 tid=0x000000005c268400 nid=0x417 in Object.wait()
>> [0x0000000040494000..0x0000000040494ed0]
>>   java.lang.Thread.State: WAITING (on object monitor)
>>        at java.lang.Object.wait(Native Method)
>>        - waiting on <0x00002aaab3b3db50> (a
>> org.apache.hadoop.ipc.RPC$Server)
>>        at java.lang.Object.wait(Object.java:485)
>>        at org.apache.hadoop.ipc.Server.join(Server.java:1023)
>>        - locked <0x00002aaab3b3db50> (a org.apache.hadoop.ipc.RPC$Server)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.join(NameNode.java:221)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:870)
>>
>> "VM Thread" prio=10 tid=0x000000005c2d3400 nid=0x420 runnable
>>
>> "GC task thread#0 (ParallelGC)" prio=10 tid=0x000000005c273000 nid=0x418
>> runnable
>>
>> "GC task thread#1 (ParallelGC)" prio=10 tid=0x000000005c274800 nid=0x419
>> runnable
>>
>> "GC task thread#2 (ParallelGC)" prio=10 tid=0x000000005c276400 nid=0x41a
>> runnable
>>
>> "GC task thread#3 (ParallelGC)" prio=10 tid=0x000000005c277c00 nid=0x41b
>> runnable
>>
>> "GC task thread#4 (ParallelGC)" prio=10 tid=0x000000005c279400 nid=0x41c
>> runnable
>>
>> "GC task thread#5 (ParallelGC)" prio=10 tid=0x000000005c27b000 nid=0x41d
>> runnable
>>
>> "GC task thread#6 (ParallelGC)" prio=10 tid=0x000000005c27c800 nid=0x41e
>> runnable
>>
>> "GC task thread#7 (ParallelGC)" prio=10 tid=0x000000005c27e000 nid=0x41f
>> runnable
>>
>> "VM Periodic Task Thread" prio=10 tid=0x000000005c308c00 nid=0x427 waiting
>> on condition
>>
>> JNI global references: 984
>>
>> Heap
>>  PSYoungGen      total 203712K, used 107631K [0x00002aaadd0b0000,
>> 0x00002aaaf1e00000, 0x00002aaaf1e00000)
>>  eden space 124096K, 86% used
>> [0x00002aaadd0b0000,0x00002aaae39cbe58,0x00002aaae49e0000)
>>  from space 79616K, 0% used
>> [0x00002aaae49e0000,0x00002aaae49e0000,0x00002aaae97a0000)
>>  to   space 113728K, 0% used
>> [0x00002aaaeaef0000,0x00002aaaeaef0000,0x00002aaaf1e00000)
>>  PSOldGen        total 682688K, used 682687K [0x00002aaab3600000,
>> 0x00002aaadd0b0000, 0x00002aaadd0b0000)
>>  object space 682688K, 99% used
>> [0x00002aaab3600000,0x00002aaadd0affe8,0x00002aaadd0b0000)
>>  PSPermGen       total 21248K, used 16689K [0x00002aaaae200000,
>> 0x00002aaaaf6c0000, 0x00002aaab3600000)
>>  object space 21248K, 78% used
>> [0x00002aaaae200000,0x00002aaaaf24c660,0x00002aaaaf6c0000)
>>
>>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>



-- 
Todd Lipcon
Software Engineer, Cloudera

Re: NameNode deadlocked (help?)

Posted by Todd Lipcon <to...@cloudera.com>.
Hey Brian,

Looks like it's not deadlocked, but rather just busy doing a lot of work:

"org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@1c778255"
daemon prio=10 tid=0x00002aaafc012c00 nid=0x493 runnable
[0x00000000413da000..0x00000000413daa10]
  java.lang.Thread.State: RUNNABLE
       at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeStoredBlock(FSNamesystem.java:3236)
       - locked <0x00002aaab3843e40> (a
org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
       at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeDatanode(FSNamesystem.java:2695)
       at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.heartbeatCheck(FSNamesystem.java:2785)
       - locked <0x00002aaab3659dd8> (a java.util.TreeMap)
       - locked <0x00002aaab3653848> (a java.util.ArrayList)
       - locked <0x00002aaab3843e40> (a
org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
       at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2312)
       at java.lang.Thread.run(Thread.java:619)

Let me dig up the 19.1 source code and see if this looks like an infinite
loop or just one that's tying it up for some number of seconds.

Anything being written into the logs?

-Todd


On Fri, May 14, 2010 at 5:22 PM, Brian Bockelman <bb...@cse.unl.edu>wrote:

> Hey guys,
>
> I know it's 5PM on a Friday, but we just saw one of our big cluster's
> namenode's deadlock.
>
> This is 0.19.1; does this ring a bell for anyone?  I haven't had any time
> to start going through source code, but I figured I'd send out a SOS in case
> if this looked familiar.
>
> We had restarted this cluster a few hours ago and made the following
> changes:
>
> 1) Increased the number of datanode handlers from 10 to 40.
> 2) Increased ipc.server.listen.queue.size from 128 to 256.
>
> If nothing else, I figure a deadlocked NN might be interesting to devs...
>
> Brian
>
> 2010-05-14 17:11:30
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (11.2-b01 mixed mode):
>
> "IPC Server handler 39 on 9000" daemon prio=10 tid=0x00002aaafc181400
> nid=0x4cd waiting for monitor entry [0x0000000045962000..0x0000000045962d90]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
>        - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 38 on 9000" daemon prio=10 tid=0x00002aaafc17f800
> nid=0x4cc waiting for monitor entry [0x0000000045861000..0x0000000045861d10]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getStats(FSNamesystem.java:3326)
>        - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getStats(NameNode.java:505)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 37 on 9000" daemon prio=10 tid=0x00002aaafc17e000
> nid=0x4cb waiting for monitor entry [0x000000004575f000..0x0000000045760a90]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>        - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 36 on 9000" daemon prio=10 tid=0x00002aaafc17c400
> nid=0x4ca waiting for monitor entry [0x000000004565e000..0x000000004565fa10]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:3281)
>        - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:649)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 35 on 9000" daemon prio=10 tid=0x00002aaafc17a800
> nid=0x4c9 waiting for monitor entry [0x000000004555e000..0x000000004555eb90]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
>        - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
>        at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 34 on 9000" daemon prio=10 tid=0x00002aaafc178c00
> nid=0x4c8 waiting for monitor entry [0x000000004545d000..0x000000004545db10]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
>        - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
>        at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 33 on 9000" daemon prio=10 tid=0x00002aaafc177400
> nid=0x4c7 waiting for monitor entry [0x000000004535c000..0x000000004535cc90]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
>        - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 32 on 9000" daemon prio=10 tid=0x00002aaafc175800
> nid=0x4c6 waiting for monitor entry [0x000000004525a000..0x000000004525bc10]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>        - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 31 on 9000" daemon prio=10 tid=0x00002aaafc173c00
> nid=0x4c5 waiting for monitor entry [0x000000004515a000..0x000000004515ad90]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:3281)
>        - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:649)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 30 on 9000" daemon prio=10 tid=0x00002aaafc172400
> nid=0x4c4 waiting for monitor entry [0x0000000045059000..0x0000000045059d10]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
>        - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 29 on 9000" daemon prio=10 tid=0x00002aaafc171000
> nid=0x4c3 waiting for monitor entry [0x0000000044f57000..0x0000000044f58a90]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
>        - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 28 on 9000" daemon prio=10 tid=0x00002aaafc16f400
> nid=0x4c2 waiting for monitor entry [0x0000000044e56000..0x0000000044e57a10]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>        - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 27 on 9000" daemon prio=10 tid=0x00002aaafc16d800
> nid=0x4c1 waiting for monitor entry [0x0000000044d56000..0x0000000044d56b90]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>        - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 26 on 9000" daemon prio=10 tid=0x00002aaafc16bc00
> nid=0x4c0 waiting for monitor entry [0x0000000044c55000..0x0000000044c55b10]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>        - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 25 on 9000" daemon prio=10 tid=0x00002aaafc16a400
> nid=0x4bf waiting for monitor entry [0x0000000044b54000..0x0000000044b54c90]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
>        - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 24 on 9000" daemon prio=10 tid=0x00002aaafc168800
> nid=0x4be waiting for monitor entry [0x0000000044a53000..0x0000000044a53c10]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
>        - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
>        at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 23 on 9000" daemon prio=10 tid=0x00002aaafc166c00
> nid=0x4bd waiting for monitor entry [0x0000000044952000..0x0000000044952d90]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>        - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 22 on 9000" daemon prio=10 tid=0x00002aaafc165400
> nid=0x4bc waiting for monitor entry [0x0000000044851000..0x0000000044851d10]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>        - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 21 on 9000" daemon prio=10 tid=0x00002aaafc164000
> nid=0x4bb waiting for monitor entry [0x0000000044750000..0x0000000044750a90]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>        - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 20 on 9000" daemon prio=10 tid=0x00002aaafc162400
> nid=0x4ba waiting for monitor entry [0x000000004464f000..0x000000004464fa10]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
>        - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
>        at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 19 on 9000" daemon prio=10 tid=0x00002aaafc488400
> nid=0x4b9 waiting for monitor entry [0x000000004454e000..0x000000004454eb90]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
>        - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
>        at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 18 on 9000" daemon prio=10 tid=0x00002aaafc486800
> nid=0x4b8 waiting for monitor entry [0x000000004444d000..0x000000004444db10]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.nextGenerationStampForBlock(FSNamesystem.java:4548)
>        - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.nextGenerationStamp(NameNode.java:402)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 17 on 9000" daemon prio=10 tid=0x00002aaafc484c00
> nid=0x4b7 waiting for monitor entry [0x000000004434c000..0x000000004434cc90]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.nextGenerationStampForBlock(FSNamesystem.java:4548)
>        - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.nextGenerationStamp(NameNode.java:402)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 16 on 9000" daemon prio=10 tid=0x00002aaafc483000
> nid=0x4b6 waiting for monitor entry [0x000000004424a000..0x000000004424bc10]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>        - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 15 on 9000" daemon prio=10 tid=0x00002aaafc481800
> nid=0x4b5 waiting for monitor entry [0x000000004414a000..0x000000004414ad90]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>        - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 14 on 9000" daemon prio=10 tid=0x00002aaafc47fc00
> nid=0x4b4 waiting for monitor entry [0x0000000044049000..0x0000000044049d10]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
>        - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 13 on 9000" daemon prio=10 tid=0x00002aaafc47e000
> nid=0x4b3 waiting for monitor entry [0x0000000043f48000..0x0000000043f48a90]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
>        - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
>        at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 12 on 9000" daemon prio=10 tid=0x00002aaafc47c400
> nid=0x4b2 waiting for monitor entry [0x0000000043e46000..0x0000000043e47a10]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>        - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 11 on 9000" daemon prio=10 tid=0x00002aaafc47b400
> nid=0x4b1 waiting for monitor entry [0x0000000043d45000..0x0000000043d46b90]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>        - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 10 on 9000" daemon prio=10 tid=0x00002aaafc479800
> nid=0x4b0 waiting for monitor entry [0x0000000043c45000..0x0000000043c45b10]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
>        - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
>        at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 9 on 9000" daemon prio=10 tid=0x00002aaafc477c00
> nid=0x4af waiting for monitor entry [0x0000000043b43000..0x0000000043b44c90]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>        - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 8 on 9000" daemon prio=10 tid=0x00002aaafc476400
> nid=0x4ae waiting for monitor entry [0x0000000043a42000..0x0000000043a43c10]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>        - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 7 on 9000" daemon prio=10 tid=0x00002aaafc474c00
> nid=0x4ad waiting for monitor entry [0x0000000043942000..0x0000000043942d90]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>        - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 6 on 9000" daemon prio=10 tid=0x00002aaafc473000
> nid=0x4ac waiting for monitor entry [0x0000000043841000..0x0000000043841d10]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>        - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 5 on 9000" daemon prio=10 tid=0x00002aaafc471400
> nid=0x4ab waiting for monitor entry [0x0000000043740000..0x0000000043740a90]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>        - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 4 on 9000" daemon prio=10 tid=0x00002aaafc121800
> nid=0x4aa waiting for monitor entry [0x000000004363f000..0x000000004363fa10]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>        - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 3 on 9000" daemon prio=10 tid=0x00002aaafc11fc00
> nid=0x4a9 waiting for monitor entry [0x000000004353d000..0x000000004353eb90]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>        - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 2 on 9000" daemon prio=10 tid=0x00002aaafc11e400
> nid=0x4a8 waiting for monitor entry [0x000000004343c000..0x000000004343db10]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>        - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 1 on 9000" daemon prio=10 tid=0x00002aaafc467800
> nid=0x4a7 waiting for monitor entry [0x000000004333b000..0x000000004333cc90]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>        - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 0 on 9000" daemon prio=10 tid=0x00002aaafc466400
> nid=0x4a6 waiting for monitor entry [0x000000004323b000..0x000000004323bc10]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>        - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server listener on 9000" daemon prio=10 tid=0x00002aaafc12fc00
> nid=0x4a5 waiting on condition [0x000000004313a000..0x000000004313ad90]
>   java.lang.Thread.State: WAITING (parking)
>        at sun.misc.Unsafe.park(Native Method)
>        - parking to wait for  <0x00002aaab3b8fd70> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>        at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
>        at
> java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:254)
>        at
> org.apache.hadoop.ipc.Server$Connection.processData(Server.java:850)
>        at
> org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:814)
>        at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:400)
>        at org.apache.hadoop.ipc.Server$Listener.run(Server.java:309)
>
> "IPC Server Responder" daemon prio=10 tid=0x00002aaafc0bbc00 nid=0x4a4
> runnable [0x0000000043039000..0x0000000043039d10]
>   java.lang.Thread.State: RUNNABLE
>        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>        - locked <0x00002aaab3b90e80> (a sun.nio.ch.Util$1)
>        - locked <0x00002aaab3b90e68> (a
> java.util.Collections$UnmodifiableSet)
>        - locked <0x00002aaab3b90b08> (a sun.nio.ch.EPollSelectorImpl)
>        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>        at org.apache.hadoop.ipc.Server$Responder.run(Server.java:458)
>
> "SocketListener0-1" prio=10 tid=0x00002aaafc0ba400 nid=0x4a3 in
> Object.wait() [0x0000000042f38000..0x0000000042f38a90]
>   java.lang.Thread.State: TIMED_WAITING (on object monitor)
>        at java.lang.Object.wait(Native Method)
>        - waiting on <0x00002aaadf36cb00> (a
> org.mortbay.util.ThreadPool$PoolThread)
>        at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:522)
>        - locked <0x00002aaadf36cb00> (a
> org.mortbay.util.ThreadPool$PoolThread)
>
> "SocketListener0-0" prio=10 tid=0x00002aaafc0b4c00 nid=0x4a2 in
> Object.wait() [0x0000000042e37000..0x0000000042e37a10]
>   java.lang.Thread.State: TIMED_WAITING (on object monitor)
>        at java.lang.Object.wait(Native Method)
>        - waiting on <0x00002aaadf36c820> (a
> org.mortbay.util.ThreadPool$PoolThread)
>        at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:522)
>        - locked <0x00002aaadf36c820> (a
> org.mortbay.util.ThreadPool$PoolThread)
>
> "Acceptor ServerSocket[addr=proxy-1/169.228.130.63,port=0,localport=50070]"
> prio=10 tid=0x00002aaafc0c8400 nid=0x4a1 runnable
> [0x0000000042d36000..0x0000000042d36b90]
>   java.lang.Thread.State: RUNNABLE
>        at java.lang.Object.<init>(Object.java:20)
>        at java.net.SocketImpl.<init>(SocketImpl.java:27)
>        at java.net.PlainSocketImpl.<init>(PlainSocketImpl.java:90)
>        at java.net.SocksSocketImpl.<init>(SocksSocketImpl.java:33)
>        at java.net.Socket.setImpl(Socket.java:434)
>        at java.net.ServerSocket.implAccept(ServerSocket.java:445)
>        at java.net.ServerSocket.accept(ServerSocket.java:421)
>        at
> org.mortbay.util.ThreadedServer.acceptSocket(ThreadedServer.java:432)
>        at
> org.mortbay.util.ThreadedServer$Acceptor.run(ThreadedServer.java:631)
>
> "SessionScavenger" daemon prio=10 tid=0x00002aaafc0c1000 nid=0x4a0 waiting
> on condition [0x0000000042c35000..0x0000000042c35b10]
>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>        at java.lang.Thread.sleep(Native Method)
>        at
> org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(AbstractSessionManager.java:587)
>
> "SessionScavenger" daemon prio=10 tid=0x00002aaafc0fa000 nid=0x49f waiting
> on condition [0x0000000040b2a000..0x0000000040b2ac90]
>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>        at java.lang.Thread.sleep(Native Method)
>        at
> org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(AbstractSessionManager.java:587)
>
> "SessionScavenger" daemon prio=10 tid=0x00002aaafc139800 nid=0x49a waiting
> on condition [0x00000000408b7000..0x00000000408b7c10]
>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>        at java.lang.Thread.sleep(Native Method)
>        at
> org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(AbstractSessionManager.java:587)
>
>
> "org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@1b0deb5f"
> daemon prio=10 tid=0x00002aaafc0a2800 nid=0x496 waiting for monitor entry
> [0x0000000042b34000..0x0000000042b34d90]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:61)
>        - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>        at java.lang.Thread.run(Thread.java:619)
>
>
> "org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@784a7df6"
> daemon prio=10 tid=0x00002aaafc090c00 nid=0x495 waiting for monitor entry
> [0x00000000411f8000..0x00000000411f8d10]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.computeReplicationWork(FSNamesystem.java:2410)
>        - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.computeDatanodeWork(FSNamesystem.java:2376)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2333)
>        at java.lang.Thread.run(Thread.java:619)
>
> "org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@d24111a"
> daemon prio=10 tid=0x00002aaafc013400 nid=0x494 waiting for monitor entry
> [0x00000000410f7000..0x00000000410f7a90]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:349)
>        - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>        at java.lang.Thread.run(Thread.java:619)
>
>
> "org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@1c778255"
> daemon prio=10 tid=0x00002aaafc012c00 nid=0x493 runnable
> [0x00000000413da000..0x00000000413daa10]
>   java.lang.Thread.State: RUNNABLE
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeStoredBlock(FSNamesystem.java:3236)
>        - locked <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeDatanode(FSNamesystem.java:2695)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.heartbeatCheck(FSNamesystem.java:2785)
>        - locked <0x00002aaab3659dd8> (a java.util.TreeMap)
>        - locked <0x00002aaab3653848> (a java.util.ArrayList)
>        - locked <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2312)
>        at java.lang.Thread.run(Thread.java:619)
>
>
> "org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@44676e3f"
> daemon prio=10 tid=0x00002aaafc012800 nid=0x492 waiting on condition
> [0x0000000040d16000..0x0000000040d16b90]
>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>        at java.lang.Thread.sleep(Native Method)
>        at
> org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
>        at java.lang.Thread.run(Thread.java:619)
>
> "Timer thread for monitoring dfs" daemon prio=10 tid=0x00002aaafc03d800
> nid=0x428 waiting for monitor entry [0x0000000040e7e000..0x0000000040e7ed10]
>   java.lang.Thread.State: BLOCKED (on object monitor)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getStats(FSNamesystem.java:3326)
>        - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getCapacityTotal(FSNamesystem.java:3335)
>        at
> org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics.doUpdates(FSNamesystemMetrics.java:90)
>        - locked <0x00002aaab3b3c580> (a
> org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics)
>        at
> org.apache.hadoop.metrics.spi.AbstractMetricsContext.timerEvent(AbstractMetricsContext.java:284)
>        at
> org.apache.hadoop.metrics.spi.AbstractMetricsContext.access$000(AbstractMetricsContext.java:50)
>        at
> org.apache.hadoop.metrics.spi.AbstractMetricsContext$1.run(AbstractMetricsContext.java:249)
>        at java.util.TimerThread.mainLoop(Timer.java:512)
>        at java.util.TimerThread.run(Timer.java:462)
>
> "Low Memory Detector" daemon prio=10 tid=0x000000005c306c00 nid=0x426
> runnable [0x0000000000000000..0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
> "CompilerThread1" daemon prio=10 tid=0x000000005c304000 nid=0x425 waiting
> on condition [0x0000000000000000..0x0000000042830420]
>   java.lang.Thread.State: RUNNABLE
>
> "CompilerThread0" daemon prio=10 tid=0x000000005c300000 nid=0x424 waiting
> on condition [0x0000000000000000..0x000000004272f3a0]
>   java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" daemon prio=10 tid=0x000000005c2fe400 nid=0x423 waiting
> on condition [0x0000000000000000..0x0000000000000000]
>   java.lang.Thread.State: RUNNABLE
>
> "Finalizer" daemon prio=10 tid=0x000000005c2da800 nid=0x422 in
> Object.wait() [0x000000004252e000..0x000000004252ec10]
>   java.lang.Thread.State: WAITING (on object monitor)
>        at java.lang.Object.wait(Native Method)
>        - waiting on <0x00002aaab38ec320> (a
> java.lang.ref.ReferenceQueue$Lock)
>        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
>        - locked <0x00002aaab38ec320> (a java.lang.ref.ReferenceQueue$Lock)
>        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
>        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>
> "Reference Handler" daemon prio=10 tid=0x000000005c2d8c00 nid=0x421 in
> Object.wait() [0x000000004177e000..0x000000004177ed90]
>   java.lang.Thread.State: WAITING (on object monitor)
>        at java.lang.Object.wait(Native Method)
>        - waiting on <0x00002aaab38e9ec0> (a java.lang.ref.Reference$Lock)
>        at java.lang.Object.wait(Object.java:485)
>        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>        - locked <0x00002aaab38e9ec0> (a java.lang.ref.Reference$Lock)
>
> "main" prio=10 tid=0x000000005c268400 nid=0x417 in Object.wait()
> [0x0000000040494000..0x0000000040494ed0]
>   java.lang.Thread.State: WAITING (on object monitor)
>        at java.lang.Object.wait(Native Method)
>        - waiting on <0x00002aaab3b3db50> (a
> org.apache.hadoop.ipc.RPC$Server)
>        at java.lang.Object.wait(Object.java:485)
>        at org.apache.hadoop.ipc.Server.join(Server.java:1023)
>        - locked <0x00002aaab3b3db50> (a org.apache.hadoop.ipc.RPC$Server)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.join(NameNode.java:221)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:870)
>
> "VM Thread" prio=10 tid=0x000000005c2d3400 nid=0x420 runnable
>
> "GC task thread#0 (ParallelGC)" prio=10 tid=0x000000005c273000 nid=0x418
> runnable
>
> "GC task thread#1 (ParallelGC)" prio=10 tid=0x000000005c274800 nid=0x419
> runnable
>
> "GC task thread#2 (ParallelGC)" prio=10 tid=0x000000005c276400 nid=0x41a
> runnable
>
> "GC task thread#3 (ParallelGC)" prio=10 tid=0x000000005c277c00 nid=0x41b
> runnable
>
> "GC task thread#4 (ParallelGC)" prio=10 tid=0x000000005c279400 nid=0x41c
> runnable
>
> "GC task thread#5 (ParallelGC)" prio=10 tid=0x000000005c27b000 nid=0x41d
> runnable
>
> "GC task thread#6 (ParallelGC)" prio=10 tid=0x000000005c27c800 nid=0x41e
> runnable
>
> "GC task thread#7 (ParallelGC)" prio=10 tid=0x000000005c27e000 nid=0x41f
> runnable
>
> "VM Periodic Task Thread" prio=10 tid=0x000000005c308c00 nid=0x427 waiting
> on condition
>
> JNI global references: 984
>
> Heap
>  PSYoungGen      total 203712K, used 107631K [0x00002aaadd0b0000,
> 0x00002aaaf1e00000, 0x00002aaaf1e00000)
>  eden space 124096K, 86% used
> [0x00002aaadd0b0000,0x00002aaae39cbe58,0x00002aaae49e0000)
>  from space 79616K, 0% used
> [0x00002aaae49e0000,0x00002aaae49e0000,0x00002aaae97a0000)
>  to   space 113728K, 0% used
> [0x00002aaaeaef0000,0x00002aaaeaef0000,0x00002aaaf1e00000)
>  PSOldGen        total 682688K, used 682687K [0x00002aaab3600000,
> 0x00002aaadd0b0000, 0x00002aaadd0b0000)
>  object space 682688K, 99% used
> [0x00002aaab3600000,0x00002aaadd0affe8,0x00002aaadd0b0000)
>  PSPermGen       total 21248K, used 16689K [0x00002aaaae200000,
> 0x00002aaaaf6c0000, 0x00002aaab3600000)
>  object space 21248K, 78% used
> [0x00002aaaae200000,0x00002aaaaf24c660,0x00002aaaaf6c0000)
>
>


-- 
Todd Lipcon
Software Engineer, Cloudera