[jira] [Commented] (HBASE-8996) TestHCM.testDeleteForZKConnLeak can go zombie

["stack (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-8996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13713307#comment-13713307 ] 

stack commented on HBASE-8996:
------------------------------

It hung here: http://54.241.6.143/job/HBase-TRUNK/425/consoleFull

Here is the stack trace at end of test.

{code}
There are 1 zombie tests, they should have been killed by surefire but survived
+ echo '************ BEGIN zombies jstack extract'
************ BEGIN zombies jstack extract
++ jps
++ grep surefirebooter
++ cut -d ' ' -f 1
++ xargs -n 1 jstack
++ grep .test
++ grep '\.java'
+ ZB_STACK='	at org.apache.hadoop.hbase.client.TestHCM.testDeleteForZKConnLeak(TestHCM.java:924)'
+ jps
+ grep surefirebooter
+ cut -d ' ' -f 1
+ xargs -n 1 jstack
2013-07-19 01:31:44
Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.14-b01 mixed mode):

"Attach Listener" daemon prio=10 tid=0x00007f187c067000 nid=0x45ce waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"RS_CLOSE_REGION-ip-10-197-73-48:39114-2" prio=10 tid=0x00007f188c0a8000 nid=0x4514 waiting on condition [0x00007f181c483000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf86610> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
	at java.lang.Thread.run(Thread.java:662)

"RS_CLOSE_REGION-ip-10-197-73-48:39114-1" prio=10 tid=0x00007f188c07f000 nid=0x4513 waiting on condition [0x00007f181d08f000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf86610> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
	at java.lang.Thread.run(Thread.java:662)

"RS_CLOSE_REGION-ip-10-197-73-48:56938-2" prio=10 tid=0x00007f187c076000 nid=0x4512 waiting on condition [0x00007f181cf8e000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf8c8e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
	at java.lang.Thread.run(Thread.java:662)

"Shutdownhook:RS:1;ip-10-197-73-48:39114" prio=10 tid=0x00007f189c019800 nid=0x4511 in Object.wait() [0x00007f1812eee000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Thread.join(Thread.java:1186)
	- locked <0x000000008c279de0> (a org.apache.hadoop.hbase.util.JVMClusterUtil$RegionServerThread)
	at org.apache.hadoop.hbase.util.Threads.shutdown(Threads.java:102)
	at org.apache.hadoop.hbase.util.Threads.shutdown(Threads.java:90)
	at org.apache.hadoop.hbase.regionserver.ShutdownHook$ShutdownHookThread.run(ShutdownHook.java:115)

"Shutdownhook:RS:0;ip-10-197-73-48:56938" prio=10 tid=0x00007f189c01d000 nid=0x4510 in Object.wait() [0x00007f1815d1c000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Thread.join(Thread.java:1186)
	- locked <0x000000008c27a3e8> (a org.apache.hadoop.hbase.util.JVMClusterUtil$RegionServerThread)
	at org.apache.hadoop.hbase.util.Threads.shutdown(Threads.java:102)
	at org.apache.hadoop.hbase.util.Threads.shutdown(Threads.java:90)
	at org.apache.hadoop.hbase.regionserver.ShutdownHook$ShutdownHookThread.run(ShutdownHook.java:115)

"SIGTERM handler" daemon prio=10 tid=0x00007f187c0ad800 nid=0x450e in Object.wait() [0x00007f181702f000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Thread.join(Thread.java:1186)
	- locked <0x000000008b4c4450> (a org.apache.hadoop.hbase.regionserver.ShutdownHook$ShutdownHookThread)
	at java.lang.Thread.join(Thread.java:1239)
	at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:79)
	at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:24)
	at java.lang.Shutdown.runHooks(Shutdown.java:79)
	at java.lang.Shutdown.sequence(Shutdown.java:123)
	at java.lang.Shutdown.exit(Shutdown.java:168)
	- locked <0x00000000838b7b08> (a java.lang.Class for java.lang.Shutdown)
	at java.lang.Terminator$1.handle(Terminator.java:35)
	at sun.misc.Signal$1.run(Signal.java:195)
	at java.lang.Thread.run(Thread.java:662)

"pool-1-thread-1-EventThread" daemon prio=10 tid=0x00007f1864afe800 nid=0x368a waiting on condition [0x00007f1813cfc000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008cdf5e58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)

"pool-1-thread-1-SendThread(localhost:56045)" daemon prio=10 tid=0x00007f1864b27800 nid=0x3689 runnable [0x00007f18138f8000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008ce16f40> (a sun.nio.ch.Util$2)
	- locked <0x000000008ce16f30> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008ce16a98> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)

"test-hcm-delete-pool-303-thread-1" daemon prio=10 tid=0x00007f1864b0b800 nid=0x31b4 waiting for monitor entry [0x00007f1817736000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:250)
	- waiting to lock <0x000000008b57de68> (a org.apache.hadoop.hbase.client.HConnectionManager$1)
	at org.apache.hadoop.hbase.client.TestHCM$4.run(TestHCM.java:906)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
	at java.lang.Thread.run(Thread.java:662)

"Old I/O datagram worker ([id: 0xbf41afe4, /226.1.1.3:60100])" daemon prio=10 tid=0x00007f1864301800 nid=0x31ac runnable [0x00007f18131f1000]
   java.lang.Thread.State: RUNNABLE
	at java.net.PlainDatagramSocketImpl.receive0(Native Method)
	- locked <0x000000008ce1ce30> (a java.net.PlainDatagramSocketImpl)
	at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:145)
	- locked <0x000000008ce1ce30> (a java.net.PlainDatagramSocketImpl)
	at java.net.DatagramSocket.receive(DatagramSocket.java:725)
	- locked <0x00000000ded21960> (a java.net.DatagramPacket)
	- locked <0x000000008ce1cdf0> (a java.net.MulticastSocket)
	at org.jboss.netty.channel.socket.oio.OioDatagramWorker.process(OioDatagramWorker.java:47)
	at org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(AbstractOioWorker.java:75)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
	at java.lang.Thread.run(Thread.java:662)

"pool-1-thread-1-EventThread" daemon prio=10 tid=0x00007f18648f4800 nid=0x31a7 waiting on condition [0x00007f18102c2000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008ce1c0d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)

"pool-1-thread-1-SendThread(localhost:56045)" daemon prio=10 tid=0x00007f18649c6000 nid=0x31a6 runnable [0x00007f1816322000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008ce3c088> (a sun.nio.ch.Util$2)
	- locked <0x000000008ce3c078> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008ce3bbe0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)

"Old I/O datagram worker ([id: 0x0d888f4d, /226.1.1.3:60100])" daemon prio=10 tid=0x00007f1864a2b000 nid=0x311e runnable [0x00007f180fcbc000]
   java.lang.Thread.State: RUNNABLE
	at java.net.PlainDatagramSocketImpl.receive0(Native Method)
	- locked <0x000000008ce411b0> (a java.net.PlainDatagramSocketImpl)
	at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:145)
	- locked <0x000000008ce411b0> (a java.net.PlainDatagramSocketImpl)
	at java.net.DatagramSocket.receive(DatagramSocket.java:725)
	- locked <0x00000000dec3a6f8> (a java.net.DatagramPacket)
	- locked <0x000000008ce41170> (a java.net.MulticastSocket)
	at org.jboss.netty.channel.socket.oio.OioDatagramWorker.process(OioDatagramWorker.java:47)
	at org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(AbstractOioWorker.java:75)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
	at java.lang.Thread.run(Thread.java:662)

"pool-1-thread-1-EventThread" daemon prio=10 tid=0x00007f1864a20800 nid=0x311c waiting on condition [0x00007f1812fef000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008ce40438> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)

"pool-1-thread-1-SendThread(localhost:56045)" daemon prio=10 tid=0x00007f18649cb800 nid=0x311b runnable [0x00007f18108c8000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008ce47c58> (a sun.nio.ch.Util$2)
	- locked <0x000000008ce47c48> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008ce477b0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)

"MASTER_CLOSE_REGION-ip-10-197-73-48:38734-2" prio=10 tid=0x00007f1874c0e800 nid=0x17ea waiting on condition [0x00007f1817433000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008c7cc590> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
	at java.lang.Thread.run(Thread.java:662)

"RS_CLOSE_REGION-ip-10-197-73-48:39114-0" prio=10 tid=0x00007f1890051000 nid=0x17e4 waiting on condition [0x00007f1818443000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf86610> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
	at java.lang.Thread.run(Thread.java:662)

"MASTER_CLOSE_REGION-ip-10-197-73-48:38734-1" prio=10 tid=0x00007f1868004800 nid=0x17c8 waiting on condition [0x00007f1813dfd000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008c7cc590> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
	at java.lang.Thread.run(Thread.java:662)

"RS_CLOSE_REGION-ip-10-197-73-48:56938-1" prio=10 tid=0x00007f1878162800 nid=0x173c waiting on condition [0x00007f18119d9000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf8c8e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
	at java.lang.Thread.run(Thread.java:662)

"org.apache.hadoop.hdfs.server.datanode.DataXceiver@5f580757" daemon prio=10 tid=0x00007f18981b4000 nid=0xcca runnable [0x00007f1813bfb000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008bf03f80> (a sun.nio.ch.Util$2)
	- locked <0x000000008bf03f70> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008bf03d28> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:332)
	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
	- locked <0x000000008c256a40> (a java.io.BufferedInputStream)
	at java.io.DataInputStream.readShort(DataInputStream.java:295)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:208)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:104)
	at java.lang.Thread.run(Thread.java:662)

"MASTER_SERVER_OPERATIONS-ip-10-197-73-48:38734-0" prio=10 tid=0x00007f186c020000 nid=0xcaf waiting on condition [0x00007f18111d1000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008c7c7d98> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
	at java.lang.Thread.run(Thread.java:662)

"Old I/O datagram worker ([id: 0xb9966ce7, /226.1.1.3:60100])" daemon prio=10 tid=0x00007f189820c000 nid=0xb2f runnable [0x00007f18116d6000]
   java.lang.Thread.State: RUNNABLE
	at java.net.PlainDatagramSocketImpl.receive0(Native Method)
	- locked <0x000000008c26d868> (a java.net.PlainDatagramSocketImpl)
	at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:145)
	- locked <0x000000008c26d868> (a java.net.PlainDatagramSocketImpl)
	at java.net.DatagramSocket.receive(DatagramSocket.java:725)
	- locked <0x00000000ded94d10> (a java.net.DatagramPacket)
	- locked <0x000000008c26d828> (a java.net.MulticastSocket)
	at org.jboss.netty.channel.socket.oio.OioDatagramWorker.process(OioDatagramWorker.java:47)
	at org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(AbstractOioWorker.java:75)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
	at java.lang.Thread.run(Thread.java:662)

"SplitLogWorker-ip-10-197-73-48.us-west-1.compute.internal,45247,1374195390550-EventThread" daemon prio=10 tid=0x00007f189820a000 nid=0xb2e waiting on condition [0x00007f18117d7000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008c26c978> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)

"SplitLogWorker-ip-10-197-73-48.us-west-1.compute.internal,45247,1374195390550-SendThread(localhost:56045)" daemon prio=10 tid=0x00007f1898208000 nid=0xb2d runnable [0x00007f18118d8000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008c270300> (a sun.nio.ch.Util$2)
	- locked <0x000000008c2702f0> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008c26fe58> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)

"Replication.RpcServer.handler=2,port=45247" daemon prio=10 tid=0x00007f18981e9000 nid=0xb2b waiting on condition [0x00007f1811ada000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008be6e0e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"Replication.RpcServer.handler=1,port=45247" daemon prio=10 tid=0x00007f18981e7000 nid=0xb2a waiting on condition [0x00007f1811bdb000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008be6e0e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"Replication.RpcServer.handler=0,port=45247" daemon prio=10 tid=0x00007f18981e4800 nid=0xb29 waiting on condition [0x00007f1811cdc000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008be6e0e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"Priority.RpcServer.handler=9,port=45247" daemon prio=10 tid=0x00007f18981e2800 nid=0xb28 waiting on condition [0x00007f1811ddd000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008be6c890> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"Priority.RpcServer.handler=8,port=45247" daemon prio=10 tid=0x00007f18981e0800 nid=0xb27 waiting on condition [0x00007f1811ede000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008be6c890> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"Priority.RpcServer.handler=7,port=45247" daemon prio=10 tid=0x00007f18981de000 nid=0xb26 waiting on condition [0x00007f1811fdf000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008be6c890> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"Priority.RpcServer.handler=6,port=45247" daemon prio=10 tid=0x00007f18981db800 nid=0xb25 waiting on condition [0x00007f18120e0000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008be6c890> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"Priority.RpcServer.handler=5,port=45247" daemon prio=10 tid=0x00007f18981d9800 nid=0xb24 waiting on condition [0x00007f18121e1000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008be6c890> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"Priority.RpcServer.handler=4,port=45247" daemon prio=10 tid=0x00007f18981d7800 nid=0xb23 waiting on condition [0x00007f18122e2000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008be6c890> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"Priority.RpcServer.handler=3,port=45247" daemon prio=10 tid=0x00007f18981d5800 nid=0xb22 waiting on condition [0x00007f18123e3000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008be6c890> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"Priority.RpcServer.handler=2,port=45247" daemon prio=10 tid=0x00007f18981d2800 nid=0xb21 waiting on condition [0x00007f18124e4000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008be6c890> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"Priority.RpcServer.handler=1,port=45247" daemon prio=10 tid=0x00007f18981d0000 nid=0xb20 waiting on condition [0x00007f18125e5000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008be6c890> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"Priority.RpcServer.handler=0,port=45247" daemon prio=10 tid=0x00007f18981ce000 nid=0xb1f waiting on condition [0x00007f18126e6000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008be6c890> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"RpcServer.handler=4,port=45247" daemon prio=10 tid=0x00007f18981cc000 nid=0xb1e waiting on condition [0x00007f18127e7000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008be6ace0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"RpcServer.handler=3,port=45247" daemon prio=10 tid=0x00007f18981c9800 nid=0xb1d waiting on condition [0x00007f18128e8000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008be6ace0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"RpcServer.handler=2,port=45247" daemon prio=10 tid=0x00007f18981c7800 nid=0xb1c waiting on condition [0x00007f18129e9000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008be6ace0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"RpcServer.handler=1,port=45247" daemon prio=10 tid=0x00007f18981c5000 nid=0xb1b waiting on condition [0x00007f1812aea000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008be6ace0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"RpcServer.handler=0,port=45247" daemon prio=10 tid=0x00007f18981c3000 nid=0xb1a waiting on condition [0x00007f1812beb000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008be6ace0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"Old I/O datagram worker ([id: 0x261745f8, /226.1.1.3:60100])" daemon prio=10 tid=0x00007f186492e000 nid=0xaf9 runnable [0x00007f1814a09000]
   java.lang.Thread.State: RUNNABLE
	at java.net.PlainDatagramSocketImpl.receive0(Native Method)
	- locked <0x000000008c2722a8> (a java.net.PlainDatagramSocketImpl)
	at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:145)
	- locked <0x000000008c2722a8> (a java.net.PlainDatagramSocketImpl)
	at java.net.DatagramSocket.receive(DatagramSocket.java:725)
	- locked <0x00000000debc9890> (a java.net.DatagramPacket)
	- locked <0x000000008c272268> (a java.net.MulticastSocket)
	at org.jboss.netty.channel.socket.oio.OioDatagramWorker.process(OioDatagramWorker.java:47)
	at org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(AbstractOioWorker.java:75)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
	at java.lang.Thread.run(Thread.java:662)

"pool-1-thread-1-EventThread" daemon prio=10 tid=0x00007f18648e3000 nid=0xaf2 waiting on condition [0x00007f1814908000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008c2713d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)

"pool-1-thread-1-SendThread(localhost:56045)" daemon prio=10 tid=0x00007f18648e2800 nid=0xaf1 runnable [0x00007f1814807000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008c274d58> (a sun.nio.ch.Util$2)
	- locked <0x000000008c274d48> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008c2748b0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)

"Old I/O datagram worker ([id: 0x9c4ab61d, /226.1.1.3:60100])" daemon prio=10 tid=0x00007f18648f8000 nid=0xaef runnable [0x00007f1814b0a000]
   java.lang.Thread.State: RUNNABLE
	at java.net.PlainDatagramSocketImpl.receive0(Native Method)
	- locked <0x000000008c276968> (a java.net.PlainDatagramSocketImpl)
	at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:145)
	- locked <0x000000008c276968> (a java.net.PlainDatagramSocketImpl)
	at java.net.DatagramSocket.receive(DatagramSocket.java:725)
	- locked <0x00000000decaee18> (a java.net.DatagramPacket)
	- locked <0x000000008c276928> (a java.net.MulticastSocket)
	at org.jboss.netty.channel.socket.oio.OioDatagramWorker.process(OioDatagramWorker.java:47)
	at org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(AbstractOioWorker.java:75)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
	at java.lang.Thread.run(Thread.java:662)

"pool-1-thread-1-EventThread" daemon prio=10 tid=0x00007f18648f6800 nid=0xaee waiting on condition [0x00007f1814c0b000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008c275a90> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)

"pool-1-thread-1-SendThread(localhost:56045)" daemon prio=10 tid=0x00007f18648f9000 nid=0xaed runnable [0x00007f1817332000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008c279438> (a sun.nio.ch.Util$2)
	- locked <0x000000008c279428> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008c278f80> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)

"PacketResponder 1 for Block blk_5897154169758322145_1036" daemon prio=10 tid=0x00007f189809e000 nid=0xa6b in Object.wait() [0x00007f1814e0d000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:843)
	- locked <0x000000008be9d918> (a org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder)
	at java.lang.Thread.run(Thread.java:662)

"ResponseProcessor for block blk_5897154169758322145_1036" daemon prio=10 tid=0x00007f186001f800 nid=0xa6a runnable [0x00007f1814f0e000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008c55a778> (a sun.nio.ch.Util$2)
	- locked <0x000000008c55a768> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008c552508> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:332)
	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
	at java.io.DataInputStream.readFully(DataInputStream.java:178)
	at java.io.DataInputStream.readLong(DataInputStream.java:399)
	at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:124)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:3127)

"PacketResponder 0 for Block blk_5897154169758322145_1036" daemon prio=10 tid=0x00007f18980a3000 nid=0xa68 in Object.wait() [0x00007f1815211000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:843)
	- locked <0x000000008bf054d0> (a org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder)
	at java.lang.Thread.run(Thread.java:662)

"org.apache.hadoop.hdfs.server.datanode.DataXceiver@2342d884" daemon prio=10 tid=0x00007f18980a1800 nid=0xa67 runnable [0x00007f181510f000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008bf03480> (a sun.nio.ch.Util$2)
	- locked <0x000000008bf03470> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008bf03228> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:332)
	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
	at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
	- locked <0x000000008bf05660> (a java.io.BufferedInputStream)
	at java.io.DataInputStream.read(DataInputStream.java:132)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:292)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:339)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:403)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:581)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:406)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:112)
	at java.lang.Thread.run(Thread.java:662)

"org.apache.hadoop.hdfs.server.datanode.DataXceiver@2ef7d41f" daemon prio=10 tid=0x00007f1860021800 nid=0xa66 runnable [0x00007f181500f000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008d949be0> (a sun.nio.ch.Util$2)
	- locked <0x000000008d949bd0> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008d949bf0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:332)
	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
	at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
	- locked <0x000000008be9de48> (a java.io.BufferedInputStream)
	at java.io.DataInputStream.read(DataInputStream.java:132)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:292)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:339)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:403)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:581)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:406)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:112)
	at java.lang.Thread.run(Thread.java:662)

"MASTER_CLOSE_REGION-ip-10-197-73-48:38734-0" prio=10 tid=0x00007f1834022800 nid=0xa40 waiting on condition [0x00007f1815413000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008c7cc590> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
	at java.lang.Thread.run(Thread.java:662)

"RS_CLOSE_REGION-ip-10-197-73-48:56938-0" prio=10 tid=0x00007f186c0b0000 nid=0xa32 waiting on condition [0x00007f1815c1b000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf8c8e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
	at java.lang.Thread.run(Thread.java:662)

"PacketResponder 1 for Block blk_-8513724430260139692_1035" daemon prio=10 tid=0x00007f189807f000 nid=0xa31 in Object.wait() [0x00007f1815716000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:843)
	- locked <0x000000008beaf338> (a org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder)
	at java.lang.Thread.run(Thread.java:662)

"ResponseProcessor for block blk_-8513724430260139692_1035" daemon prio=10 tid=0x00007f1858039000 nid=0xa30 runnable [0x00007f1815a19000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008c6bdc88> (a sun.nio.ch.Util$2)
	- locked <0x000000008c6bdc78> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008c6bda50> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:332)
	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
	at java.io.DataInputStream.readFully(DataInputStream.java:178)
	at java.io.DataInputStream.readLong(DataInputStream.java:399)
	at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:124)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:3127)

"PacketResponder 0 for Block blk_-8513724430260139692_1035" daemon prio=10 tid=0x00007f1898088000 nid=0xa2f in Object.wait() [0x00007f1815918000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:843)
	- locked <0x000000008bf16798> (a org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder)
	at java.lang.Thread.run(Thread.java:662)

"org.apache.hadoop.hdfs.server.datanode.DataXceiver@6112c9f" daemon prio=10 tid=0x00007f1898055000 nid=0xa2e runnable [0x00007f1815f1e000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x00000000db8fac40> (a sun.nio.ch.Util$2)
	- locked <0x00000000db8fac30> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000000db8faa28> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:332)
	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
	at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
	- locked <0x000000008bf16928> (a java.io.BufferedInputStream)
	at java.io.DataInputStream.read(DataInputStream.java:132)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:292)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:339)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:403)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:581)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:406)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:112)
	at java.lang.Thread.run(Thread.java:662)

"org.apache.hadoop.hdfs.server.datanode.DataXceiver@5c3f3b9b" daemon prio=10 tid=0x00007f1860017000 nid=0xa2d runnable [0x00007f1815312000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x00000000f19cf080> (a sun.nio.ch.Util$2)
	- locked <0x00000000f19cf070> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000000f19cee48> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:332)
	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
	at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
	- locked <0x000000008beaf868> (a java.io.BufferedInputStream)
	at java.io.DataInputStream.read(DataInputStream.java:132)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:292)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:339)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:403)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:581)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:406)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:112)
	at java.lang.Thread.run(Thread.java:662)

"RS_OPEN_REGION-ip-10-197-73-48:56938-2" prio=10 tid=0x00007f1874b75000 nid=0x91f waiting on condition [0x00007f1816827000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf87a80> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
	at java.lang.Thread.run(Thread.java:662)

"MASTER_OPEN_REGION-ip-10-197-73-48:38734-4" prio=10 tid=0x00007f183401f000 nid=0x90c waiting on condition [0x00007f1815514000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008c7dcbf8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
	at java.lang.Thread.run(Thread.java:662)

"MASTER_OPEN_REGION-ip-10-197-73-48:38734-3" prio=10 tid=0x00007f1868002000 nid=0x900 waiting on condition [0x00007f1815b1a000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008c7dcbf8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
	at java.lang.Thread.run(Thread.java:662)

"MASTER_OPEN_REGION-ip-10-197-73-48:38734-2" prio=10 tid=0x00007f1840027800 nid=0x8f0 waiting on condition [0x00007f1816524000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008c7dcbf8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
	at java.lang.Thread.run(Thread.java:662)

"RS_OPEN_REGION-ip-10-197-73-48:39114-2" prio=10 tid=0x00007f18a000a800 nid=0x8e8 waiting on condition [0x00007f1816120000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf83f58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
	at java.lang.Thread.run(Thread.java:662)

"RS_OPEN_REGION-ip-10-197-73-48:56938-1" prio=10 tid=0x00007f186c060000 nid=0x8db waiting on condition [0x00007f1816726000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf87a80> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
	at java.lang.Thread.run(Thread.java:662)

"RS_OPEN_REGION-ip-10-197-73-48:56938-0" prio=10 tid=0x00007f18780e7800 nid=0x8d5 waiting on condition [0x00007f1816e2d000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf87a80> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
	at java.lang.Thread.run(Thread.java:662)

"RS_OPEN_REGION-ip-10-197-73-48:39114-1" prio=10 tid=0x00007f189001d000 nid=0x8ca waiting on condition [0x00007f1817130000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf83f58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
	at java.lang.Thread.run(Thread.java:662)

"MASTER_OPEN_REGION-ip-10-197-73-48:38734-1" prio=10 tid=0x00007f184c008800 nid=0x892 waiting on condition [0x00007f1817837000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008c7dcbf8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
	at java.lang.Thread.run(Thread.java:662)

"RS_OPEN_REGION-ip-10-197-73-48:39114-0" prio=10 tid=0x00007f189804e000 nid=0x87a waiting on condition [0x00007f1817a39000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf83f58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
	at java.lang.Thread.run(Thread.java:662)

"PacketResponder 1 for Block blk_5254598857189966085_1010" daemon prio=10 tid=0x00007f183c028000 nid=0x86a in Object.wait() [0x00007f181803f000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:843)
	- locked <0x000000008beb0d80> (a org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder)
	at java.lang.Thread.run(Thread.java:662)

"ResponseProcessor for block blk_5254598857189966085_1010" daemon prio=10 tid=0x00007f18648f4000 nid=0x869 runnable [0x00007f1817c3b000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008ce4ce70> (a sun.nio.ch.Util$2)
	- locked <0x000000008ce4ce60> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008ce4cc48> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:332)
	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
	at java.io.DataInputStream.readFully(DataInputStream.java:178)
	at java.io.DataInputStream.readLong(DataInputStream.java:399)
	at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:124)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:3127)

"PacketResponder 0 for Block blk_5254598857189966085_1010" daemon prio=10 tid=0x00007f1840015800 nid=0x868 in Object.wait() [0x00007f1817b3a000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:843)
	- locked <0x000000008bf17aa0> (a org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder)
	at java.lang.Thread.run(Thread.java:662)

"org.apache.hadoop.hdfs.server.datanode.DataXceiver@129719f4" daemon prio=10 tid=0x00007f1898017800 nid=0x867 runnable [0x00007f1817e3d000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008da5e9b0> (a sun.nio.ch.Util$2)
	- locked <0x000000008da5e9a0> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008da5e778> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:332)
	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
	at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
	- locked <0x000000008bf17c08> (a java.io.BufferedInputStream)
	at java.io.DataInputStream.read(DataInputStream.java:132)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:292)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:339)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:403)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:581)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:406)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:112)
	at java.lang.Thread.run(Thread.java:662)

"org.apache.hadoop.hdfs.server.datanode.DataXceiver@7d91275b" daemon prio=10 tid=0x00007f1860013800 nid=0x866 runnable [0x00007f1817d3c000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008ccc9fa0> (a sun.nio.ch.Util$2)
	- locked <0x000000008ccc9f90> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008ccc9d68> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:332)
	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
	at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
	- locked <0x000000008bec12e8> (a java.io.BufferedInputStream)
	at java.io.DataInputStream.read(DataInputStream.java:132)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:292)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:339)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:403)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:581)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:406)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:112)
	at java.lang.Thread.run(Thread.java:662)

"MASTER_TABLE_OPERATIONS-ip-10-197-73-48:38734-0" prio=10 tid=0x00007f18940aa000 nid=0x82e waiting on condition [0x00007f1818948000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008c7caa98> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
	at java.lang.Thread.run(Thread.java:662)

"Old I/O datagram worker ([id: 0x14e60bee, /226.1.1.3:60100])" daemon prio=10 tid=0x00007f18648ea000 nid=0x81c runnable [0x00007f1818544000]
   java.lang.Thread.State: RUNNABLE
	at java.net.PlainDatagramSocketImpl.receive0(Native Method)
	- locked <0x000000008c761288> (a java.net.PlainDatagramSocketImpl)
	at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:145)
	- locked <0x000000008c761288> (a java.net.PlainDatagramSocketImpl)
	at java.net.DatagramSocket.receive(DatagramSocket.java:725)
	- locked <0x00000000dee09e40> (a java.net.DatagramPacket)
	- locked <0x000000008c761248> (a java.net.MulticastSocket)
	at org.jboss.netty.channel.socket.oio.OioDatagramWorker.process(OioDatagramWorker.java:47)
	at org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(AbstractOioWorker.java:75)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
	at java.lang.Thread.run(Thread.java:662)

"pool-1-thread-1-EventThread" daemon prio=10 tid=0x00007f18648e4800 nid=0x818 waiting on condition [0x00007f1818847000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bd703e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)

"pool-1-thread-1-SendThread(localhost:56045)" daemon prio=10 tid=0x00007f18648e6000 nid=0x817 runnable [0x00007f1818746000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008bd6fe90> (a sun.nio.ch.Util$2)
	- locked <0x000000008bd6fe80> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008bd6f9e8> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)

"CatalogJanitor-ip-10-197-73-48:38734" daemon prio=10 tid=0x00007f186c0bb800 nid=0x7f7 waiting for monitor entry [0x00007f1818a49000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:250)
	- waiting to lock <0x000000008b57de68> (a org.apache.hadoop.hbase.client.HConnectionManager$1)
	at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:173)
	at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:127)
	at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:102)
	at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:80)
	at org.apache.hadoop.hbase.master.CatalogJanitor.getMergedRegionsAndSplitParents(CatalogJanitor.java:165)
	at org.apache.hadoop.hbase.master.CatalogJanitor.getMergedRegionsAndSplitParents(CatalogJanitor.java:117)
	at org.apache.hadoop.hbase.master.CatalogJanitor.scan(CatalogJanitor.java:219)
	at org.apache.hadoop.hbase.master.CatalogJanitor.chore(CatalogJanitor.java:99)
	at org.apache.hadoop.hbase.Chore.run(Chore.java:80)
	at java.lang.Thread.run(Thread.java:662)

"ip-10-197-73-48.us-west-1.compute.internal,38734,1374195379749-BalancerChore" daemon prio=10 tid=0x00007f186c0b9800 nid=0x7f6 in Object.wait() [0x00007f1818b4a000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:94)
	- locked <0x000000008c7ea470> (a java.lang.Object)
	at org.apache.hadoop.hbase.Chore.run(Chore.java:88)
	at java.lang.Thread.run(Thread.java:662)

"ip-10-197-73-48.us-west-1.compute.internal,38734,1374195379749-ClusterStatusChore" daemon prio=10 tid=0x00007f186c0b8000 nid=0x7f5 in Object.wait() [0x00007f1818c4b000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:94)
	- locked <0x000000008c7ea728> (a java.lang.Object)
	at org.apache.hadoop.hbase.Chore.run(Chore.java:88)
	at java.lang.Thread.run(Thread.java:662)

"MASTER_OPEN_REGION-ip-10-197-73-48:38734-0" prio=10 tid=0x00007f186c0b1000 nid=0x7f0 waiting on condition [0x00007f181904f000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008c7dcbf8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
	at java.lang.Thread.run(Thread.java:662)

"RS_OPEN_META-ip-10-197-73-48:39114-0.logSyncer" daemon prio=10 tid=0x00007f186000f000 nid=0x7e4 in Object.wait() [0x00007f1819251000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.hadoop.hbase.regionserver.wal.FSHLog$LogSyncer.run(FSHLog.java:951)
	- locked <0x000000008c8134c8> (a java.util.concurrent.atomic.AtomicBoolean)
	at java.lang.Thread.run(Thread.java:662)

"DataStreamer for file /user/ec2-user/hbase/.logs/ip-10-197-73-48.us-west-1.compute.internal,39114,1374195381104/ip-10-197-73-48.us-west-1.compute.internal%2C39114%2C1374195381104.1374195384757.meta block blk_5254598857189966085_1010" daemon prio=10 tid=0x00007f186000c000 nid=0x7e3 in Object.wait() [0x00007f1819352000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2966)
	- locked <0x000000008c7a00f0> (a java.util.LinkedList)

"RS_OPEN_META-ip-10-197-73-48:39114-0" prio=10 tid=0x00007f189001a800 nid=0x7d9 waiting on condition [0x00007f1819655000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf85ab0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
	at java.lang.Thread.run(Thread.java:662)

"Old I/O datagram worker ([id: 0xa29f9b65, /226.1.1.3:60100])" daemon prio=10 tid=0x00007f1844026000 nid=0x784 runnable [0x00007f1819857000]
   java.lang.Thread.State: RUNNABLE
	at java.net.PlainDatagramSocketImpl.receive0(Native Method)
	- locked <0x000000008c7a4ba8> (a java.net.PlainDatagramSocketImpl)
	at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:145)
	- locked <0x000000008c7a4ba8> (a java.net.PlainDatagramSocketImpl)
	at java.net.DatagramSocket.receive(DatagramSocket.java:725)
	- locked <0x00000000deef0360> (a java.net.DatagramPacket)
	- locked <0x000000008c7a5608> (a java.net.MulticastSocket)
	at org.jboss.netty.channel.socket.oio.OioDatagramWorker.process(OioDatagramWorker.java:47)
	at org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(AbstractOioWorker.java:75)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
	at java.lang.Thread.run(Thread.java:662)

"SplitLogWorker-ip-10-197-73-48.us-west-1.compute.internal,56938,1374195380953-EventThread" daemon prio=10 tid=0x00007f1844006800 nid=0x782 waiting on condition [0x00007f1819958000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008c7a83b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)

"SplitLogWorker-ip-10-197-73-48.us-west-1.compute.internal,56938,1374195380953-SendThread(localhost:56045)" daemon prio=10 tid=0x00007f1844002000 nid=0x781 runnable [0x00007f1819a59000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008c7a7df0> (a sun.nio.ch.Util$2)
	- locked <0x000000008c7a7de0> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008c7a7958> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)

"Old I/O datagram worker ([id: 0x633f8407, /226.1.1.3:60100])" daemon prio=10 tid=0x00007f1898013000 nid=0x77f runnable [0x00007f1819c5b000]
   java.lang.Thread.State: RUNNABLE
	at java.net.PlainDatagramSocketImpl.receive0(Native Method)
	- locked <0x000000008c7a5188> (a java.net.PlainDatagramSocketImpl)
	at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:145)
	- locked <0x000000008c7a5188> (a java.net.PlainDatagramSocketImpl)
	at java.net.DatagramSocket.receive(DatagramSocket.java:725)
	- locked <0x00000000dee7dbb0> (a java.net.DatagramPacket)
	- locked <0x000000008bf90f10> (a java.net.MulticastSocket)
	at org.jboss.netty.channel.socket.oio.OioDatagramWorker.process(OioDatagramWorker.java:47)
	at org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(AbstractOioWorker.java:75)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
	at java.lang.Thread.run(Thread.java:662)

"SplitLogWorker-ip-10-197-73-48.us-west-1.compute.internal,39114,1374195381104-EventThread" daemon prio=10 tid=0x00007f1898011000 nid=0x77e waiting on condition [0x00007f1819d5c000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf93d88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)

"SplitLogWorker-ip-10-197-73-48.us-west-1.compute.internal,39114,1374195381104-SendThread(localhost:56045)" daemon prio=10 tid=0x00007f1898008800 nid=0x77d runnable [0x00007f1819e5d000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008bf93788> (a sun.nio.ch.Util$2)
	- locked <0x000000008bf93778> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008bf932e0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)

"Replication.RpcServer.handler=2,port=56938" daemon prio=10 tid=0x00007f187c0a0000 nid=0x77c waiting on condition [0x00007f1819f5e000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf6ebc0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"Replication.RpcServer.handler=1,port=56938" daemon prio=10 tid=0x00007f187c09e000 nid=0x77b waiting on condition [0x00007f181a05f000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf6ebc0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"Replication.RpcServer.handler=0,port=56938" daemon prio=10 tid=0x00007f187c09c000 nid=0x77a waiting on condition [0x00007f181a160000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf6ebc0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"Priority.RpcServer.handler=9,port=56938" daemon prio=10 tid=0x00007f187c09a000 nid=0x779 waiting on condition [0x00007f181a261000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf6d600> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"Priority.RpcServer.handler=8,port=56938" daemon prio=10 tid=0x00007f187c098000 nid=0x778 waiting on condition [0x00007f181a362000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf6d600> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"Priority.RpcServer.handler=7,port=56938" daemon prio=10 tid=0x00007f187c096000 nid=0x777 waiting on condition [0x00007f181a463000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf6d600> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"Priority.RpcServer.handler=6,port=56938" daemon prio=10 tid=0x00007f187c094000 nid=0x776 waiting on condition [0x00007f181a564000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf6d600> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"Priority.RpcServer.handler=5,port=56938" daemon prio=10 tid=0x00007f187c092000 nid=0x775 waiting on condition [0x00007f181a665000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf6d600> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"Priority.RpcServer.handler=4,port=56938" daemon prio=10 tid=0x00007f187c090000 nid=0x774 waiting on condition [0x00007f181a766000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf6d600> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"Priority.RpcServer.handler=3,port=56938" daemon prio=10 tid=0x00007f187c08e000 nid=0x771 waiting on condition [0x00007f181a968000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf6d600> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"Priority.RpcServer.handler=2,port=56938" daemon prio=10 tid=0x00007f187c08c000 nid=0x770 waiting on condition [0x00007f181aa69000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf6d600> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"Priority.RpcServer.handler=1,port=56938" daemon prio=10 tid=0x00007f187c08a000 nid=0x76f waiting on condition [0x00007f181ab6a000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf6d600> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"Priority.RpcServer.handler=0,port=56938" daemon prio=10 tid=0x00007f187c088800 nid=0x76e waiting on condition [0x00007f181ac6b000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf6d600> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"RpcServer.handler=4,port=56938" daemon prio=10 tid=0x00007f187c087000 nid=0x76d waiting on condition [0x00007f181ad6c000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf6aa38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"RpcServer.handler=3,port=56938" daemon prio=10 tid=0x00007f187c085000 nid=0x76c waiting on condition [0x00007f181ae6d000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf6aa38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"RpcServer.handler=2,port=56938" daemon prio=10 tid=0x00007f187c083000 nid=0x76b waiting on condition [0x00007f181af6e000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf6aa38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"RpcServer.handler=1,port=56938" daemon prio=10 tid=0x00007f187c081800 nid=0x769 waiting on condition [0x00007f181b06f000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf6aa38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"RpcServer.handler=0,port=56938" daemon prio=10 tid=0x00007f187c080000 nid=0x768 waiting on condition [0x00007f181b170000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf6aa38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"Replication.RpcServer.handler=2,port=39114" daemon prio=10 tid=0x00007f188c0a5800 nid=0x762 waiting on condition [0x00007f181b574000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf7f660> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"Replication.RpcServer.handler=1,port=39114" daemon prio=10 tid=0x00007f188c0a3800 nid=0x75e waiting on condition [0x00007f181b978000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf7f660> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"Replication.RpcServer.handler=0,port=39114" daemon prio=10 tid=0x00007f188c0a1800 nid=0x75d waiting on condition [0x00007f181ba79000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf7f660> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"Priority.RpcServer.handler=9,port=39114" daemon prio=10 tid=0x00007f188c09f800 nid=0x75c waiting on condition [0x00007f181bb7a000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf7a068> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"Priority.RpcServer.handler=8,port=39114" daemon prio=10 tid=0x00007f188c09d800 nid=0x75b waiting on condition [0x00007f181bc7b000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf7a068> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"Priority.RpcServer.handler=7,port=39114" daemon prio=10 tid=0x00007f188c09b800 nid=0x75a waiting on condition [0x00007f181bd7c000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf7a068> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"Priority.RpcServer.handler=6,port=39114" daemon prio=10 tid=0x00007f188c099800 nid=0x759 waiting on condition [0x00007f181be7d000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf7a068> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"Priority.RpcServer.handler=5,port=39114" daemon prio=10 tid=0x00007f188c097800 nid=0x758 waiting on condition [0x00007f181bf7e000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf7a068> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"Priority.RpcServer.handler=4,port=39114" daemon prio=10 tid=0x00007f188c095800 nid=0x756 waiting on condition [0x00007f181c07f000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf7a068> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"Priority.RpcServer.handler=3,port=39114" daemon prio=10 tid=0x00007f188c093800 nid=0x755 waiting on condition [0x00007f181c180000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf7a068> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"Priority.RpcServer.handler=2,port=39114" daemon prio=10 tid=0x00007f188c091800 nid=0x754 waiting on condition [0x00007f181c281000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf7a068> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"Priority.RpcServer.handler=1,port=39114" daemon prio=10 tid=0x00007f188c08f800 nid=0x753 waiting on condition [0x00007f181c382000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf7a068> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"Priority.RpcServer.handler=0,port=39114" daemon prio=10 tid=0x00007f188c08d800 nid=0x751 waiting on condition [0x00007f181c584000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf7a068> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"RpcServer.handler=4,port=39114" daemon prio=10 tid=0x00007f188c08b800 nid=0x750 waiting on condition [0x00007f181c685000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf77ef0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"RpcServer.handler=3,port=39114" daemon prio=10 tid=0x00007f188c089800 nid=0x74f waiting on condition [0x00007f181c786000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf77ef0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"RpcServer.handler=2,port=39114" daemon prio=10 tid=0x00007f188c087000 nid=0x74e waiting on condition [0x00007f181c887000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf77ef0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"RpcServer.handler=1,port=39114" daemon prio=10 tid=0x00007f188c085000 nid=0x74d waiting on condition [0x00007f181c988000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf77ef0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"RpcServer.handler=0,port=39114" daemon prio=10 tid=0x00007f188c083000 nid=0x74b waiting on condition [0x00007f181ca89000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf77ef0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"HBase-Metrics2-1" daemon prio=10 tid=0x00007f187c061000 nid=0x743 waiting on condition [0x00007f181d291000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008c6a6b08> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
	at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
	at java.lang.Thread.run(Thread.java:662)

"RS:1;ip-10-197-73-48:39114.logSyncer" daemon prio=10 tid=0x00007f188c073000 nid=0x742 in Object.wait() [0x00007f181d392000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.hadoop.hbase.regionserver.wal.FSHLog$LogSyncer.run(FSHLog.java:951)
	- locked <0x000000008cc2e520> (a java.util.concurrent.atomic.AtomicBoolean)
	at java.lang.Thread.run(Thread.java:662)

"RS:0;ip-10-197-73-48:56938.logSyncer" daemon prio=10 tid=0x00007f187c060800 nid=0x741 in Object.wait() [0x00007f181d493000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.hadoop.hbase.regionserver.wal.FSHLog$LogSyncer.run(FSHLog.java:951)
	- locked <0x000000008bfeaac8> (a java.util.concurrent.atomic.AtomicBoolean)
	at java.lang.Thread.run(Thread.java:662)

"org.apache.hadoop.hdfs.LeaseRenewer$1@129a41d2" daemon prio=10 tid=0x00007f188c04d800 nid=0x73f waiting on condition [0x00007f181d594000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:397)
	at org.apache.hadoop.hdfs.LeaseRenewer.access$600(LeaseRenewer.java:69)
	at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:273)
	at java.lang.Thread.run(Thread.java:662)

"DataStreamer for file /user/ec2-user/hbase/.logs/ip-10-197-73-48.us-west-1.compute.internal,39114,1374195381104/ip-10-197-73-48.us-west-1.compute.internal%2C39114%2C1374195381104.1374195383640 block blk_5897154169758322145_1036" daemon prio=10 tid=0x00007f188c069000 nid=0x73e in Object.wait() [0x00007f181d695000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2966)
	- locked <0x000000008c7a3a70> (a java.util.LinkedList)

"org.apache.hadoop.hdfs.LeaseRenewer$1@5c335297" daemon prio=10 tid=0x00007f187c05f800 nid=0x73d waiting on condition [0x00007f181d796000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:397)
	at org.apache.hadoop.hdfs.LeaseRenewer.access$600(LeaseRenewer.java:69)
	at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:273)
	at java.lang.Thread.run(Thread.java:662)

"DataStreamer for file /user/ec2-user/hbase/.logs/ip-10-197-73-48.us-west-1.compute.internal,56938,1374195380953/ip-10-197-73-48.us-west-1.compute.internal%2C56938%2C1374195380953.1374195383640 block blk_-8513724430260139692_1035" daemon prio=10 tid=0x00007f187c04b800 nid=0x73c in Object.wait() [0x00007f181d897000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2966)
	- locked <0x000000008c6bde70> (a java.util.LinkedList)

"M:0;ip-10-197-73-48:38734.archivedHFileCleaner" daemon prio=10 tid=0x00007f186c0a7000 nid=0x6f1 in Object.wait() [0x00007f181dd9c000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:94)
	- locked <0x000000008c7f0d68> (a java.lang.Object)
	at org.apache.hadoop.hbase.Chore.run(Chore.java:88)
	at java.lang.Thread.run(Thread.java:662)

"snapshot-hfile-cleaner-cache-refresher" daemon prio=10 tid=0x00007f186c08d000 nid=0x6f0 in Object.wait() [0x00007f181de9d000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.util.TimerThread.mainLoop(Timer.java:509)
	- locked <0x000000008c7f12b8> (a java.util.TaskQueue)
	at java.util.TimerThread.run(Timer.java:462)

"M:0;ip-10-197-73-48:38734.oldLogCleaner" daemon prio=10 tid=0x00007f186c08c800 nid=0x6ee in Object.wait() [0x00007f181df9e000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:94)
	- locked <0x000000008c7ec258> (a java.lang.Object)
	at org.apache.hadoop.hbase.Chore.run(Chore.java:88)
	at java.lang.Thread.run(Thread.java:662)

"snapshot-log-cleaner-cache-refresher" daemon prio=10 tid=0x00007f186c089000 nid=0x6ed in Object.wait() [0x00007f181e09f000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.util.TimerThread.mainLoop(Timer.java:509)
	- locked <0x000000008c7ec618> (a java.util.TaskQueue)
	at java.util.TimerThread.run(Timer.java:462)

"Old I/O datagram worker ([id: 0x6cf0d0dd, /226.1.1.3:60100])" daemon prio=10 tid=0x00007f187c03c800 nid=0x6d2 runnable [0x00007f181e1a0000]
   java.lang.Thread.State: RUNNABLE
	at java.net.PlainDatagramSocketImpl.receive0(Native Method)
	- locked <0x000000008c741650> (a java.net.PlainDatagramSocketImpl)
	at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:145)
	- locked <0x000000008c741650> (a java.net.PlainDatagramSocketImpl)
	at java.net.DatagramSocket.receive(DatagramSocket.java:725)
	- locked <0x00000000df0c5ec0> (a java.net.DatagramPacket)
	- locked <0x000000008c741610> (a java.net.MulticastSocket)
	at org.jboss.netty.channel.socket.oio.OioDatagramWorker.process(OioDatagramWorker.java:47)
	at org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(AbstractOioWorker.java:75)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
	at java.lang.Thread.run(Thread.java:662)

"RS:0;ip-10-197-73-48:56938-EventThread" daemon prio=10 tid=0x00007f187c03b000 nid=0x6d1 waiting on condition [0x00007f181e2a1000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008c744778> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)

"RS:0;ip-10-197-73-48:56938-SendThread(localhost:56045)" daemon prio=10 tid=0x00007f187c039800 nid=0x6d0 runnable [0x00007f181e3a2000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008c7441e0> (a sun.nio.ch.Util$2)
	- locked <0x000000008c7441d0> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008c743d48> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)

"Old I/O datagram worker ([id: 0x98cbc901, /226.1.1.3:60100])" daemon prio=10 tid=0x00007f188c040800 nid=0x6cf runnable [0x00007f181e8a7000]
   java.lang.Thread.State: RUNNABLE
	at java.net.PlainDatagramSocketImpl.receive0(Native Method)
	- locked <0x000000008c7eda28> (a java.net.PlainDatagramSocketImpl)
	at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:145)
	- locked <0x000000008c7eda28> (a java.net.PlainDatagramSocketImpl)
	at java.net.DatagramSocket.receive(DatagramSocket.java:725)
	- locked <0x00000000defdabf0> (a java.net.DatagramPacket)
	- locked <0x000000008c7ed9e8> (a java.net.MulticastSocket)
	at org.jboss.netty.channel.socket.oio.OioDatagramWorker.process(OioDatagramWorker.java:47)
	at org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(AbstractOioWorker.java:75)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
	at java.lang.Thread.run(Thread.java:662)

"RS:1;ip-10-197-73-48:39114-EventThread" daemon prio=10 tid=0x00007f188c03f800 nid=0x6ce waiting on condition [0x00007f181efae000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008c7fa958> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)

"RS:1;ip-10-197-73-48:39114-SendThread(localhost:56045)" daemon prio=10 tid=0x00007f188c03e800 nid=0x6cd runnable [0x00007f181e4a3000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008c7f63d0> (a sun.nio.ch.Util$2)
	- locked <0x000000008c7f63c0> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008c7f5f38> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)

"Old I/O datagram worker ([id: 0xdb461551, /226.1.1.3:60100])" daemon prio=10 tid=0x00007f186c06d000 nid=0x6cc runnable [0x00007f181e5a4000]
   java.lang.Thread.State: RUNNABLE
	at java.net.PlainDatagramSocketImpl.receive0(Native Method)
	- locked <0x000000008c7bd9e0> (a java.net.PlainDatagramSocketImpl)
	at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:145)
	- locked <0x000000008c7bd9e0> (a java.net.PlainDatagramSocketImpl)
	at java.net.DatagramSocket.receive(DatagramSocket.java:725)
	- locked <0x00000000df050328> (a java.net.DatagramPacket)
	- locked <0x000000008c7bd9a0> (a java.net.MulticastSocket)
	at org.jboss.netty.channel.socket.oio.OioDatagramWorker.process(OioDatagramWorker.java:47)
	at org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(AbstractOioWorker.java:75)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
	at java.lang.Thread.run(Thread.java:662)

"M:0;ip-10-197-73-48:38734-EventThread" daemon prio=10 tid=0x00007f186c06c800 nid=0x6cb waiting on condition [0x00007f181e7a6000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008c7c0360> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)

"M:0;ip-10-197-73-48:38734-SendThread(localhost:56045)" daemon prio=10 tid=0x00007f186c052000 nid=0x6ca runnable [0x00007f181e6a5000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008c7bfe30> (a sun.nio.ch.Util$2)
	- locked <0x000000008c7bfe20> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008c7bf998> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)

"RS:1;ip-10-197-73-48:39114-EventThread" daemon prio=10 tid=0x00007f188c023000 nid=0x656 waiting on condition [0x00007f181e9a8000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008c7fcfd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)

"RS:1;ip-10-197-73-48:39114-SendThread(localhost:56045)" daemon prio=10 tid=0x00007f188c021800 nid=0x653 runnable [0x00007f181eaa9000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008c7fca78> (a sun.nio.ch.Util$2)
	- locked <0x000000008c7fca68> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008c7fc5c8> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)

"RS:0;ip-10-197-73-48:56938-EventThread" daemon prio=10 tid=0x00007f187c01e000 nid=0x64f waiting on condition [0x00007f181ebaa000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008c746e98> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)

"RS:0;ip-10-197-73-48:56938-SendThread(localhost:56045)" daemon prio=10 tid=0x00007f187c01c800 nid=0x64e runnable [0x00007f181ecab000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008c746940> (a sun.nio.ch.Util$2)
	- locked <0x000000008c746930> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008c746430> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)

"RS:1;ip-10-197-73-48:39114" prio=10 tid=0x00007f18648c7000 nid=0x644 waiting for monitor entry [0x00007f181edab000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hbase.client.HConnectionManager.deleteConnection(HConnectionManager.java:336)
	- waiting to lock <0x000000008b57de68> (a org.apache.hadoop.hbase.client.HConnectionManager$1)
	at org.apache.hadoop.hbase.client.HConnectionManager.access$1400(HConnectionManager.java:200)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.close(HConnectionManager.java:2282)
	at org.apache.hadoop.hbase.catalog.CatalogTracker.stop(CatalogTracker.java:213)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:924)
	at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:158)
	at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:110)
	at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:142)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:337)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1131)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.hbase.util.Methods.call(Methods.java:41)
	at org.apache.hadoop.hbase.security.User.call(User.java:420)
	at org.apache.hadoop.hbase.security.User.access$300(User.java:51)
	at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:260)
	at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:140)
	at java.lang.Thread.run(Thread.java:662)

"RS:0;ip-10-197-73-48:56938" prio=10 tid=0x00007f18648c5000 nid=0x642 waiting for monitor entry [0x00007f181eeac000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hbase.client.HConnectionManager.deleteConnection(HConnectionManager.java:336)
	- waiting to lock <0x000000008b57de68> (a org.apache.hadoop.hbase.client.HConnectionManager$1)
	at org.apache.hadoop.hbase.client.HConnectionManager.access$1400(HConnectionManager.java:200)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.close(HConnectionManager.java:2282)
	at org.apache.hadoop.hbase.catalog.CatalogTracker.stop(CatalogTracker.java:213)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:924)
	at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:158)
	at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:110)
	at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:142)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:337)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1131)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.hbase.util.Methods.call(Methods.java:41)
	at org.apache.hadoop.hbase.security.User.call(User.java:420)
	at org.apache.hadoop.hbase.security.User.access$300(User.java:51)
	at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:260)
	at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:140)
	at java.lang.Thread.run(Thread.java:662)

"ip-10-197-73-48.us-west-1.compute.internal,38734,1374195379749.splitLogManagerTimeoutMonitor" daemon prio=10 tid=0x00007f186c004800 nid=0x61d in Object.wait() [0x00007f181f0af000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:94)
	- locked <0x000000008c7c1d98> (a java.lang.Object)
	at org.apache.hadoop.hbase.Chore.run(Chore.java:88)
	at java.lang.Thread.run(Thread.java:662)

"M:0;ip-10-197-73-48:38734" prio=10 tid=0x00007f18648c1000 nid=0x5f0 in Object.wait() [0x00007f181f1b0000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:94)
	- locked <0x000000008c7f8c38> (a java.lang.Object)
	at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:58)
	at org.apache.hadoop.hbase.master.HMaster.loop(HMaster.java:699)
	at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:569)
	at java.lang.Thread.run(Thread.java:662)

"HBase clusterStatusPublisher for master:ip-10-197-73-48:38734" daemon prio=10 tid=0x00007f1864638000 nid=0x513 in Object.wait() [0x00007f18208c7000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:94)
	- locked <0x000000008c7eaaa0> (a java.lang.Object)
	at org.apache.hadoop.hbase.Chore.run(Chore.java:88)
	at java.lang.Thread.run(Thread.java:662)

"Old I/O datagram worker ([id: 0x3c36f806, 0.0.0.0/0.0.0.0:59862])" daemon prio=10 tid=0x00007f1864637000 nid=0x511 runnable [0x00007f18209c8000]
   java.lang.Thread.State: RUNNABLE
	at java.net.PlainDatagramSocketImpl.receive0(Native Method)
	- locked <0x000000008c7ebab8> (a java.net.PlainDatagramSocketImpl)
	at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:145)
	- locked <0x000000008c7ebab8> (a java.net.PlainDatagramSocketImpl)
	at java.net.DatagramSocket.receive(DatagramSocket.java:725)
	- locked <0x00000000def63338> (a java.net.DatagramPacket)
	- locked <0x000000008c7eba78> (a java.net.MulticastSocket)
	at org.jboss.netty.channel.socket.oio.OioDatagramWorker.process(OioDatagramWorker.java:47)
	at org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(AbstractOioWorker.java:75)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
	at java.lang.Thread.run(Thread.java:662)

"RpcServer.handler=4,port=38734" daemon prio=10 tid=0x00007f186460f800 nid=0x4e8 waiting on condition [0x00007f1820ac9000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008c7df9d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"RpcServer.handler=3,port=38734" daemon prio=10 tid=0x00007f186460e000 nid=0x4e5 waiting on condition [0x00007f1820bca000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008c7df9d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"RpcServer.handler=2,port=38734" daemon prio=10 tid=0x00007f186460c800 nid=0x4e3 waiting on condition [0x00007f1820ccb000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008c7df9d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"RpcServer.handler=1,port=38734" daemon prio=10 tid=0x00007f18644d9800 nid=0x4e0 waiting on condition [0x00007f1820dcc000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008c7df9d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"RpcServer.handler=0,port=38734" daemon prio=10 tid=0x00007f18644d8000 nid=0x4dd waiting on condition [0x00007f1820ecd000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008c7df9d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
	at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
	at java.lang.Thread.run(Thread.java:662)

"RpcServer.listener,port=38734" daemon prio=10 tid=0x00007f18644ad000 nid=0x4d9 runnable [0x00007f1820fce000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008c7d6fc0> (a sun.nio.ch.Util$2)
	- locked <0x000000008c7d6fb0> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008c7d6b38> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
	at org.apache.hadoop.hbase.ipc.RpcServer$Listener.run(RpcServer.java:643)

"RpcServer.responder" daemon prio=10 tid=0x00007f18644ae000 nid=0x4d5 runnable [0x00007f18210cf000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008c7df6e0> (a sun.nio.ch.Util$2)
	- locked <0x000000008c7df6d0> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008c7df4c8> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.apache.hadoop.hbase.ipc.RpcServer$Responder.doRunLoop(RpcServer.java:829)
	at org.apache.hadoop.hbase.ipc.RpcServer$Responder.run(RpcServer.java:812)

"pool-1-thread-1-EventThread" daemon prio=10 tid=0x00007f18644a6000 nid=0x4b0 waiting on condition [0x00007f18211d0000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008c7ba778> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)

"pool-1-thread-1-SendThread(localhost:56045)" daemon prio=10 tid=0x00007f18644dc800 nid=0x4a8 runnable [0x00007f18212d1000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008c7ba168> (a sun.nio.ch.Util$2)
	- locked <0x000000008c7ba158> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008c7b9cd0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)

"RpcServer.reader=9,port=38734" daemon prio=10 tid=0x00007f18644a0800 nid=0x463 runnable [0x00007f18213d2000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008c7d7430> (a sun.nio.ch.Util$2)
	- locked <0x000000008c7d7420> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008c7d7208> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
	at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:535)
	- locked <0x000000008c7d71f0> (a org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader)
	at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:521)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
	at java.lang.Thread.run(Thread.java:662)

"RpcServer.reader=8,port=38734" daemon prio=10 tid=0x00007f18644a9000 nid=0x45f runnable [0x00007f18214d3000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008c7d7810> (a sun.nio.ch.Util$2)
	- locked <0x000000008c7d7800> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008c7d75e8> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
	at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:535)
	- locked <0x000000008c7d75d0> (a org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader)
	at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:521)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
	at java.lang.Thread.run(Thread.java:662)

"RpcServer.reader=7,port=38734" daemon prio=10 tid=0x00007f1864444800 nid=0x45c runnable [0x00007f18215d4000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008c7d7bf0> (a sun.nio.ch.Util$2)
	- locked <0x000000008c7d7be0> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008c7d79c8> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
	at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:535)
	- locked <0x000000008c7d79b0> (a org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader)
	at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:521)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
	at java.lang.Thread.run(Thread.java:662)

"RpcServer.reader=6,port=38734" daemon prio=10 tid=0x00007f1864443800 nid=0x45b runnable [0x00007f18216d5000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008c7d7fe0> (a sun.nio.ch.Util$2)
	- locked <0x000000008c7d7fd0> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008c7d7da8> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
	at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:535)
	- locked <0x000000008c7d7d90> (a org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader)
	at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:521)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
	at java.lang.Thread.run(Thread.java:662)

"RpcServer.reader=5,port=38734" daemon prio=10 tid=0x00007f18644b1800 nid=0x456 runnable [0x00007f18217d6000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008c7d83e0> (a sun.nio.ch.Util$2)
	- locked <0x000000008c7d83d0> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008c7d8198> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
	at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:535)
	- locked <0x000000008c7d8180> (a org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader)
	at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:521)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
	at java.lang.Thread.run(Thread.java:662)

"RpcServer.reader=4,port=38734" daemon prio=10 tid=0x00007f18644b0800 nid=0x454 runnable [0x00007f18218d7000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008c7da808> (a sun.nio.ch.Util$2)
	- locked <0x000000008c7da7f8> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008c7d85b0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
	at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:535)
	- locked <0x000000008c7d8598> (a org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader)
	at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:521)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
	at java.lang.Thread.run(Thread.java:662)

"RpcServer.reader=3,port=38734" daemon prio=10 tid=0x00007f1864435800 nid=0x452 runnable [0x00007f1821ddc000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008c7dac20> (a sun.nio.ch.Util$2)
	- locked <0x000000008c7dac10> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008c7da9d8> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
	at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:535)
	- locked <0x000000008c7da9c0> (a org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader)
	at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:521)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
	at java.lang.Thread.run(Thread.java:662)

"RpcServer.reader=2,port=38734" daemon prio=10 tid=0x00007f1864490000 nid=0x44f runnable [0x00007f18219d8000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008c7d6020> (a sun.nio.ch.Util$2)
	- locked <0x000000008c7d6010> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008c7d5dc8> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
	at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:535)
	- locked <0x000000008c7dadd8> (a org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader)
	at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:521)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
	at java.lang.Thread.run(Thread.java:662)

"RpcServer.reader=1,port=38734" daemon prio=10 tid=0x00007f1864433800 nid=0x44e runnable [0x00007f1821ad9000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008c7d5450> (a sun.nio.ch.Util$2)
	- locked <0x000000008c7d5440> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008c7d51f8> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
	at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:535)
	- locked <0x000000008c7dadf0> (a org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader)
	at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:521)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
	at java.lang.Thread.run(Thread.java:662)

"RpcServer.reader=0,port=38734" daemon prio=10 tid=0x00007f1864441000 nid=0x44c runnable [0x00007f1821bda000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008c7db048> (a sun.nio.ch.Util$2)
	- locked <0x000000008c7db038> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008c7dae20> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
	at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:535)
	- locked <0x000000008c7dae08> (a org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader)
	at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:521)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
	at java.lang.Thread.run(Thread.java:662)

"ProcessThread(sid:0 cport:-1):" prio=10 tid=0x00007f1864095000 nid=0x27c waiting on condition [0x00007f1821edd000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008be2db18> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:120)

"SyncThread:0" prio=10 tid=0x00007f18640a5800 nid=0x276 waiting on condition [0x00007f1821fde000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008be2e040> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:97)

"SessionTracker" prio=10 tid=0x00007f1864415800 nid=0x274 in Object.wait() [0x00007f18220df000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146)
	- locked <0x000000008bdaf720> (a org.apache.zookeeper.server.SessionTrackerImpl)

"NIOServerCxn.Factory:0.0.0.0/0.0.0.0:56045" daemon prio=10 tid=0x00007f186409a800 nid=0x271 runnable [0x00007f18221e0000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008bdabbf8> (a sun.nio.ch.Util$2)
	- locked <0x000000008bdabbe8> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008bda7dc8> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:178)
	at java.lang.Thread.run(Thread.java:662)

"org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@67cec874" daemon prio=10 tid=0x00007f189c011800 nid=0x266 waiting on condition [0x00007f18222e1000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:621)
	at java.lang.Thread.run(Thread.java:662)

"IPC Server handler 2 on 44186" daemon prio=10 tid=0x00007f189c00e000 nid=0x259 waiting on condition [0x00007f18223e2000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf50740> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1369)

"IPC Server handler 1 on 44186" daemon prio=10 tid=0x00007f189c00c000 nid=0x258 waiting on condition [0x00007f18224e3000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf50740> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1369)

"IPC Server handler 0 on 44186" daemon prio=10 tid=0x00007f189c00a000 nid=0x257 waiting on condition [0x00007f18225e4000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bf50740> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1369)

"IPC Server listener on 44186" daemon prio=10 tid=0x00007f189c008000 nid=0x256 runnable [0x00007f18226e5000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008bf514d8> (a sun.nio.ch.Util$2)
	- locked <0x000000008bf514c8> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008bf51060> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
	at org.apache.hadoop.ipc.Server$Listener.run(Server.java:440)

"IPC Server Responder" daemon prio=10 tid=0x00007f189c006000 nid=0x254 runnable [0x00007f18227e6000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008bf52450> (a sun.nio.ch.Util$2)
	- locked <0x000000008bf52440> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008bf52200> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.apache.hadoop.ipc.Server$Responder.run(Server.java:606)

"org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@74122d9c" daemon prio=10 tid=0x00007f189c004800 nid=0x253 runnable [0x00007f18228e7000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:150)
	- locked <0x000000008bef8e38> (a java.lang.Object)
	at sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
	- locked <0x000000008bef8e00> (a java.lang.Object)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
	at java.lang.Thread.run(Thread.java:662)

"DataNode: [/home/ec2-user/jenkins/workspace/HBase-TRUNK/hbase-server/target/test-data/6d557dd2-6b61-49d4-9727-3ffcb13952b6/dfscluster_52a2ec77-f3a4-4587-8988-c1168ffa84e7/dfs/data/data3,/home/ec2-user/jenkins/workspace/HBase-TRUNK/hbase-server/target/test-data/6d557dd2-6b61-49d4-9727-3ffcb13952b6/dfscluster_52a2ec77-f3a4-4587-8988-c1168ffa84e7/dfs/data/data4]" daemon prio=10 tid=0x00007f1864091800 nid=0x250 in Object.wait() [0x00007f18229e8000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:1091)
	- locked <0x000000008bf45a00> (a java.util.LinkedList)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1527)
	at java.lang.Thread.run(Thread.java:662)

"pool-4-thread-1" prio=10 tid=0x00007f1864090000 nid=0x24b runnable [0x00007f1822ae9000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008bf51938> (a sun.nio.ch.Util$2)
	- locked <0x000000008bf51928> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008bf51700> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
	at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:334)
	- locked <0x000000008bf516e8> (a org.apache.hadoop.ipc.Server$Listener$Reader)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
	at java.lang.Thread.run(Thread.java:662)

"Timer-2" daemon prio=10 tid=0x00007f1864068000 nid=0x249 in Object.wait() [0x00007f1822bea000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.util.TimerThread.mainLoop(Timer.java:509)
	- locked <0x000000008befa218> (a java.util.TaskQueue)
	at java.util.TimerThread.run(Timer.java:462)

"2118205382@qtp-1394528614-1 - Acceptor0 SelectChannelConnector@localhost:35201" prio=10 tid=0x00007f1864066000 nid=0x1e1 runnable [0x00007f1822ceb000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008bef9ad8> (a sun.nio.ch.Util$2)
	- locked <0x000000008bef9ac8> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008bef9870> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
	at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
	at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
	at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

"1026179434@qtp-1394528614-0" prio=10 tid=0x00007f1864064000 nid=0x1de in Object.wait() [0x00007f1822dec000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
	- locked <0x000000008bf46178> (a org.mortbay.thread.QueuedThreadPool$PoolThread)

"Async Block Report Generator" daemon prio=10 tid=0x00007f1864062800 nid=0x1dd in Object.wait() [0x00007f1822eed000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.hadoop.hdfs.server.datanode.FSDataset$AsyncBlockReport.waitForReportRequest(FSDataset.java:2253)
	- locked <0x000000008bf45468> (a org.apache.hadoop.hdfs.server.datanode.FSDataset$AsyncBlockReport)
	at org.apache.hadoop.hdfs.server.datanode.FSDataset$AsyncBlockReport.run(FSDataset.java:2223)
	at java.lang.Thread.run(Thread.java:662)

"refreshUsed-/home/ec2-user/jenkins/workspace/HBase-TRUNK/hbase-server/target/test-data/6d557dd2-6b61-49d4-9727-3ffcb13952b6/dfscluster_52a2ec77-f3a4-4587-8988-c1168ffa84e7/dfs/data/data4" daemon prio=10 tid=0x00007f186405f000 nid=0x1dc waiting on condition [0x00007f18237f6000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
	at java.lang.Thread.run(Thread.java:662)

"refreshUsed-/home/ec2-user/jenkins/workspace/HBase-TRUNK/hbase-server/target/test-data/6d557dd2-6b61-49d4-9727-3ffcb13952b6/dfscluster_52a2ec77-f3a4-4587-8988-c1168ffa84e7/dfs/data/data3" daemon prio=10 tid=0x00007f1864060800 nid=0x1d4 waiting on condition [0x00007f18230ef000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
	at java.lang.Thread.run(Thread.java:662)

"org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@3de6696c" daemon prio=10 tid=0x00007f185800e000 nid=0x1a2 waiting on condition [0x00007f1822fee000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:621)
	at java.lang.Thread.run(Thread.java:662)

"IPC Server handler 2 on 48909" daemon prio=10 tid=0x00007f185800a800 nid=0x189 waiting on condition [0x00007f18231f0000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bfc13b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1369)

"IPC Server handler 1 on 48909" daemon prio=10 tid=0x00007f1858008800 nid=0x188 waiting on condition [0x00007f18232f1000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bfc13b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1369)

"IPC Server handler 0 on 48909" daemon prio=10 tid=0x00007f1858006800 nid=0x184 waiting on condition [0x00007f18233f2000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bfc13b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1369)

"IPC Server listener on 48909" daemon prio=10 tid=0x00007f1858004800 nid=0x181 runnable [0x00007f18234f3000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008bef70e0> (a sun.nio.ch.Util$2)
	- locked <0x000000008bef70d0> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008bef6e88> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
	at org.apache.hadoop.ipc.Server$Listener.run(Server.java:440)

"IPC Server Responder" daemon prio=10 tid=0x00007f1858003000 nid=0x180 runnable [0x00007f18235f4000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008bfc21b0> (a sun.nio.ch.Util$2)
	- locked <0x000000008bfc21a0> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008bfc1f88> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.apache.hadoop.ipc.Server$Responder.run(Server.java:606)

"org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@8a2023" daemon prio=10 tid=0x00007f1858001800 nid=0x17d runnable [0x00007f18236f5000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:150)
	- locked <0x000000008be9b320> (a java.lang.Object)
	at sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
	- locked <0x000000008be9b2e8> (a java.lang.Object)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
	at java.lang.Thread.run(Thread.java:662)

"DataNode: [/home/ec2-user/jenkins/workspace/HBase-TRUNK/hbase-server/target/test-data/6d557dd2-6b61-49d4-9727-3ffcb13952b6/dfscluster_52a2ec77-f3a4-4587-8988-c1168ffa84e7/dfs/data/data1,/home/ec2-user/jenkins/workspace/HBase-TRUNK/hbase-server/target/test-data/6d557dd2-6b61-49d4-9727-3ffcb13952b6/dfscluster_52a2ec77-f3a4-4587-8988-c1168ffa84e7/dfs/data/data2]" daemon prio=10 tid=0x00007f1864323800 nid=0x178 in Object.wait() [0x00007f18238f7000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:1091)
	- locked <0x000000008bee0b58> (a java.util.LinkedList)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1527)
	at java.lang.Thread.run(Thread.java:662)

"pool-3-thread-1" prio=10 tid=0x00007f186405a000 nid=0x167 runnable [0x00007f18239f8000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008bef79d0> (a sun.nio.ch.Util$2)
	- locked <0x000000008bef79c0> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008bef77a8> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
	at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:334)
	- locked <0x000000008bfc1ce8> (a org.apache.hadoop.ipc.Server$Listener$Reader)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
	at java.lang.Thread.run(Thread.java:662)

"Timer-1" daemon prio=10 tid=0x00007f1864059800 nid=0x158 in Object.wait() [0x00007f1823af9000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.util.TimerThread.mainLoop(Timer.java:509)
	- locked <0x000000008bef6460> (a java.util.TaskQueue)
	at java.util.TimerThread.run(Timer.java:462)

"21243558@qtp-925451180-1 - Acceptor0 SelectChannelConnector@localhost:35849" prio=10 tid=0x00007f1864332000 nid=0x7f81 runnable [0x00007f1823bfa000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008bd5ccb0> (a sun.nio.ch.Util$2)
	- locked <0x000000008bd5cca0> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008bd5ca48> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
	at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
	at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
	at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

"1841186403@qtp-925451180-0" prio=10 tid=0x00007f186432c000 nid=0x7f80 in Object.wait() [0x00007f1823dfc000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
	- locked <0x000000008bfbb938> (a org.mortbay.thread.QueuedThreadPool$PoolThread)

"Async Block Report Generator" daemon prio=10 tid=0x00007f186432e000 nid=0x7f5c in Object.wait() [0x00007f1823cfb000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.hadoop.hdfs.server.datanode.FSDataset$AsyncBlockReport.waitForReportRequest(FSDataset.java:2253)
	- locked <0x000000008bee05c0> (a org.apache.hadoop.hdfs.server.datanode.FSDataset$AsyncBlockReport)
	at org.apache.hadoop.hdfs.server.datanode.FSDataset$AsyncBlockReport.run(FSDataset.java:2223)
	at java.lang.Thread.run(Thread.java:662)

"refreshUsed-/home/ec2-user/jenkins/workspace/HBase-TRUNK/hbase-server/target/test-data/6d557dd2-6b61-49d4-9727-3ffcb13952b6/dfscluster_52a2ec77-f3a4-4587-8988-c1168ffa84e7/dfs/data/data2" daemon prio=10 tid=0x00007f186432b800 nid=0x7f54 waiting on condition [0x00007f1823efd000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
	at java.lang.Thread.run(Thread.java:662)

"refreshUsed-/home/ec2-user/jenkins/workspace/HBase-TRUNK/hbase-server/target/test-data/6d557dd2-6b61-49d4-9727-3ffcb13952b6/dfscluster_52a2ec77-f3a4-4587-8988-c1168ffa84e7/dfs/data/data1" daemon prio=10 tid=0x00007f1864328800 nid=0x7f42 waiting on condition [0x00007f1823ffe000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
	at java.lang.Thread.run(Thread.java:662)

"IPC Client (47) connection to localhost/127.0.0.1:58454 from ec2-user" daemon prio=10 tid=0x00007f1864305000 nid=0x7f19 in Object.wait() [0x00007f1880182000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:743)
	- locked <0x000000008c55abc0> (a org.apache.hadoop.ipc.Client$Connection)
	at org.apache.hadoop.ipc.Client$Connection.run(Client.java:785)

"IPC Server handler 9 on 58454" daemon prio=10 tid=0x00007f18642f9000 nid=0x7ecc waiting on condition [0x00007f1880384000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bd37b60> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1369)

"IPC Server handler 8 on 58454" daemon prio=10 tid=0x00007f18642f7000 nid=0x7ecb waiting on condition [0x00007f1880485000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bd37b60> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1369)

"IPC Server handler 7 on 58454" daemon prio=10 tid=0x00007f18642f5800 nid=0x7eca waiting on condition [0x00007f1880586000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bd37b60> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1369)

"IPC Server handler 6 on 58454" daemon prio=10 tid=0x00007f18642db000 nid=0x7ec8 waiting on condition [0x00007f1880687000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bd37b60> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1369)

"IPC Server handler 5 on 58454" daemon prio=10 tid=0x00007f18642d9000 nid=0x7ec4 waiting on condition [0x00007f1880788000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bd37b60> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1369)

"IPC Server handler 4 on 58454" daemon prio=10 tid=0x00007f18642d7000 nid=0x7ec1 waiting on condition [0x00007f1880889000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bd37b60> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1369)

"IPC Server handler 3 on 58454" daemon prio=10 tid=0x00007f18642ce000 nid=0x7ebe waiting on condition [0x00007f188098a000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bd37b60> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1369)

"IPC Server handler 2 on 58454" daemon prio=10 tid=0x00007f18642cc000 nid=0x7ebd waiting on condition [0x00007f1880a8b000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bd37b60> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1369)

"IPC Server handler 1 on 58454" daemon prio=10 tid=0x00007f18642d5800 nid=0x7ebc waiting on condition [0x00007f1880b8c000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bd37b60> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1369)

"IPC Server handler 0 on 58454" daemon prio=10 tid=0x00007f18642d4800 nid=0x7ebb waiting on condition [0x00007f1880c8d000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008bd37b60> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1369)

"IPC Server listener on 58454" daemon prio=10 tid=0x00007f18642d1000 nid=0x7eb9 runnable [0x00007f1880d8e000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008bd3efa0> (a sun.nio.ch.Util$2)
	- locked <0x000000008bd3ef90> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008bd3eac0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
	at org.apache.hadoop.ipc.Server$Listener.run(Server.java:440)

"IPC Server Responder" daemon prio=10 tid=0x00007f18642c7800 nid=0x7eb7 runnable [0x00007f1880e8f000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008bd3fe30> (a sun.nio.ch.Util$2)
	- locked <0x000000008bd3fe20> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008bd3fc08> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.apache.hadoop.ipc.Server$Responder.run(Server.java:606)

"Timer-0" daemon prio=10 tid=0x00007f18642a1000 nid=0x7eac in Object.wait() [0x00007f1880f90000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.util.TimerThread.mainLoop(Timer.java:509)
	- locked <0x000000008c67b7e0> (a java.util.TaskQueue)
	at java.util.TimerThread.run(Timer.java:462)

"392609752@qtp-1690235745-1 - Acceptor0 SelectChannelConnector@localhost:41141" prio=10 tid=0x00007f1864295800 nid=0x7e6d runnable [0x00007f1881091000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008c67fb08> (a sun.nio.ch.Util$2)
	- locked <0x000000008c67faf8> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008c67f8b0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
	at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
	at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
	at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

"1382259697@qtp-1690235745-0" prio=10 tid=0x00007f1864294000 nid=0x7e6b in Object.wait() [0x00007f1881192000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
	- locked <0x000000008bd40b20> (a org.mortbay.thread.QueuedThreadPool$PoolThread)

"pool-2-thread-1" prio=10 tid=0x00007f18640d5800 nid=0x7e61 runnable [0x00007f1881293000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x000000008bd3df60> (a sun.nio.ch.Util$2)
	- locked <0x000000008bd3df50> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000008bd3c768> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
	at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:334)
	- locked <0x000000008bd3f1e8> (a org.apache.hadoop.ipc.Server$Listener$Reader)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
	at java.lang.Thread.run(Thread.java:662)

"org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@293bdd36" daemon prio=10 tid=0x00007f186406a000 nid=0x7e5e waiting on condition [0x00007f1881394000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65)
	at java.lang.Thread.run(Thread.java:662)

"org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@6f18278a" daemon prio=10 tid=0x00007f1864069800 nid=0x7e5d waiting on condition [0x00007f1881495000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2866)
	at java.lang.Thread.run(Thread.java:662)

"org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@66b51404" daemon prio=10 tid=0x00007f186420b800 nid=0x7e5c waiting on condition [0x00007f1881596000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:368)
	at java.lang.Thread.run(Thread.java:662)

"org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@5a4bf53e" daemon prio=10 tid=0x00007f18640ac000 nid=0x7e5a waiting on condition [0x00007f1881697000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2845)
	at java.lang.Thread.run(Thread.java:662)

"org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@256eb6bc" daemon prio=10 tid=0x00007f18640ba000 nid=0x7e59 waiting on condition [0x00007f1881798000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
	at java.lang.Thread.run(Thread.java:662)

"pool-1-thread-1" prio=10 tid=0x00007f18a8508800 nid=0x7a6e in Object.wait() [0x00007f1881af4000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1309)
	- locked <0x000000008cdf60a8> (a org.apache.zookeeper.ClientCnxn$Packet)
	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1036)
	at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:191)
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:482)
	at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)
	at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:83)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.retrieveClusterId(HConnectionManager.java:588)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:506)
	at sun.reflect.GeneratedConstructorAccessor25.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
	at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:293)
	at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:252)
	- locked <0x000000008b57de68> (a org.apache.hadoop.hbase.client.HConnectionManager$1)
	at org.apache.hadoop.hbase.client.TestHCM.testDeleteForZKConnLeak(TestHCM.java:924)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
	at org.junit.runners.Suite.runChild(Suite.java:127)
	at org.junit.runners.Suite.runChild(Suite.java:26)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
	at java.lang.Thread.run(Thread.java:662)

"Low Memory Detector" daemon prio=10 tid=0x00007f18a80b7800 nid=0x7971 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" daemon prio=10 tid=0x00007f18a80b5000 nid=0x7970 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" daemon prio=10 tid=0x00007f18a80b2800 nid=0x796b waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x00007f18a80b0000 nid=0x796a runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x00007f18a8094000 nid=0x7950 waiting for monitor entry [0x00007f18a4bf9000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hbase.client.HConnectionManager.deleteConnection(HConnectionManager.java:336)
	- waiting to lock <0x000000008b57de68> (a org.apache.hadoop.hbase.client.HConnectionManager$1)
	at org.apache.hadoop.hbase.client.HConnectionManager.access$1400(HConnectionManager.java:200)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.close(HConnectionManager.java:2282)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.finalize(HConnectionManager.java:2305)
	at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)
	at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:83)
	at java.lang.ref.Finalizer.access$100(Finalizer.java:14)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:160)

"Reference Handler" daemon prio=10 tid=0x00007f18a8092000 nid=0x794d in Object.wait() [0x00007f18a4cfb000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
	- locked <0x000000008bfa31c8> (a java.lang.ref.Reference$Lock)

"main" prio=10 tid=0x00007f18a8007000 nid=0x792d waiting on condition [0x00007f18ad9e6000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000008c5615d8> (a java.util.concurrent.FutureTask$Sync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
	at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:218)
	at java.util.concurrent.FutureTask.get(FutureTask.java:83)
	at org.apache.maven.surefire.junitcore.AsynchronousRunner.waitForCompletion(AsynchronousRunner.java:73)
	at org.apache.maven.surefire.junitcore.AsynchronousRunner.finished(AsynchronousRunner.java:58)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:242)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:138)
	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:61)
	at org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:137)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:188)
	at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:166)
	at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:86)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:101)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)

"VM Thread" prio=10 tid=0x00007f18a808b800 nid=0x7946 runnable 

"GC task thread#0 (ParallelGC)" prio=10 tid=0x00007f18a8019800 nid=0x792f runnable 

"GC task thread#1 (ParallelGC)" prio=10 tid=0x00007f18a801b800 nid=0x7932 runnable 

"GC task thread#2 (ParallelGC)" prio=10 tid=0x00007f18a801d800 nid=0x7935 runnable 

"GC task thread#3 (ParallelGC)" prio=10 tid=0x00007f18a801f000 nid=0x7937 runnable 

"VM Periodic Task Thread" prio=10 tid=0x00007f18a80c2000 nid=0x7972 waiting on condition 

JNI global references: 1191

+ echo '************ END  zombies jstack extract'
************ END  zombies jstack extract
+ JIRA_COMMENT='

     {color:red}-1 core zombie tests{color}.  There are 1 zombie test(s): 	at org.apache.hadoop.hbase.client.TestHCM.testDeleteForZKConnLeak(TestHCM.java:924)'
+ BAD=1
+ jps
+ grep surefirebooter
+ cut -d ' ' -f 1
+ xargs kill -9
Sending e-mails to: builds@hbase.apache.org
Finished: FAILURE
{code}
                
> TestHCM.testDeleteForZKConnLeak can go zombie
> ---------------------------------------------
>
>                 Key: HBASE-8996
>                 URL: https://issues.apache.org/jira/browse/HBASE-8996
>             Project: HBase
>          Issue Type: Bug
>            Reporter: stack
>
> Going to just disable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira