You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "nkeywal (Created) (JIRA)" <ji...@apache.org> on 2011/11/02 07:39:32 UTC
[jira] [Created] (HBASE-4724) TestAdmin hangs randomly in trunk
TestAdmin hangs randomly in trunk
---------------------------------
Key: HBASE-4724
URL: https://issues.apache.org/jira/browse/HBASE-4724
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 0.94.0
Reporter: nkeywal
Priority: Critical
fom the logs in my env
{noformat}
2011-11-01 15:48:40,744 WARN [Master:0;localhost,39664,1320187706355] master.AssignmentManager(1471): Failed assignment of -ROOT-,,0.70236052 to localhost,44046,1320187706849, trying to assign elsewhere instead; retry=1
org.apache.hadoop.hbase.ipc.HBaseRPC$VersionMismatch: Protocol org.apache.hadoop.hbase.ipc.HRegionInterface version mismatch. (client = 28, server = 29)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:185)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:300)
{noformat}
Anyway, after this the logs finishes with:
{noformat}
2011-11-01 15:54:35,132 INFO [Master:0;localhost,39664,1320187706355.oldLogCleaner] hbase.Chore(80): Master:0;localhost,39664,1320187706355.oldLogCleaner exiting
Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on Master:0;localhost,39664,1320187706355
{noformat}
it's in
{noformat}
sun.management.ThreadImpl.getThreadInfo1(Native Method)
sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
{noformat}
So that's at least why adding a timeout wont help and may be why it does not end at all. Adding a maximum retry to Threads#threadDumpingIsAlive could help.
I also wonder if the root cause of the non ending is my modif on the wal, with some threads surprised to have updates that were not written in the wal. Here is the full stack dump:
{noformat}
Thread 354 (IPC Client (47) connection to localhost/127.0.0.1:52227 from nkeywal):
State: TIMED_WAITING
Blocked count: 360
Waited count: 359
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:702)
org.apache.hadoop.ipc.Client$Connection.run(Client.java:744)
Thread 272 (Master:0;localhost,39664,1320187706355-EventThread):
State: WAITING
Blocked count: 0
Waited count: 4
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@107b954b
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
Thread 271 (Master:0;localhost,39664,1320187706355-SendThread(localhost:21819)):
State: RUNNABLE
Blocked count: 2
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
Thread 152 (Master:0;localhost,39664,1320187706355):
State: WAITING
Blocked count: 217
Waited count: 174
Waiting on org.apache.hadoop.hbase.zookeeper.RootRegionTracker@6621477c
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:131)
org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:104)
org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRoot(CatalogTracker.java:277)
org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:523)
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:468)
org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)
java.lang.Thread.run(Thread.java:662)
Thread 165 (LruBlockCache.EvictionThread):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread@3e9d7b56
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:593)
java.lang.Thread.run(Thread.java:662)
Thread 151 (main-EventThread):
State: WAITING
Blocked count: 40
Waited count: 39
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22531380
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
Thread 150 (main-SendThread(localhost:21819)):
State: RUNNABLE
Blocked count: 55
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
Thread 149 (IPC Server handler 4 on 39664):
State: WAITING
Blocked count: 3
Waited count: 1968
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
Thread 148 (IPC Server handler 3 on 39664):
State: WAITING
Blocked count: 7
Waited count: 1969
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
Thread 147 (IPC Server handler 2 on 39664):
State: WAITING
Blocked count: 11
Waited count: 1968
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
Thread 146 (IPC Server handler 1 on 39664):
State: WAITING
Blocked count: 6
Waited count: 1969
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
Thread 145 (IPC Server handler 0 on 39664):
State: WAITING
Blocked count: 3
Waited count: 1970
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
Thread 133 (IPC Server listener on 39664):
State: RUNNABLE
Blocked count: 1
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:580)
Thread 144 (IPC Server Responder):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:754)
Thread 143 (IPC Reader 9 on port 39664):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 142 (IPC Reader 8 on port 39664):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 141 (IPC Reader 7 on port 39664):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 140 (IPC Reader 6 on port 39664):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 139 (IPC Reader 5 on port 39664):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 138 (IPC Reader 4 on port 39664):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 137 (IPC Reader 3 on port 39664):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 136 (IPC Reader 2 on port 39664):
State: RUNNABLE
Blocked count: 1
Waited count: 1
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 135 (IPC Reader 1 on port 39664):
State: RUNNABLE
Blocked count: 1
Waited count: 1
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 134 (IPC Reader 0 on port 39664):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 124 (LeaseChecker):
State: TIMED_WAITING
Blocked count: 0
Waited count: 407
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1252)
java.lang.Thread.run(Thread.java:662)
Thread 121 (ProcessThread:-1):
State: WAITING
Blocked count: 0
Waited count: 269
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f5d84e0
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:103)
Thread 120 (SyncThread:0):
State: WAITING
Blocked count: 2
Waited count: 252
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b25b27c
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:94)
Thread 119 (SessionTracker):
State: TIMED_WAITING
Blocked count: 0
Waited count: 207
Stack:
java.lang.Object.wait(Native Method)
org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:142)
Thread 118 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:21819):
State: RUNNABLE
Blocked count: 38
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:232)
Thread 116 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@2e6f947b):
State: TIMED_WAITING
Blocked count: 0
Waited count: 407
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
java.lang.Thread.run(Thread.java:662)
Thread 113 (IPC Server handler 2 on 39899):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 112 (IPC Server handler 1 on 39899):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 111 (IPC Server handler 0 on 39899):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 106 (IPC Server listener on 39899):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
Thread 108 (IPC Server Responder):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
Thread 102 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@45d1c3cd):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
java.lang.Thread.run(Thread.java:662)
Thread 109 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6]):
State: TIMED_WAITING
Blocked count: 137
Waited count: 283
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
java.lang.Thread.run(Thread.java:662)
Thread 107 (pool-4-thread-1):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 105 (Timer-3):
State: TIMED_WAITING
Blocked count: 0
Waited count: 15
Stack:
java.lang.Object.wait(Native Method)
java.util.TimerThread.mainLoop(Timer.java:509)
java.util.TimerThread.run(Timer.java:462)
Thread 104 (1353906974@qtp-1197660496-1 - Acceptor0 SelectChannelConnector@localhost:59858):
State: RUNNABLE
Blocked count: 1
Waited count: 1
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Thread 103 (1185716172@qtp-1197660496-0):
State: TIMED_WAITING
Blocked count: 0
Waited count: 8
Stack:
java.lang.Object.wait(Native Method)
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
Thread 101 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6):
State: TIMED_WAITING
Blocked count: 0
Waited count: 1
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
java.lang.Thread.run(Thread.java:662)
Thread 98 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5):
State: TIMED_WAITING
Blocked count: 0
Waited count: 1
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
java.lang.Thread.run(Thread.java:662)
Thread 90 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@605b28c9):
State: TIMED_WAITING
Blocked count: 0
Waited count: 408
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
java.lang.Thread.run(Thread.java:662)
Thread 86 (IPC Server handler 2 on 56098):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 85 (IPC Server handler 1 on 56098):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 84 (IPC Server handler 0 on 56098):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 80 (IPC Server listener on 56098):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
Thread 82 (IPC Server Responder):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
Thread 76 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@2b68989e):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
java.lang.Thread.run(Thread.java:662)
Thread 83 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4]):
State: TIMED_WAITING
Blocked count: 149
Waited count: 290
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
java.lang.Thread.run(Thread.java:662)
Thread 81 (pool-3-thread-1):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 79 (Timer-2):
State: TIMED_WAITING
Blocked count: 0
Waited count: 15
Stack:
java.lang.Object.wait(Native Method)
java.util.TimerThread.mainLoop(Timer.java:509)
java.util.TimerThread.run(Timer.java:462)
Thread 78 (348878159@qtp-164967086-1 - Acceptor0 SelectChannelConnector@localhost:36427):
State: RUNNABLE
Blocked count: 2
Waited count: 1
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Thread 77 (1905729203@qtp-164967086-0):
State: TIMED_WAITING
Blocked count: 0
Waited count: 8
Stack:
java.lang.Object.wait(Native Method)
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
Thread 75 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4):
State: TIMED_WAITING
Blocked count: 0
Waited count: 1
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
java.lang.Thread.run(Thread.java:662)
Thread 72 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3):
State: TIMED_WAITING
Blocked count: 0
Waited count: 1
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
java.lang.Thread.run(Thread.java:662)
Thread 61 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@6709da93):
State: TIMED_WAITING
Blocked count: 0
Waited count: 408
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
java.lang.Thread.run(Thread.java:662)
Thread 60 (IPC Server handler 2 on 40388):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 59 (IPC Server handler 1 on 40388):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 58 (IPC Server handler 0 on 40388):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 53 (IPC Server listener on 40388):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
Thread 55 (IPC Server Responder):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
Thread 49 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@48baa31b):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
java.lang.Thread.run(Thread.java:662)
Thread 57 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2]):
State: TIMED_WAITING
Blocked count: 148
Waited count: 283
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
java.lang.Thread.run(Thread.java:662)
Thread 54 (pool-2-thread-1):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 52 (Timer-1):
State: TIMED_WAITING
Blocked count: 0
Waited count: 15
Stack:
java.lang.Object.wait(Native Method)
java.util.TimerThread.mainLoop(Timer.java:509)
java.util.TimerThread.run(Timer.java:462)
Thread 51 (1031841975@qtp-1068922666-1 - Acceptor0 SelectChannelConnector@localhost:56535):
State: RUNNABLE
Blocked count: 1
Waited count: 1
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Thread 50 (1191055321@qtp-1068922666-0):
State: TIMED_WAITING
Blocked count: 0
Waited count: 8
Stack:
java.lang.Object.wait(Native Method)
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
Thread 48 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2):
State: TIMED_WAITING
Blocked count: 0
Waited count: 1
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
java.lang.Thread.run(Thread.java:662)
Thread 45 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1):
State: TIMED_WAITING
Blocked count: 0
Waited count: 1
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
java.lang.Thread.run(Thread.java:662)
Thread 31 (IPC Server handler 9 on 52227):
State: WAITING
Blocked count: 0
Waited count: 64
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 30 (IPC Server handler 8 on 52227):
State: WAITING
Blocked count: 3
Waited count: 65
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 29 (IPC Server handler 7 on 52227):
State: WAITING
Blocked count: 2
Waited count: 66
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 28 (IPC Server handler 6 on 52227):
State: WAITING
Blocked count: 1
Waited count: 64
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 27 (IPC Server handler 5 on 52227):
State: WAITING
Blocked count: 4
Waited count: 68
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 26 (IPC Server handler 4 on 52227):
State: WAITING
Blocked count: 0
Waited count: 64
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 25 (IPC Server handler 3 on 52227):
State: WAITING
Blocked count: 0
Waited count: 65
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 24 (IPC Server handler 2 on 52227):
State: WAITING
Blocked count: 1
Waited count: 66
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 23 (IPC Server handler 1 on 52227):
State: WAITING
Blocked count: 0
Waited count: 64
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 22 (IPC Server handler 0 on 52227):
State: WAITING
Blocked count: 3
Waited count: 65
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 15 (IPC Server listener on 52227):
State: RUNNABLE
Blocked count: 43
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
Thread 17 (IPC Server Responder):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
Thread 21 (Timer-0):
State: TIMED_WAITING
Blocked count: 1
Waited count: 17
Stack:
java.lang.Object.wait(Native Method)
java.util.TimerThread.mainLoop(Timer.java:509)
java.util.TimerThread.run(Timer.java:462)
Thread 20 (245022965@qtp-589734556-1 - Acceptor0 SelectChannelConnector@localhost:33079):
State: RUNNABLE
Blocked count: 2
Waited count: 1
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Thread 19 (1821050251@qtp-589734556-0):
State: TIMED_WAITING
Blocked count: 0
Waited count: 8
Stack:
java.lang.Object.wait(Native Method)
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
Thread 16 (pool-1-thread-1):
State: RUNNABLE
Blocked count: 45
Waited count: 44
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 14 (org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@620fa83):
State: TIMED_WAITING
Blocked count: 1
Waited count: 147
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65)
java.lang.Thread.run(Thread.java:662)
Thread 13 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@d5d4de6):
State: TIMED_WAITING
Blocked count: 2
Waited count: 147
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2689)
java.lang.Thread.run(Thread.java:662)
Thread 12 (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@73d32e45):
State: TIMED_WAITING
Blocked count: 0
Waited count: 220
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:368)
java.lang.Thread.run(Thread.java:662)
Thread 11 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@19be4777):
State: TIMED_WAITING
Blocked count: 0
Waited count: 89
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2666)
java.lang.Thread.run(Thread.java:662)
Thread 10 (org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@658f7386):
State: TIMED_WAITING
Blocked count: 0
Waited count: 2
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
java.lang.Thread.run(Thread.java:662)
Thread 4 (Signal Dispatcher):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
Thread 3 (Finalizer):
State: WAITING
Blocked count: 9
Waited count: 10
Waiting on java.lang.ref.ReferenceQueue$Lock@3d1a70a7
Stack:
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
Thread 2 (Reference Handler):
State: WAITING
Blocked count: 11
Waited count: 12
Waiting on java.lang.ref.Reference$Lock@270d75a3
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
Thread 1 (main):
State: RUNNABLE
Blocked count: 104
Waited count: 97
Stack:
sun.management.ThreadImpl.getThreadInfo1(Native Method)
sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
org.junit.runners.ParentRunner.run(ParentRunner.java:236)
org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4724) TestAdmin hangs randomly in trunk
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142239#comment-13142239 ]
stack commented on HBASE-4724:
------------------------------
Before applying the patch, I got the below on random run:
{code}Tests in error:
testDisableAndEnableTable(org.apache.hadoop.hbase.client.TestAdmin): org.apache.hadoop.hbase.TableNotEnabledException: testDisableAndEnableTable{code}
Trying w/ your v2 patch now.
> TestAdmin hangs randomly in trunk
> ---------------------------------
>
> Key: HBASE-4724
> URL: https://issues.apache.org/jira/browse/HBASE-4724
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 20111102_4724_TestAdmin.patch, 20111102_4724_TestAdmin.v2.patch
>
>
> fom the logs in my env
> {noformat}
> 2011-11-01 15:48:40,744 WARN [Master:0;localhost,39664,1320187706355] master.AssignmentManager(1471): Failed assignment of -ROOT-,,0.70236052 to localhost,44046,1320187706849, trying to assign elsewhere instead; retry=1
> org.apache.hadoop.hbase.ipc.HBaseRPC$VersionMismatch: Protocol org.apache.hadoop.hbase.ipc.HRegionInterface version mismatch. (client = 28, server = 29)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:185)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:300)
> {noformat}
> Anyway, after this the logs finishes with:
> {noformat}
> 2011-11-01 15:54:35,132 INFO [Master:0;localhost,39664,1320187706355.oldLogCleaner] hbase.Chore(80): Master:0;localhost,39664,1320187706355.oldLogCleaner exiting
> Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on Master:0;localhost,39664,1320187706355
> {noformat}
> it's in
> {noformat}
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> {noformat}
> So that's at least why adding a timeout wont help and may be why it does not end at all. Adding a maximum retry to Threads#threadDumpingIsAlive could help.
> I also wonder if the root cause of the non ending is my modif on the wal, with some threads surprised to have updates that were not written in the wal. Here is the full stack dump:
> {noformat}
> Thread 354 (IPC Client (47) connection to localhost/127.0.0.1:52227 from nkeywal):
> State: TIMED_WAITING
> Blocked count: 360
> Waited count: 359
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:702)
> org.apache.hadoop.ipc.Client$Connection.run(Client.java:744)
> Thread 272 (Master:0;localhost,39664,1320187706355-EventThread):
> State: WAITING
> Blocked count: 0
> Waited count: 4
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@107b954b
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 271 (Master:0;localhost,39664,1320187706355-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 152 (Master:0;localhost,39664,1320187706355):
> State: WAITING
> Blocked count: 217
> Waited count: 174
> Waiting on org.apache.hadoop.hbase.zookeeper.RootRegionTracker@6621477c
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:131)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:104)
> org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRoot(CatalogTracker.java:277)
> org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:523)
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:468)
> org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)
> java.lang.Thread.run(Thread.java:662)
> Thread 165 (LruBlockCache.EvictionThread):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread@3e9d7b56
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:593)
> java.lang.Thread.run(Thread.java:662)
> Thread 151 (main-EventThread):
> State: WAITING
> Blocked count: 40
> Waited count: 39
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22531380
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 150 (main-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 55
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 149 (IPC Server handler 4 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 148 (IPC Server handler 3 on 39664):
> State: WAITING
> Blocked count: 7
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 147 (IPC Server handler 2 on 39664):
> State: WAITING
> Blocked count: 11
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 146 (IPC Server handler 1 on 39664):
> State: WAITING
> Blocked count: 6
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 145 (IPC Server handler 0 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1970
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 133 (IPC Server listener on 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:580)
> Thread 144 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:754)
> Thread 143 (IPC Reader 9 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 142 (IPC Reader 8 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 141 (IPC Reader 7 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 140 (IPC Reader 6 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 139 (IPC Reader 5 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 138 (IPC Reader 4 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 137 (IPC Reader 3 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 136 (IPC Reader 2 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 135 (IPC Reader 1 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 134 (IPC Reader 0 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 124 (LeaseChecker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1252)
> java.lang.Thread.run(Thread.java:662)
> Thread 121 (ProcessThread:-1):
> State: WAITING
> Blocked count: 0
> Waited count: 269
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f5d84e0
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:103)
> Thread 120 (SyncThread:0):
> State: WAITING
> Blocked count: 2
> Waited count: 252
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b25b27c
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:94)
> Thread 119 (SessionTracker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 207
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:142)
> Thread 118 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:21819):
> State: RUNNABLE
> Blocked count: 38
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:232)
> Thread 116 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@2e6f947b):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 113 (IPC Server handler 2 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 112 (IPC Server handler 1 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 111 (IPC Server handler 0 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 106 (IPC Server listener on 39899):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 108 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 102 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@45d1c3cd):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 109 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6]):
> State: TIMED_WAITING
> Blocked count: 137
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 107 (pool-4-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 105 (Timer-3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 104 (1353906974@qtp-1197660496-1 - Acceptor0 SelectChannelConnector@localhost:59858):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 103 (1185716172@qtp-1197660496-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 101 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 98 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 90 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@605b28c9):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 86 (IPC Server handler 2 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 85 (IPC Server handler 1 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 84 (IPC Server handler 0 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 80 (IPC Server listener on 56098):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 82 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 76 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@2b68989e):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 83 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4]):
> State: TIMED_WAITING
> Blocked count: 149
> Waited count: 290
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 81 (pool-3-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 79 (Timer-2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 78 (348878159@qtp-164967086-1 - Acceptor0 SelectChannelConnector@localhost:36427):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 77 (1905729203@qtp-164967086-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 75 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 72 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 61 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@6709da93):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 60 (IPC Server handler 2 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 59 (IPC Server handler 1 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 58 (IPC Server handler 0 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 53 (IPC Server listener on 40388):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 55 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 49 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@48baa31b):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 57 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2]):
> State: TIMED_WAITING
> Blocked count: 148
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 54 (pool-2-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 52 (Timer-1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 51 (1031841975@qtp-1068922666-1 - Acceptor0 SelectChannelConnector@localhost:56535):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 50 (1191055321@qtp-1068922666-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 48 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 45 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 31 (IPC Server handler 9 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 30 (IPC Server handler 8 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 29 (IPC Server handler 7 on 52227):
> State: WAITING
> Blocked count: 2
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 28 (IPC Server handler 6 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 27 (IPC Server handler 5 on 52227):
> State: WAITING
> Blocked count: 4
> Waited count: 68
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 26 (IPC Server handler 4 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 25 (IPC Server handler 3 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 24 (IPC Server handler 2 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 23 (IPC Server handler 1 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 22 (IPC Server handler 0 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 15 (IPC Server listener on 52227):
> State: RUNNABLE
> Blocked count: 43
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 17 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 21 (Timer-0):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 17
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 20 (245022965@qtp-589734556-1 - Acceptor0 SelectChannelConnector@localhost:33079):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 19 (1821050251@qtp-589734556-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 16 (pool-1-thread-1):
> State: RUNNABLE
> Blocked count: 45
> Waited count: 44
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 14 (org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@620fa83):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65)
> java.lang.Thread.run(Thread.java:662)
> Thread 13 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@d5d4de6):
> State: TIMED_WAITING
> Blocked count: 2
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2689)
> java.lang.Thread.run(Thread.java:662)
> Thread 12 (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@73d32e45):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 220
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:368)
> java.lang.Thread.run(Thread.java:662)
> Thread 11 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@19be4777):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 89
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2666)
> java.lang.Thread.run(Thread.java:662)
> Thread 10 (org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@658f7386):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 2
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
> java.lang.Thread.run(Thread.java:662)
> Thread 4 (Signal Dispatcher):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> Thread 3 (Finalizer):
> State: WAITING
> Blocked count: 9
> Waited count: 10
> Waiting on java.lang.ref.ReferenceQueue$Lock@3d1a70a7
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> Thread 2 (Reference Handler):
> State: WAITING
> Blocked count: 11
> Waited count: 12
> Waiting on java.lang.ref.Reference$Lock@270d75a3
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> Thread 1 (main):
> State: RUNNABLE
> Blocked count: 104
> Waited count: 97
> Stack:
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> java.lang.reflect.Method.invoke(Method.java:597)
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
> org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4724) TestAdmin hangs randomly in trunk
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4724:
-------------------------
Priority: Major (was: Critical)
Downgrading to major. We've not seen this happen since commit. Will leave open another while but no longer seems critical.
> TestAdmin hangs randomly in trunk
> ---------------------------------
>
> Key: HBASE-4724
> URL: https://issues.apache.org/jira/browse/HBASE-4724
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Fix For: 0.92.0
>
> Attachments: 20111102_4724_TestAdmin.patch, 20111102_4724_TestAdmin.v2.patch
>
>
> fom the logs in my env
> {noformat}
> 2011-11-01 15:48:40,744 WARN [Master:0;localhost,39664,1320187706355] master.AssignmentManager(1471): Failed assignment of -ROOT-,,0.70236052 to localhost,44046,1320187706849, trying to assign elsewhere instead; retry=1
> org.apache.hadoop.hbase.ipc.HBaseRPC$VersionMismatch: Protocol org.apache.hadoop.hbase.ipc.HRegionInterface version mismatch. (client = 28, server = 29)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:185)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:300)
> {noformat}
> Anyway, after this the logs finishes with:
> {noformat}
> 2011-11-01 15:54:35,132 INFO [Master:0;localhost,39664,1320187706355.oldLogCleaner] hbase.Chore(80): Master:0;localhost,39664,1320187706355.oldLogCleaner exiting
> Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on Master:0;localhost,39664,1320187706355
> {noformat}
> it's in
> {noformat}
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> {noformat}
> So that's at least why adding a timeout wont help and may be why it does not end at all. Adding a maximum retry to Threads#threadDumpingIsAlive could help.
> I also wonder if the root cause of the non ending is my modif on the wal, with some threads surprised to have updates that were not written in the wal. Here is the full stack dump:
> {noformat}
> Thread 354 (IPC Client (47) connection to localhost/127.0.0.1:52227 from nkeywal):
> State: TIMED_WAITING
> Blocked count: 360
> Waited count: 359
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:702)
> org.apache.hadoop.ipc.Client$Connection.run(Client.java:744)
> Thread 272 (Master:0;localhost,39664,1320187706355-EventThread):
> State: WAITING
> Blocked count: 0
> Waited count: 4
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@107b954b
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 271 (Master:0;localhost,39664,1320187706355-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 152 (Master:0;localhost,39664,1320187706355):
> State: WAITING
> Blocked count: 217
> Waited count: 174
> Waiting on org.apache.hadoop.hbase.zookeeper.RootRegionTracker@6621477c
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:131)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:104)
> org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRoot(CatalogTracker.java:277)
> org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:523)
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:468)
> org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)
> java.lang.Thread.run(Thread.java:662)
> Thread 165 (LruBlockCache.EvictionThread):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread@3e9d7b56
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:593)
> java.lang.Thread.run(Thread.java:662)
> Thread 151 (main-EventThread):
> State: WAITING
> Blocked count: 40
> Waited count: 39
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22531380
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 150 (main-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 55
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 149 (IPC Server handler 4 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 148 (IPC Server handler 3 on 39664):
> State: WAITING
> Blocked count: 7
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 147 (IPC Server handler 2 on 39664):
> State: WAITING
> Blocked count: 11
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 146 (IPC Server handler 1 on 39664):
> State: WAITING
> Blocked count: 6
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 145 (IPC Server handler 0 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1970
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 133 (IPC Server listener on 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:580)
> Thread 144 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:754)
> Thread 143 (IPC Reader 9 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 142 (IPC Reader 8 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 141 (IPC Reader 7 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 140 (IPC Reader 6 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 139 (IPC Reader 5 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 138 (IPC Reader 4 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 137 (IPC Reader 3 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 136 (IPC Reader 2 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 135 (IPC Reader 1 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 134 (IPC Reader 0 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 124 (LeaseChecker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1252)
> java.lang.Thread.run(Thread.java:662)
> Thread 121 (ProcessThread:-1):
> State: WAITING
> Blocked count: 0
> Waited count: 269
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f5d84e0
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:103)
> Thread 120 (SyncThread:0):
> State: WAITING
> Blocked count: 2
> Waited count: 252
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b25b27c
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:94)
> Thread 119 (SessionTracker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 207
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:142)
> Thread 118 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:21819):
> State: RUNNABLE
> Blocked count: 38
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:232)
> Thread 116 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@2e6f947b):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 113 (IPC Server handler 2 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 112 (IPC Server handler 1 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 111 (IPC Server handler 0 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 106 (IPC Server listener on 39899):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 108 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 102 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@45d1c3cd):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 109 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6]):
> State: TIMED_WAITING
> Blocked count: 137
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 107 (pool-4-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 105 (Timer-3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 104 (1353906974@qtp-1197660496-1 - Acceptor0 SelectChannelConnector@localhost:59858):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 103 (1185716172@qtp-1197660496-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 101 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 98 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 90 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@605b28c9):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 86 (IPC Server handler 2 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 85 (IPC Server handler 1 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 84 (IPC Server handler 0 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 80 (IPC Server listener on 56098):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 82 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 76 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@2b68989e):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 83 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4]):
> State: TIMED_WAITING
> Blocked count: 149
> Waited count: 290
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 81 (pool-3-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 79 (Timer-2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 78 (348878159@qtp-164967086-1 - Acceptor0 SelectChannelConnector@localhost:36427):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 77 (1905729203@qtp-164967086-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 75 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 72 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 61 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@6709da93):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 60 (IPC Server handler 2 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 59 (IPC Server handler 1 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 58 (IPC Server handler 0 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 53 (IPC Server listener on 40388):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 55 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 49 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@48baa31b):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 57 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2]):
> State: TIMED_WAITING
> Blocked count: 148
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 54 (pool-2-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 52 (Timer-1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 51 (1031841975@qtp-1068922666-1 - Acceptor0 SelectChannelConnector@localhost:56535):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 50 (1191055321@qtp-1068922666-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 48 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 45 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 31 (IPC Server handler 9 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 30 (IPC Server handler 8 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 29 (IPC Server handler 7 on 52227):
> State: WAITING
> Blocked count: 2
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 28 (IPC Server handler 6 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 27 (IPC Server handler 5 on 52227):
> State: WAITING
> Blocked count: 4
> Waited count: 68
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 26 (IPC Server handler 4 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 25 (IPC Server handler 3 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 24 (IPC Server handler 2 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 23 (IPC Server handler 1 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 22 (IPC Server handler 0 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 15 (IPC Server listener on 52227):
> State: RUNNABLE
> Blocked count: 43
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 17 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 21 (Timer-0):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 17
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 20 (245022965@qtp-589734556-1 - Acceptor0 SelectChannelConnector@localhost:33079):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 19 (1821050251@qtp-589734556-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 16 (pool-1-thread-1):
> State: RUNNABLE
> Blocked count: 45
> Waited count: 44
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 14 (org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@620fa83):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65)
> java.lang.Thread.run(Thread.java:662)
> Thread 13 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@d5d4de6):
> State: TIMED_WAITING
> Blocked count: 2
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2689)
> java.lang.Thread.run(Thread.java:662)
> Thread 12 (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@73d32e45):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 220
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:368)
> java.lang.Thread.run(Thread.java:662)
> Thread 11 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@19be4777):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 89
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2666)
> java.lang.Thread.run(Thread.java:662)
> Thread 10 (org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@658f7386):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 2
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
> java.lang.Thread.run(Thread.java:662)
> Thread 4 (Signal Dispatcher):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> Thread 3 (Finalizer):
> State: WAITING
> Blocked count: 9
> Waited count: 10
> Waiting on java.lang.ref.ReferenceQueue$Lock@3d1a70a7
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> Thread 2 (Reference Handler):
> State: WAITING
> Blocked count: 11
> Waited count: 12
> Waiting on java.lang.ref.Reference$Lock@270d75a3
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> Thread 1 (main):
> State: RUNNABLE
> Blocked count: 104
> Waited count: 97
> Stack:
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> java.lang.reflect.Method.invoke(Method.java:597)
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
> org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4724) TestAdmin hangs randomly in trunk
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142011#comment-13142011 ]
nkeywal commented on HBASE-4724:
--------------------------------
I don't see the same stuff in the global build. I am gonna give a try by removing the modifications linked to the wal in admin.
> TestAdmin hangs randomly in trunk
> ---------------------------------
>
> Key: HBASE-4724
> URL: https://issues.apache.org/jira/browse/HBASE-4724
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Critical
> Attachments: 20111102_4724_TestAdmin.patch
>
>
> fom the logs in my env
> {noformat}
> 2011-11-01 15:48:40,744 WARN [Master:0;localhost,39664,1320187706355] master.AssignmentManager(1471): Failed assignment of -ROOT-,,0.70236052 to localhost,44046,1320187706849, trying to assign elsewhere instead; retry=1
> org.apache.hadoop.hbase.ipc.HBaseRPC$VersionMismatch: Protocol org.apache.hadoop.hbase.ipc.HRegionInterface version mismatch. (client = 28, server = 29)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:185)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:300)
> {noformat}
> Anyway, after this the logs finishes with:
> {noformat}
> 2011-11-01 15:54:35,132 INFO [Master:0;localhost,39664,1320187706355.oldLogCleaner] hbase.Chore(80): Master:0;localhost,39664,1320187706355.oldLogCleaner exiting
> Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on Master:0;localhost,39664,1320187706355
> {noformat}
> it's in
> {noformat}
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> {noformat}
> So that's at least why adding a timeout wont help and may be why it does not end at all. Adding a maximum retry to Threads#threadDumpingIsAlive could help.
> I also wonder if the root cause of the non ending is my modif on the wal, with some threads surprised to have updates that were not written in the wal. Here is the full stack dump:
> {noformat}
> Thread 354 (IPC Client (47) connection to localhost/127.0.0.1:52227 from nkeywal):
> State: TIMED_WAITING
> Blocked count: 360
> Waited count: 359
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:702)
> org.apache.hadoop.ipc.Client$Connection.run(Client.java:744)
> Thread 272 (Master:0;localhost,39664,1320187706355-EventThread):
> State: WAITING
> Blocked count: 0
> Waited count: 4
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@107b954b
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 271 (Master:0;localhost,39664,1320187706355-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 152 (Master:0;localhost,39664,1320187706355):
> State: WAITING
> Blocked count: 217
> Waited count: 174
> Waiting on org.apache.hadoop.hbase.zookeeper.RootRegionTracker@6621477c
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:131)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:104)
> org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRoot(CatalogTracker.java:277)
> org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:523)
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:468)
> org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)
> java.lang.Thread.run(Thread.java:662)
> Thread 165 (LruBlockCache.EvictionThread):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread@3e9d7b56
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:593)
> java.lang.Thread.run(Thread.java:662)
> Thread 151 (main-EventThread):
> State: WAITING
> Blocked count: 40
> Waited count: 39
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22531380
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 150 (main-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 55
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 149 (IPC Server handler 4 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 148 (IPC Server handler 3 on 39664):
> State: WAITING
> Blocked count: 7
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 147 (IPC Server handler 2 on 39664):
> State: WAITING
> Blocked count: 11
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 146 (IPC Server handler 1 on 39664):
> State: WAITING
> Blocked count: 6
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 145 (IPC Server handler 0 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1970
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 133 (IPC Server listener on 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:580)
> Thread 144 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:754)
> Thread 143 (IPC Reader 9 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 142 (IPC Reader 8 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 141 (IPC Reader 7 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 140 (IPC Reader 6 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 139 (IPC Reader 5 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 138 (IPC Reader 4 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 137 (IPC Reader 3 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 136 (IPC Reader 2 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 135 (IPC Reader 1 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 134 (IPC Reader 0 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 124 (LeaseChecker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1252)
> java.lang.Thread.run(Thread.java:662)
> Thread 121 (ProcessThread:-1):
> State: WAITING
> Blocked count: 0
> Waited count: 269
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f5d84e0
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:103)
> Thread 120 (SyncThread:0):
> State: WAITING
> Blocked count: 2
> Waited count: 252
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b25b27c
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:94)
> Thread 119 (SessionTracker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 207
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:142)
> Thread 118 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:21819):
> State: RUNNABLE
> Blocked count: 38
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:232)
> Thread 116 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@2e6f947b):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 113 (IPC Server handler 2 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 112 (IPC Server handler 1 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 111 (IPC Server handler 0 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 106 (IPC Server listener on 39899):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 108 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 102 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@45d1c3cd):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 109 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6]):
> State: TIMED_WAITING
> Blocked count: 137
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 107 (pool-4-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 105 (Timer-3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 104 (1353906974@qtp-1197660496-1 - Acceptor0 SelectChannelConnector@localhost:59858):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 103 (1185716172@qtp-1197660496-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 101 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 98 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 90 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@605b28c9):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 86 (IPC Server handler 2 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 85 (IPC Server handler 1 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 84 (IPC Server handler 0 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 80 (IPC Server listener on 56098):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 82 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 76 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@2b68989e):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 83 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4]):
> State: TIMED_WAITING
> Blocked count: 149
> Waited count: 290
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 81 (pool-3-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 79 (Timer-2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 78 (348878159@qtp-164967086-1 - Acceptor0 SelectChannelConnector@localhost:36427):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 77 (1905729203@qtp-164967086-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 75 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 72 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 61 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@6709da93):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 60 (IPC Server handler 2 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 59 (IPC Server handler 1 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 58 (IPC Server handler 0 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 53 (IPC Server listener on 40388):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 55 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 49 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@48baa31b):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 57 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2]):
> State: TIMED_WAITING
> Blocked count: 148
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 54 (pool-2-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 52 (Timer-1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 51 (1031841975@qtp-1068922666-1 - Acceptor0 SelectChannelConnector@localhost:56535):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 50 (1191055321@qtp-1068922666-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 48 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 45 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 31 (IPC Server handler 9 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 30 (IPC Server handler 8 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 29 (IPC Server handler 7 on 52227):
> State: WAITING
> Blocked count: 2
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 28 (IPC Server handler 6 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 27 (IPC Server handler 5 on 52227):
> State: WAITING
> Blocked count: 4
> Waited count: 68
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 26 (IPC Server handler 4 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 25 (IPC Server handler 3 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 24 (IPC Server handler 2 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 23 (IPC Server handler 1 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 22 (IPC Server handler 0 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 15 (IPC Server listener on 52227):
> State: RUNNABLE
> Blocked count: 43
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 17 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 21 (Timer-0):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 17
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 20 (245022965@qtp-589734556-1 - Acceptor0 SelectChannelConnector@localhost:33079):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 19 (1821050251@qtp-589734556-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 16 (pool-1-thread-1):
> State: RUNNABLE
> Blocked count: 45
> Waited count: 44
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 14 (org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@620fa83):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65)
> java.lang.Thread.run(Thread.java:662)
> Thread 13 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@d5d4de6):
> State: TIMED_WAITING
> Blocked count: 2
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2689)
> java.lang.Thread.run(Thread.java:662)
> Thread 12 (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@73d32e45):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 220
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:368)
> java.lang.Thread.run(Thread.java:662)
> Thread 11 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@19be4777):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 89
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2666)
> java.lang.Thread.run(Thread.java:662)
> Thread 10 (org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@658f7386):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 2
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
> java.lang.Thread.run(Thread.java:662)
> Thread 4 (Signal Dispatcher):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> Thread 3 (Finalizer):
> State: WAITING
> Blocked count: 9
> Waited count: 10
> Waiting on java.lang.ref.ReferenceQueue$Lock@3d1a70a7
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> Thread 2 (Reference Handler):
> State: WAITING
> Blocked count: 11
> Waited count: 12
> Waiting on java.lang.ref.Reference$Lock@270d75a3
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> Thread 1 (main):
> State: RUNNABLE
> Blocked count: 104
> Waited count: 97
> Stack:
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> java.lang.reflect.Method.invoke(Method.java:597)
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
> org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4724) TestAdmin hangs randomly in trunk
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13141979#comment-13141979 ]
nkeywal commented on HBASE-4724:
--------------------------------
I believe that this log:
{noformat}
org.apache.hadoop.hbase.ipc.HBaseRPC$VersionMismatch: Protocol org.apache.hadoop.hbase.ipc.HRegionInterface version mismatch. (client = 28, server = 29)
[...]
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1150)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1145)
at org.apache.hadoop.hbase.master.AssignmentManager.assignRoot(AssignmentManager.java:1821)
at org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:522)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:468)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)
at java.lang.Thread.run(Thread.java:662)
{noformat}
could explain why we have this in the final threads dump:
{noformat}
Thread 149 (Master:0;localhost,36968,1320216715828):
State: WAITING
Blocked count: 148
Waited count: 148
Waiting on org.apache.hadoop.hbase.zookeeper.RootRegionTracker@1d4f0fb4
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:131)
org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:104)
org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRoot(CatalogTracker.java:277)
org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:523)
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:468)
org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)
java.lang.Thread.run(Thread.java:662)
{noformat}
> TestAdmin hangs randomly in trunk
> ---------------------------------
>
> Key: HBASE-4724
> URL: https://issues.apache.org/jira/browse/HBASE-4724
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Priority: Critical
>
> fom the logs in my env
> {noformat}
> 2011-11-01 15:48:40,744 WARN [Master:0;localhost,39664,1320187706355] master.AssignmentManager(1471): Failed assignment of -ROOT-,,0.70236052 to localhost,44046,1320187706849, trying to assign elsewhere instead; retry=1
> org.apache.hadoop.hbase.ipc.HBaseRPC$VersionMismatch: Protocol org.apache.hadoop.hbase.ipc.HRegionInterface version mismatch. (client = 28, server = 29)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:185)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:300)
> {noformat}
> Anyway, after this the logs finishes with:
> {noformat}
> 2011-11-01 15:54:35,132 INFO [Master:0;localhost,39664,1320187706355.oldLogCleaner] hbase.Chore(80): Master:0;localhost,39664,1320187706355.oldLogCleaner exiting
> Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on Master:0;localhost,39664,1320187706355
> {noformat}
> it's in
> {noformat}
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> {noformat}
> So that's at least why adding a timeout wont help and may be why it does not end at all. Adding a maximum retry to Threads#threadDumpingIsAlive could help.
> I also wonder if the root cause of the non ending is my modif on the wal, with some threads surprised to have updates that were not written in the wal. Here is the full stack dump:
> {noformat}
> Thread 354 (IPC Client (47) connection to localhost/127.0.0.1:52227 from nkeywal):
> State: TIMED_WAITING
> Blocked count: 360
> Waited count: 359
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:702)
> org.apache.hadoop.ipc.Client$Connection.run(Client.java:744)
> Thread 272 (Master:0;localhost,39664,1320187706355-EventThread):
> State: WAITING
> Blocked count: 0
> Waited count: 4
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@107b954b
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 271 (Master:0;localhost,39664,1320187706355-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 152 (Master:0;localhost,39664,1320187706355):
> State: WAITING
> Blocked count: 217
> Waited count: 174
> Waiting on org.apache.hadoop.hbase.zookeeper.RootRegionTracker@6621477c
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:131)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:104)
> org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRoot(CatalogTracker.java:277)
> org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:523)
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:468)
> org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)
> java.lang.Thread.run(Thread.java:662)
> Thread 165 (LruBlockCache.EvictionThread):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread@3e9d7b56
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:593)
> java.lang.Thread.run(Thread.java:662)
> Thread 151 (main-EventThread):
> State: WAITING
> Blocked count: 40
> Waited count: 39
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22531380
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 150 (main-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 55
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 149 (IPC Server handler 4 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 148 (IPC Server handler 3 on 39664):
> State: WAITING
> Blocked count: 7
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 147 (IPC Server handler 2 on 39664):
> State: WAITING
> Blocked count: 11
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 146 (IPC Server handler 1 on 39664):
> State: WAITING
> Blocked count: 6
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 145 (IPC Server handler 0 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1970
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 133 (IPC Server listener on 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:580)
> Thread 144 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:754)
> Thread 143 (IPC Reader 9 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 142 (IPC Reader 8 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 141 (IPC Reader 7 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 140 (IPC Reader 6 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 139 (IPC Reader 5 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 138 (IPC Reader 4 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 137 (IPC Reader 3 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 136 (IPC Reader 2 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 135 (IPC Reader 1 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 134 (IPC Reader 0 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 124 (LeaseChecker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1252)
> java.lang.Thread.run(Thread.java:662)
> Thread 121 (ProcessThread:-1):
> State: WAITING
> Blocked count: 0
> Waited count: 269
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f5d84e0
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:103)
> Thread 120 (SyncThread:0):
> State: WAITING
> Blocked count: 2
> Waited count: 252
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b25b27c
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:94)
> Thread 119 (SessionTracker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 207
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:142)
> Thread 118 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:21819):
> State: RUNNABLE
> Blocked count: 38
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:232)
> Thread 116 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@2e6f947b):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 113 (IPC Server handler 2 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 112 (IPC Server handler 1 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 111 (IPC Server handler 0 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 106 (IPC Server listener on 39899):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 108 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 102 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@45d1c3cd):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 109 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6]):
> State: TIMED_WAITING
> Blocked count: 137
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 107 (pool-4-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 105 (Timer-3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 104 (1353906974@qtp-1197660496-1 - Acceptor0 SelectChannelConnector@localhost:59858):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 103 (1185716172@qtp-1197660496-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 101 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 98 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 90 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@605b28c9):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 86 (IPC Server handler 2 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 85 (IPC Server handler 1 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 84 (IPC Server handler 0 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 80 (IPC Server listener on 56098):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 82 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 76 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@2b68989e):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 83 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4]):
> State: TIMED_WAITING
> Blocked count: 149
> Waited count: 290
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 81 (pool-3-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 79 (Timer-2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 78 (348878159@qtp-164967086-1 - Acceptor0 SelectChannelConnector@localhost:36427):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 77 (1905729203@qtp-164967086-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 75 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 72 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 61 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@6709da93):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 60 (IPC Server handler 2 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 59 (IPC Server handler 1 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 58 (IPC Server handler 0 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 53 (IPC Server listener on 40388):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 55 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 49 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@48baa31b):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 57 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2]):
> State: TIMED_WAITING
> Blocked count: 148
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 54 (pool-2-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 52 (Timer-1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 51 (1031841975@qtp-1068922666-1 - Acceptor0 SelectChannelConnector@localhost:56535):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 50 (1191055321@qtp-1068922666-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 48 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 45 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 31 (IPC Server handler 9 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 30 (IPC Server handler 8 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 29 (IPC Server handler 7 on 52227):
> State: WAITING
> Blocked count: 2
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 28 (IPC Server handler 6 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 27 (IPC Server handler 5 on 52227):
> State: WAITING
> Blocked count: 4
> Waited count: 68
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 26 (IPC Server handler 4 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 25 (IPC Server handler 3 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 24 (IPC Server handler 2 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 23 (IPC Server handler 1 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 22 (IPC Server handler 0 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 15 (IPC Server listener on 52227):
> State: RUNNABLE
> Blocked count: 43
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 17 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 21 (Timer-0):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 17
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 20 (245022965@qtp-589734556-1 - Acceptor0 SelectChannelConnector@localhost:33079):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 19 (1821050251@qtp-589734556-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 16 (pool-1-thread-1):
> State: RUNNABLE
> Blocked count: 45
> Waited count: 44
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 14 (org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@620fa83):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65)
> java.lang.Thread.run(Thread.java:662)
> Thread 13 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@d5d4de6):
> State: TIMED_WAITING
> Blocked count: 2
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2689)
> java.lang.Thread.run(Thread.java:662)
> Thread 12 (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@73d32e45):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 220
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:368)
> java.lang.Thread.run(Thread.java:662)
> Thread 11 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@19be4777):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 89
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2666)
> java.lang.Thread.run(Thread.java:662)
> Thread 10 (org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@658f7386):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 2
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
> java.lang.Thread.run(Thread.java:662)
> Thread 4 (Signal Dispatcher):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> Thread 3 (Finalizer):
> State: WAITING
> Blocked count: 9
> Waited count: 10
> Waiting on java.lang.ref.ReferenceQueue$Lock@3d1a70a7
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> Thread 2 (Reference Handler):
> State: WAITING
> Blocked count: 11
> Waited count: 12
> Waiting on java.lang.ref.Reference$Lock@270d75a3
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> Thread 1 (main):
> State: RUNNABLE
> Blocked count: 104
> Waited count: 97
> Stack:
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> java.lang.reflect.Method.invoke(Method.java:597)
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
> org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4724) TestAdmin hangs randomly in trunk
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142217#comment-13142217 ]
stack commented on HBASE-4724:
------------------------------
Oh, you need to do --no-prefix when making patches for patch-build.
> TestAdmin hangs randomly in trunk
> ---------------------------------
>
> Key: HBASE-4724
> URL: https://issues.apache.org/jira/browse/HBASE-4724
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 20111102_4724_TestAdmin.patch, 20111102_4724_TestAdmin.v2.patch
>
>
> fom the logs in my env
> {noformat}
> 2011-11-01 15:48:40,744 WARN [Master:0;localhost,39664,1320187706355] master.AssignmentManager(1471): Failed assignment of -ROOT-,,0.70236052 to localhost,44046,1320187706849, trying to assign elsewhere instead; retry=1
> org.apache.hadoop.hbase.ipc.HBaseRPC$VersionMismatch: Protocol org.apache.hadoop.hbase.ipc.HRegionInterface version mismatch. (client = 28, server = 29)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:185)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:300)
> {noformat}
> Anyway, after this the logs finishes with:
> {noformat}
> 2011-11-01 15:54:35,132 INFO [Master:0;localhost,39664,1320187706355.oldLogCleaner] hbase.Chore(80): Master:0;localhost,39664,1320187706355.oldLogCleaner exiting
> Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on Master:0;localhost,39664,1320187706355
> {noformat}
> it's in
> {noformat}
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> {noformat}
> So that's at least why adding a timeout wont help and may be why it does not end at all. Adding a maximum retry to Threads#threadDumpingIsAlive could help.
> I also wonder if the root cause of the non ending is my modif on the wal, with some threads surprised to have updates that were not written in the wal. Here is the full stack dump:
> {noformat}
> Thread 354 (IPC Client (47) connection to localhost/127.0.0.1:52227 from nkeywal):
> State: TIMED_WAITING
> Blocked count: 360
> Waited count: 359
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:702)
> org.apache.hadoop.ipc.Client$Connection.run(Client.java:744)
> Thread 272 (Master:0;localhost,39664,1320187706355-EventThread):
> State: WAITING
> Blocked count: 0
> Waited count: 4
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@107b954b
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 271 (Master:0;localhost,39664,1320187706355-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 152 (Master:0;localhost,39664,1320187706355):
> State: WAITING
> Blocked count: 217
> Waited count: 174
> Waiting on org.apache.hadoop.hbase.zookeeper.RootRegionTracker@6621477c
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:131)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:104)
> org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRoot(CatalogTracker.java:277)
> org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:523)
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:468)
> org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)
> java.lang.Thread.run(Thread.java:662)
> Thread 165 (LruBlockCache.EvictionThread):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread@3e9d7b56
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:593)
> java.lang.Thread.run(Thread.java:662)
> Thread 151 (main-EventThread):
> State: WAITING
> Blocked count: 40
> Waited count: 39
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22531380
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 150 (main-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 55
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 149 (IPC Server handler 4 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 148 (IPC Server handler 3 on 39664):
> State: WAITING
> Blocked count: 7
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 147 (IPC Server handler 2 on 39664):
> State: WAITING
> Blocked count: 11
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 146 (IPC Server handler 1 on 39664):
> State: WAITING
> Blocked count: 6
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 145 (IPC Server handler 0 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1970
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 133 (IPC Server listener on 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:580)
> Thread 144 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:754)
> Thread 143 (IPC Reader 9 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 142 (IPC Reader 8 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 141 (IPC Reader 7 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 140 (IPC Reader 6 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 139 (IPC Reader 5 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 138 (IPC Reader 4 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 137 (IPC Reader 3 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 136 (IPC Reader 2 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 135 (IPC Reader 1 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 134 (IPC Reader 0 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 124 (LeaseChecker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1252)
> java.lang.Thread.run(Thread.java:662)
> Thread 121 (ProcessThread:-1):
> State: WAITING
> Blocked count: 0
> Waited count: 269
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f5d84e0
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:103)
> Thread 120 (SyncThread:0):
> State: WAITING
> Blocked count: 2
> Waited count: 252
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b25b27c
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:94)
> Thread 119 (SessionTracker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 207
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:142)
> Thread 118 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:21819):
> State: RUNNABLE
> Blocked count: 38
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:232)
> Thread 116 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@2e6f947b):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 113 (IPC Server handler 2 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 112 (IPC Server handler 1 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 111 (IPC Server handler 0 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 106 (IPC Server listener on 39899):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 108 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 102 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@45d1c3cd):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 109 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6]):
> State: TIMED_WAITING
> Blocked count: 137
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 107 (pool-4-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 105 (Timer-3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 104 (1353906974@qtp-1197660496-1 - Acceptor0 SelectChannelConnector@localhost:59858):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 103 (1185716172@qtp-1197660496-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 101 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 98 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 90 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@605b28c9):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 86 (IPC Server handler 2 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 85 (IPC Server handler 1 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 84 (IPC Server handler 0 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 80 (IPC Server listener on 56098):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 82 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 76 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@2b68989e):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 83 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4]):
> State: TIMED_WAITING
> Blocked count: 149
> Waited count: 290
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 81 (pool-3-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 79 (Timer-2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 78 (348878159@qtp-164967086-1 - Acceptor0 SelectChannelConnector@localhost:36427):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 77 (1905729203@qtp-164967086-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 75 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 72 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 61 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@6709da93):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 60 (IPC Server handler 2 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 59 (IPC Server handler 1 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 58 (IPC Server handler 0 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 53 (IPC Server listener on 40388):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 55 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 49 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@48baa31b):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 57 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2]):
> State: TIMED_WAITING
> Blocked count: 148
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 54 (pool-2-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 52 (Timer-1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 51 (1031841975@qtp-1068922666-1 - Acceptor0 SelectChannelConnector@localhost:56535):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 50 (1191055321@qtp-1068922666-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 48 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 45 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 31 (IPC Server handler 9 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 30 (IPC Server handler 8 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 29 (IPC Server handler 7 on 52227):
> State: WAITING
> Blocked count: 2
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 28 (IPC Server handler 6 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 27 (IPC Server handler 5 on 52227):
> State: WAITING
> Blocked count: 4
> Waited count: 68
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 26 (IPC Server handler 4 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 25 (IPC Server handler 3 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 24 (IPC Server handler 2 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 23 (IPC Server handler 1 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 22 (IPC Server handler 0 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 15 (IPC Server listener on 52227):
> State: RUNNABLE
> Blocked count: 43
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 17 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 21 (Timer-0):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 17
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 20 (245022965@qtp-589734556-1 - Acceptor0 SelectChannelConnector@localhost:33079):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 19 (1821050251@qtp-589734556-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 16 (pool-1-thread-1):
> State: RUNNABLE
> Blocked count: 45
> Waited count: 44
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 14 (org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@620fa83):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65)
> java.lang.Thread.run(Thread.java:662)
> Thread 13 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@d5d4de6):
> State: TIMED_WAITING
> Blocked count: 2
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2689)
> java.lang.Thread.run(Thread.java:662)
> Thread 12 (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@73d32e45):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 220
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:368)
> java.lang.Thread.run(Thread.java:662)
> Thread 11 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@19be4777):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 89
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2666)
> java.lang.Thread.run(Thread.java:662)
> Thread 10 (org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@658f7386):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 2
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
> java.lang.Thread.run(Thread.java:662)
> Thread 4 (Signal Dispatcher):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> Thread 3 (Finalizer):
> State: WAITING
> Blocked count: 9
> Waited count: 10
> Waiting on java.lang.ref.ReferenceQueue$Lock@3d1a70a7
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> Thread 2 (Reference Handler):
> State: WAITING
> Blocked count: 11
> Waited count: 12
> Waiting on java.lang.ref.Reference$Lock@270d75a3
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> Thread 1 (main):
> State: RUNNABLE
> Blocked count: 104
> Waited count: 97
> Stack:
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> java.lang.reflect.Method.invoke(Method.java:597)
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
> org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4724) TestAdmin hangs randomly in trunk
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142526#comment-13142526 ]
Hudson commented on HBASE-4724:
-------------------------------
Integrated in HBase-0.92 #101 (See [https://builds.apache.org/job/HBase-0.92/101/])
HBASE-4724 TestAdmin hangs randomly in trunk
stack :
Files :
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java
> TestAdmin hangs randomly in trunk
> ---------------------------------
>
> Key: HBASE-4724
> URL: https://issues.apache.org/jira/browse/HBASE-4724
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 20111102_4724_TestAdmin.patch, 20111102_4724_TestAdmin.v2.patch
>
>
> fom the logs in my env
> {noformat}
> 2011-11-01 15:48:40,744 WARN [Master:0;localhost,39664,1320187706355] master.AssignmentManager(1471): Failed assignment of -ROOT-,,0.70236052 to localhost,44046,1320187706849, trying to assign elsewhere instead; retry=1
> org.apache.hadoop.hbase.ipc.HBaseRPC$VersionMismatch: Protocol org.apache.hadoop.hbase.ipc.HRegionInterface version mismatch. (client = 28, server = 29)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:185)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:300)
> {noformat}
> Anyway, after this the logs finishes with:
> {noformat}
> 2011-11-01 15:54:35,132 INFO [Master:0;localhost,39664,1320187706355.oldLogCleaner] hbase.Chore(80): Master:0;localhost,39664,1320187706355.oldLogCleaner exiting
> Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on Master:0;localhost,39664,1320187706355
> {noformat}
> it's in
> {noformat}
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> {noformat}
> So that's at least why adding a timeout wont help and may be why it does not end at all. Adding a maximum retry to Threads#threadDumpingIsAlive could help.
> I also wonder if the root cause of the non ending is my modif on the wal, with some threads surprised to have updates that were not written in the wal. Here is the full stack dump:
> {noformat}
> Thread 354 (IPC Client (47) connection to localhost/127.0.0.1:52227 from nkeywal):
> State: TIMED_WAITING
> Blocked count: 360
> Waited count: 359
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:702)
> org.apache.hadoop.ipc.Client$Connection.run(Client.java:744)
> Thread 272 (Master:0;localhost,39664,1320187706355-EventThread):
> State: WAITING
> Blocked count: 0
> Waited count: 4
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@107b954b
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 271 (Master:0;localhost,39664,1320187706355-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 152 (Master:0;localhost,39664,1320187706355):
> State: WAITING
> Blocked count: 217
> Waited count: 174
> Waiting on org.apache.hadoop.hbase.zookeeper.RootRegionTracker@6621477c
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:131)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:104)
> org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRoot(CatalogTracker.java:277)
> org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:523)
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:468)
> org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)
> java.lang.Thread.run(Thread.java:662)
> Thread 165 (LruBlockCache.EvictionThread):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread@3e9d7b56
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:593)
> java.lang.Thread.run(Thread.java:662)
> Thread 151 (main-EventThread):
> State: WAITING
> Blocked count: 40
> Waited count: 39
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22531380
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 150 (main-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 55
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 149 (IPC Server handler 4 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 148 (IPC Server handler 3 on 39664):
> State: WAITING
> Blocked count: 7
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 147 (IPC Server handler 2 on 39664):
> State: WAITING
> Blocked count: 11
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 146 (IPC Server handler 1 on 39664):
> State: WAITING
> Blocked count: 6
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 145 (IPC Server handler 0 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1970
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 133 (IPC Server listener on 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:580)
> Thread 144 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:754)
> Thread 143 (IPC Reader 9 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 142 (IPC Reader 8 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 141 (IPC Reader 7 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 140 (IPC Reader 6 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 139 (IPC Reader 5 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 138 (IPC Reader 4 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 137 (IPC Reader 3 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 136 (IPC Reader 2 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 135 (IPC Reader 1 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 134 (IPC Reader 0 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 124 (LeaseChecker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1252)
> java.lang.Thread.run(Thread.java:662)
> Thread 121 (ProcessThread:-1):
> State: WAITING
> Blocked count: 0
> Waited count: 269
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f5d84e0
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:103)
> Thread 120 (SyncThread:0):
> State: WAITING
> Blocked count: 2
> Waited count: 252
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b25b27c
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:94)
> Thread 119 (SessionTracker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 207
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:142)
> Thread 118 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:21819):
> State: RUNNABLE
> Blocked count: 38
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:232)
> Thread 116 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@2e6f947b):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 113 (IPC Server handler 2 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 112 (IPC Server handler 1 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 111 (IPC Server handler 0 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 106 (IPC Server listener on 39899):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 108 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 102 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@45d1c3cd):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 109 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6]):
> State: TIMED_WAITING
> Blocked count: 137
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 107 (pool-4-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 105 (Timer-3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 104 (1353906974@qtp-1197660496-1 - Acceptor0 SelectChannelConnector@localhost:59858):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 103 (1185716172@qtp-1197660496-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 101 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 98 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 90 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@605b28c9):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 86 (IPC Server handler 2 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 85 (IPC Server handler 1 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 84 (IPC Server handler 0 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 80 (IPC Server listener on 56098):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 82 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 76 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@2b68989e):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 83 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4]):
> State: TIMED_WAITING
> Blocked count: 149
> Waited count: 290
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 81 (pool-3-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 79 (Timer-2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 78 (348878159@qtp-164967086-1 - Acceptor0 SelectChannelConnector@localhost:36427):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 77 (1905729203@qtp-164967086-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 75 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 72 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 61 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@6709da93):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 60 (IPC Server handler 2 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 59 (IPC Server handler 1 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 58 (IPC Server handler 0 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 53 (IPC Server listener on 40388):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 55 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 49 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@48baa31b):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 57 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2]):
> State: TIMED_WAITING
> Blocked count: 148
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 54 (pool-2-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 52 (Timer-1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 51 (1031841975@qtp-1068922666-1 - Acceptor0 SelectChannelConnector@localhost:56535):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 50 (1191055321@qtp-1068922666-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 48 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 45 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 31 (IPC Server handler 9 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 30 (IPC Server handler 8 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 29 (IPC Server handler 7 on 52227):
> State: WAITING
> Blocked count: 2
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 28 (IPC Server handler 6 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 27 (IPC Server handler 5 on 52227):
> State: WAITING
> Blocked count: 4
> Waited count: 68
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 26 (IPC Server handler 4 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 25 (IPC Server handler 3 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 24 (IPC Server handler 2 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 23 (IPC Server handler 1 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 22 (IPC Server handler 0 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 15 (IPC Server listener on 52227):
> State: RUNNABLE
> Blocked count: 43
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 17 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 21 (Timer-0):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 17
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 20 (245022965@qtp-589734556-1 - Acceptor0 SelectChannelConnector@localhost:33079):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 19 (1821050251@qtp-589734556-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 16 (pool-1-thread-1):
> State: RUNNABLE
> Blocked count: 45
> Waited count: 44
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 14 (org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@620fa83):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65)
> java.lang.Thread.run(Thread.java:662)
> Thread 13 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@d5d4de6):
> State: TIMED_WAITING
> Blocked count: 2
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2689)
> java.lang.Thread.run(Thread.java:662)
> Thread 12 (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@73d32e45):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 220
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:368)
> java.lang.Thread.run(Thread.java:662)
> Thread 11 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@19be4777):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 89
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2666)
> java.lang.Thread.run(Thread.java:662)
> Thread 10 (org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@658f7386):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 2
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
> java.lang.Thread.run(Thread.java:662)
> Thread 4 (Signal Dispatcher):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> Thread 3 (Finalizer):
> State: WAITING
> Blocked count: 9
> Waited count: 10
> Waiting on java.lang.ref.ReferenceQueue$Lock@3d1a70a7
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> Thread 2 (Reference Handler):
> State: WAITING
> Blocked count: 11
> Waited count: 12
> Waiting on java.lang.ref.Reference$Lock@270d75a3
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> Thread 1 (main):
> State: RUNNABLE
> Blocked count: 104
> Waited count: 97
> Stack:
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> java.lang.reflect.Method.invoke(Method.java:597)
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
> org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4724) TestAdmin hangs randomly in trunk
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142251#comment-13142251 ]
stack commented on HBASE-4724:
------------------------------
Let me look at the above.
I reran the TestAdmin a few times and got this again:
{code}
-------------------------------------------------------------------------------
Test set: org.apache.hadoop.hbase.client.TestAdmin
-------------------------------------------------------------------------------
Tests run: 33, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 509.79 sec <<< FAILURE!
testEnableDisableAddColumnDeleteColumn(org.apache.hadoop.hbase.client.TestAdmin) Time elapsed: 0.841 sec <<< ERROR!
org.apache.hadoop.hbase.TableNotEnabledException: org.apache.hadoop.hbase.TableNotEnabledException: testMasterAdmin
{code}
You think some of the cuts in timers too aggressive still?
> TestAdmin hangs randomly in trunk
> ---------------------------------
>
> Key: HBASE-4724
> URL: https://issues.apache.org/jira/browse/HBASE-4724
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 20111102_4724_TestAdmin.patch, 20111102_4724_TestAdmin.v2.patch
>
>
> fom the logs in my env
> {noformat}
> 2011-11-01 15:48:40,744 WARN [Master:0;localhost,39664,1320187706355] master.AssignmentManager(1471): Failed assignment of -ROOT-,,0.70236052 to localhost,44046,1320187706849, trying to assign elsewhere instead; retry=1
> org.apache.hadoop.hbase.ipc.HBaseRPC$VersionMismatch: Protocol org.apache.hadoop.hbase.ipc.HRegionInterface version mismatch. (client = 28, server = 29)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:185)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:300)
> {noformat}
> Anyway, after this the logs finishes with:
> {noformat}
> 2011-11-01 15:54:35,132 INFO [Master:0;localhost,39664,1320187706355.oldLogCleaner] hbase.Chore(80): Master:0;localhost,39664,1320187706355.oldLogCleaner exiting
> Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on Master:0;localhost,39664,1320187706355
> {noformat}
> it's in
> {noformat}
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> {noformat}
> So that's at least why adding a timeout wont help and may be why it does not end at all. Adding a maximum retry to Threads#threadDumpingIsAlive could help.
> I also wonder if the root cause of the non ending is my modif on the wal, with some threads surprised to have updates that were not written in the wal. Here is the full stack dump:
> {noformat}
> Thread 354 (IPC Client (47) connection to localhost/127.0.0.1:52227 from nkeywal):
> State: TIMED_WAITING
> Blocked count: 360
> Waited count: 359
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:702)
> org.apache.hadoop.ipc.Client$Connection.run(Client.java:744)
> Thread 272 (Master:0;localhost,39664,1320187706355-EventThread):
> State: WAITING
> Blocked count: 0
> Waited count: 4
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@107b954b
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 271 (Master:0;localhost,39664,1320187706355-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 152 (Master:0;localhost,39664,1320187706355):
> State: WAITING
> Blocked count: 217
> Waited count: 174
> Waiting on org.apache.hadoop.hbase.zookeeper.RootRegionTracker@6621477c
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:131)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:104)
> org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRoot(CatalogTracker.java:277)
> org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:523)
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:468)
> org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)
> java.lang.Thread.run(Thread.java:662)
> Thread 165 (LruBlockCache.EvictionThread):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread@3e9d7b56
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:593)
> java.lang.Thread.run(Thread.java:662)
> Thread 151 (main-EventThread):
> State: WAITING
> Blocked count: 40
> Waited count: 39
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22531380
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 150 (main-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 55
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 149 (IPC Server handler 4 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 148 (IPC Server handler 3 on 39664):
> State: WAITING
> Blocked count: 7
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 147 (IPC Server handler 2 on 39664):
> State: WAITING
> Blocked count: 11
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 146 (IPC Server handler 1 on 39664):
> State: WAITING
> Blocked count: 6
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 145 (IPC Server handler 0 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1970
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 133 (IPC Server listener on 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:580)
> Thread 144 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:754)
> Thread 143 (IPC Reader 9 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 142 (IPC Reader 8 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 141 (IPC Reader 7 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 140 (IPC Reader 6 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 139 (IPC Reader 5 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 138 (IPC Reader 4 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 137 (IPC Reader 3 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 136 (IPC Reader 2 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 135 (IPC Reader 1 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 134 (IPC Reader 0 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 124 (LeaseChecker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1252)
> java.lang.Thread.run(Thread.java:662)
> Thread 121 (ProcessThread:-1):
> State: WAITING
> Blocked count: 0
> Waited count: 269
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f5d84e0
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:103)
> Thread 120 (SyncThread:0):
> State: WAITING
> Blocked count: 2
> Waited count: 252
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b25b27c
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:94)
> Thread 119 (SessionTracker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 207
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:142)
> Thread 118 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:21819):
> State: RUNNABLE
> Blocked count: 38
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:232)
> Thread 116 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@2e6f947b):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 113 (IPC Server handler 2 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 112 (IPC Server handler 1 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 111 (IPC Server handler 0 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 106 (IPC Server listener on 39899):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 108 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 102 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@45d1c3cd):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 109 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6]):
> State: TIMED_WAITING
> Blocked count: 137
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 107 (pool-4-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 105 (Timer-3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 104 (1353906974@qtp-1197660496-1 - Acceptor0 SelectChannelConnector@localhost:59858):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 103 (1185716172@qtp-1197660496-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 101 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 98 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 90 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@605b28c9):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 86 (IPC Server handler 2 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 85 (IPC Server handler 1 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 84 (IPC Server handler 0 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 80 (IPC Server listener on 56098):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 82 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 76 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@2b68989e):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 83 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4]):
> State: TIMED_WAITING
> Blocked count: 149
> Waited count: 290
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 81 (pool-3-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 79 (Timer-2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 78 (348878159@qtp-164967086-1 - Acceptor0 SelectChannelConnector@localhost:36427):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 77 (1905729203@qtp-164967086-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 75 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 72 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 61 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@6709da93):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 60 (IPC Server handler 2 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 59 (IPC Server handler 1 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 58 (IPC Server handler 0 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 53 (IPC Server listener on 40388):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 55 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 49 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@48baa31b):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 57 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2]):
> State: TIMED_WAITING
> Blocked count: 148
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 54 (pool-2-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 52 (Timer-1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 51 (1031841975@qtp-1068922666-1 - Acceptor0 SelectChannelConnector@localhost:56535):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 50 (1191055321@qtp-1068922666-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 48 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 45 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 31 (IPC Server handler 9 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 30 (IPC Server handler 8 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 29 (IPC Server handler 7 on 52227):
> State: WAITING
> Blocked count: 2
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 28 (IPC Server handler 6 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 27 (IPC Server handler 5 on 52227):
> State: WAITING
> Blocked count: 4
> Waited count: 68
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 26 (IPC Server handler 4 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 25 (IPC Server handler 3 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 24 (IPC Server handler 2 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 23 (IPC Server handler 1 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 22 (IPC Server handler 0 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 15 (IPC Server listener on 52227):
> State: RUNNABLE
> Blocked count: 43
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 17 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 21 (Timer-0):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 17
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 20 (245022965@qtp-589734556-1 - Acceptor0 SelectChannelConnector@localhost:33079):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 19 (1821050251@qtp-589734556-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 16 (pool-1-thread-1):
> State: RUNNABLE
> Blocked count: 45
> Waited count: 44
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 14 (org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@620fa83):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65)
> java.lang.Thread.run(Thread.java:662)
> Thread 13 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@d5d4de6):
> State: TIMED_WAITING
> Blocked count: 2
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2689)
> java.lang.Thread.run(Thread.java:662)
> Thread 12 (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@73d32e45):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 220
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:368)
> java.lang.Thread.run(Thread.java:662)
> Thread 11 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@19be4777):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 89
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2666)
> java.lang.Thread.run(Thread.java:662)
> Thread 10 (org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@658f7386):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 2
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
> java.lang.Thread.run(Thread.java:662)
> Thread 4 (Signal Dispatcher):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> Thread 3 (Finalizer):
> State: WAITING
> Blocked count: 9
> Waited count: 10
> Waiting on java.lang.ref.ReferenceQueue$Lock@3d1a70a7
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> Thread 2 (Reference Handler):
> State: WAITING
> Blocked count: 11
> Waited count: 12
> Waiting on java.lang.ref.Reference$Lock@270d75a3
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> Thread 1 (main):
> State: RUNNABLE
> Blocked count: 104
> Waited count: 97
> Stack:
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> java.lang.reflect.Method.invoke(Method.java:597)
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
> org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4724) TestAdmin hangs randomly in trunk
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4724:
-------------------------
Description:
fom the logs in my env
{noformat}
2011-11-01 15:48:40,744 WARN [Master:0;localhost,39664,1320187706355] master.AssignmentManager(1471): Failed assignment of -ROOT-,,0.70236052 to localhost,44046,1320187706849, trying to assign elsewhere instead; retry=1
org.apache.hadoop.hbase.ipc.HBaseRPC$VersionMismatch: Protocol org.apache.hadoop.hbase.ipc.HRegionInterface version mismatch. (client = 28, server = 29)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:185)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:300)
{noformat}
Anyway, after this the logs finishes with:
{noformat}
2011-11-01 15:54:35,132 INFO [Master:0;localhost,39664,1320187706355.oldLogCleaner] hbase.Chore(80): Master:0;localhost,39664,1320187706355.oldLogCleaner exiting
Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on Master:0;localhost,39664,1320187706355
{noformat}
it's in
{noformat}
sun.management.ThreadImpl.getThreadInfo1(Native Method)
sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
{noformat}
So that's at least why adding a timeout wont help and may be why it does not end at all. Adding a maximum retry to Threads#threadDumpingIsAlive could help.
I also wonder if the root cause of the non ending is my modif on the wal, with some threads surprised to have updates that were not written in the wal. Here is the full stack dump:
{noformat}
Thread 354 (IPC Client (47) connection to localhost/127.0.0.1:52227 from nkeywal):
State: TIMED_WAITING
Blocked count: 360
Waited count: 359
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:702)
org.apache.hadoop.ipc.Client$Connection.run(Client.java:744)
Thread 272 (Master:0;localhost,39664,1320187706355-EventThread):
State: WAITING
Blocked count: 0
Waited count: 4
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@107b954b
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
Thread 271 (Master:0;localhost,39664,1320187706355-SendThread(localhost:21819)):
State: RUNNABLE
Blocked count: 2
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
Thread 152 (Master:0;localhost,39664,1320187706355):
State: WAITING
Blocked count: 217
Waited count: 174
Waiting on org.apache.hadoop.hbase.zookeeper.RootRegionTracker@6621477c
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:131)
org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:104)
org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRoot(CatalogTracker.java:277)
org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:523)
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:468)
org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)
java.lang.Thread.run(Thread.java:662)
Thread 165 (LruBlockCache.EvictionThread):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread@3e9d7b56
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:593)
java.lang.Thread.run(Thread.java:662)
Thread 151 (main-EventThread):
State: WAITING
Blocked count: 40
Waited count: 39
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22531380
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
Thread 150 (main-SendThread(localhost:21819)):
State: RUNNABLE
Blocked count: 55
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
Thread 149 (IPC Server handler 4 on 39664):
State: WAITING
Blocked count: 3
Waited count: 1968
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
Thread 148 (IPC Server handler 3 on 39664):
State: WAITING
Blocked count: 7
Waited count: 1969
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
Thread 147 (IPC Server handler 2 on 39664):
State: WAITING
Blocked count: 11
Waited count: 1968
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
Thread 146 (IPC Server handler 1 on 39664):
State: WAITING
Blocked count: 6
Waited count: 1969
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
Thread 145 (IPC Server handler 0 on 39664):
State: WAITING
Blocked count: 3
Waited count: 1970
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
Thread 133 (IPC Server listener on 39664):
State: RUNNABLE
Blocked count: 1
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:580)
Thread 144 (IPC Server Responder):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:754)
Thread 143 (IPC Reader 9 on port 39664):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 142 (IPC Reader 8 on port 39664):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 141 (IPC Reader 7 on port 39664):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 140 (IPC Reader 6 on port 39664):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 139 (IPC Reader 5 on port 39664):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 138 (IPC Reader 4 on port 39664):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 137 (IPC Reader 3 on port 39664):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 136 (IPC Reader 2 on port 39664):
State: RUNNABLE
Blocked count: 1
Waited count: 1
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 135 (IPC Reader 1 on port 39664):
State: RUNNABLE
Blocked count: 1
Waited count: 1
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 134 (IPC Reader 0 on port 39664):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 124 (LeaseChecker):
State: TIMED_WAITING
Blocked count: 0
Waited count: 407
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1252)
java.lang.Thread.run(Thread.java:662)
Thread 121 (ProcessThread:-1):
State: WAITING
Blocked count: 0
Waited count: 269
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f5d84e0
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:103)
Thread 120 (SyncThread:0):
State: WAITING
Blocked count: 2
Waited count: 252
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b25b27c
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:94)
Thread 119 (SessionTracker):
State: TIMED_WAITING
Blocked count: 0
Waited count: 207
Stack:
java.lang.Object.wait(Native Method)
org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:142)
Thread 118 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:21819):
State: RUNNABLE
Blocked count: 38
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:232)
Thread 116 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@2e6f947b):
State: TIMED_WAITING
Blocked count: 0
Waited count: 407
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
java.lang.Thread.run(Thread.java:662)
Thread 113 (IPC Server handler 2 on 39899):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 112 (IPC Server handler 1 on 39899):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 111 (IPC Server handler 0 on 39899):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 106 (IPC Server listener on 39899):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
Thread 108 (IPC Server Responder):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
Thread 102 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@45d1c3cd):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
java.lang.Thread.run(Thread.java:662)
Thread 109 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6]):
State: TIMED_WAITING
Blocked count: 137
Waited count: 283
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
java.lang.Thread.run(Thread.java:662)
Thread 107 (pool-4-thread-1):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 105 (Timer-3):
State: TIMED_WAITING
Blocked count: 0
Waited count: 15
Stack:
java.lang.Object.wait(Native Method)
java.util.TimerThread.mainLoop(Timer.java:509)
java.util.TimerThread.run(Timer.java:462)
Thread 104 (1353906974@qtp-1197660496-1 - Acceptor0 SelectChannelConnector@localhost:59858):
State: RUNNABLE
Blocked count: 1
Waited count: 1
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Thread 103 (1185716172@qtp-1197660496-0):
State: TIMED_WAITING
Blocked count: 0
Waited count: 8
Stack:
java.lang.Object.wait(Native Method)
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
Thread 101 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6):
State: TIMED_WAITING
Blocked count: 0
Waited count: 1
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
java.lang.Thread.run(Thread.java:662)
Thread 98 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5):
State: TIMED_WAITING
Blocked count: 0
Waited count: 1
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
java.lang.Thread.run(Thread.java:662)
Thread 90 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@605b28c9):
State: TIMED_WAITING
Blocked count: 0
Waited count: 408
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
java.lang.Thread.run(Thread.java:662)
Thread 86 (IPC Server handler 2 on 56098):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 85 (IPC Server handler 1 on 56098):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 84 (IPC Server handler 0 on 56098):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 80 (IPC Server listener on 56098):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
Thread 82 (IPC Server Responder):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
Thread 76 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@2b68989e):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
java.lang.Thread.run(Thread.java:662)
Thread 83 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4]):
State: TIMED_WAITING
Blocked count: 149
Waited count: 290
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
java.lang.Thread.run(Thread.java:662)
Thread 81 (pool-3-thread-1):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 79 (Timer-2):
State: TIMED_WAITING
Blocked count: 0
Waited count: 15
Stack:
java.lang.Object.wait(Native Method)
java.util.TimerThread.mainLoop(Timer.java:509)
java.util.TimerThread.run(Timer.java:462)
Thread 78 (348878159@qtp-164967086-1 - Acceptor0 SelectChannelConnector@localhost:36427):
State: RUNNABLE
Blocked count: 2
Waited count: 1
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Thread 77 (1905729203@qtp-164967086-0):
State: TIMED_WAITING
Blocked count: 0
Waited count: 8
Stack:
java.lang.Object.wait(Native Method)
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
Thread 75 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4):
State: TIMED_WAITING
Blocked count: 0
Waited count: 1
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
java.lang.Thread.run(Thread.java:662)
Thread 72 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3):
State: TIMED_WAITING
Blocked count: 0
Waited count: 1
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
java.lang.Thread.run(Thread.java:662)
Thread 61 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@6709da93):
State: TIMED_WAITING
Blocked count: 0
Waited count: 408
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
java.lang.Thread.run(Thread.java:662)
Thread 60 (IPC Server handler 2 on 40388):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 59 (IPC Server handler 1 on 40388):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 58 (IPC Server handler 0 on 40388):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 53 (IPC Server listener on 40388):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
Thread 55 (IPC Server Responder):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
Thread 49 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@48baa31b):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
java.lang.Thread.run(Thread.java:662)
Thread 57 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2]):
State: TIMED_WAITING
Blocked count: 148
Waited count: 283
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
java.lang.Thread.run(Thread.java:662)
Thread 54 (pool-2-thread-1):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 52 (Timer-1):
State: TIMED_WAITING
Blocked count: 0
Waited count: 15
Stack:
java.lang.Object.wait(Native Method)
java.util.TimerThread.mainLoop(Timer.java:509)
java.util.TimerThread.run(Timer.java:462)
Thread 51 (1031841975@qtp-1068922666-1 - Acceptor0 SelectChannelConnector@localhost:56535):
State: RUNNABLE
Blocked count: 1
Waited count: 1
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Thread 50 (1191055321@qtp-1068922666-0):
State: TIMED_WAITING
Blocked count: 0
Waited count: 8
Stack:
java.lang.Object.wait(Native Method)
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
Thread 48 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2):
State: TIMED_WAITING
Blocked count: 0
Waited count: 1
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
java.lang.Thread.run(Thread.java:662)
Thread 45 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1):
State: TIMED_WAITING
Blocked count: 0
Waited count: 1
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
java.lang.Thread.run(Thread.java:662)
Thread 31 (IPC Server handler 9 on 52227):
State: WAITING
Blocked count: 0
Waited count: 64
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 30 (IPC Server handler 8 on 52227):
State: WAITING
Blocked count: 3
Waited count: 65
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 29 (IPC Server handler 7 on 52227):
State: WAITING
Blocked count: 2
Waited count: 66
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 28 (IPC Server handler 6 on 52227):
State: WAITING
Blocked count: 1
Waited count: 64
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 27 (IPC Server handler 5 on 52227):
State: WAITING
Blocked count: 4
Waited count: 68
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 26 (IPC Server handler 4 on 52227):
State: WAITING
Blocked count: 0
Waited count: 64
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 25 (IPC Server handler 3 on 52227):
State: WAITING
Blocked count: 0
Waited count: 65
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 24 (IPC Server handler 2 on 52227):
State: WAITING
Blocked count: 1
Waited count: 66
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 23 (IPC Server handler 1 on 52227):
State: WAITING
Blocked count: 0
Waited count: 64
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 22 (IPC Server handler 0 on 52227):
State: WAITING
Blocked count: 3
Waited count: 65
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 15 (IPC Server listener on 52227):
State: RUNNABLE
Blocked count: 43
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
Thread 17 (IPC Server Responder):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
Thread 21 (Timer-0):
State: TIMED_WAITING
Blocked count: 1
Waited count: 17
Stack:
java.lang.Object.wait(Native Method)
java.util.TimerThread.mainLoop(Timer.java:509)
java.util.TimerThread.run(Timer.java:462)
Thread 20 (245022965@qtp-589734556-1 - Acceptor0 SelectChannelConnector@localhost:33079):
State: RUNNABLE
Blocked count: 2
Waited count: 1
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Thread 19 (1821050251@qtp-589734556-0):
State: TIMED_WAITING
Blocked count: 0
Waited count: 8
Stack:
java.lang.Object.wait(Native Method)
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
Thread 16 (pool-1-thread-1):
State: RUNNABLE
Blocked count: 45
Waited count: 44
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 14 (org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@620fa83):
State: TIMED_WAITING
Blocked count: 1
Waited count: 147
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65)
java.lang.Thread.run(Thread.java:662)
Thread 13 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@d5d4de6):
State: TIMED_WAITING
Blocked count: 2
Waited count: 147
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2689)
java.lang.Thread.run(Thread.java:662)
Thread 12 (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@73d32e45):
State: TIMED_WAITING
Blocked count: 0
Waited count: 220
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:368)
java.lang.Thread.run(Thread.java:662)
Thread 11 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@19be4777):
State: TIMED_WAITING
Blocked count: 0
Waited count: 89
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2666)
java.lang.Thread.run(Thread.java:662)
Thread 10 (org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@658f7386):
State: TIMED_WAITING
Blocked count: 0
Waited count: 2
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
java.lang.Thread.run(Thread.java:662)
Thread 4 (Signal Dispatcher):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
Thread 3 (Finalizer):
State: WAITING
Blocked count: 9
Waited count: 10
Waiting on java.lang.ref.ReferenceQueue$Lock@3d1a70a7
Stack:
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
Thread 2 (Reference Handler):
State: WAITING
Blocked count: 11
Waited count: 12
Waiting on java.lang.ref.Reference$Lock@270d75a3
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
Thread 1 (main):
State: RUNNABLE
Blocked count: 104
Waited count: 97
Stack:
sun.management.ThreadImpl.getThreadInfo1(Native Method)
sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
org.junit.runners.ParentRunner.run(ParentRunner.java:236)
org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
{noformat}
was:
fom the logs in my env
{noformat}
2011-11-01 15:48:40,744 WARN [Master:0;localhost,39664,1320187706355] master.AssignmentManager(1471): Failed assignment of -ROOT-,,0.70236052 to localhost,44046,1320187706849, trying to assign elsewhere instead; retry=1
org.apache.hadoop.hbase.ipc.HBaseRPC$VersionMismatch: Protocol org.apache.hadoop.hbase.ipc.HRegionInterface version mismatch. (client = 28, server = 29)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:185)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:300)
{noformat}
Anyway, after this the logs finishes with:
{noformat}
2011-11-01 15:54:35,132 INFO [Master:0;localhost,39664,1320187706355.oldLogCleaner] hbase.Chore(80): Master:0;localhost,39664,1320187706355.oldLogCleaner exiting
Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on Master:0;localhost,39664,1320187706355
{noformat}
it's in
{noformat}
sun.management.ThreadImpl.getThreadInfo1(Native Method)
sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
{noformat}
So that's at least why adding a timeout wont help and may be why it does not end at all. Adding a maximum retry to Threads#threadDumpingIsAlive could help.
I also wonder if the root cause of the non ending is my modif on the wal, with some threads surprised to have updates that were not written in the wal. Here is the full stack dump:
{noformat}
Thread 354 (IPC Client (47) connection to localhost/127.0.0.1:52227 from nkeywal):
State: TIMED_WAITING
Blocked count: 360
Waited count: 359
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:702)
org.apache.hadoop.ipc.Client$Connection.run(Client.java:744)
Thread 272 (Master:0;localhost,39664,1320187706355-EventThread):
State: WAITING
Blocked count: 0
Waited count: 4
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@107b954b
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
Thread 271 (Master:0;localhost,39664,1320187706355-SendThread(localhost:21819)):
State: RUNNABLE
Blocked count: 2
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
Thread 152 (Master:0;localhost,39664,1320187706355):
State: WAITING
Blocked count: 217
Waited count: 174
Waiting on org.apache.hadoop.hbase.zookeeper.RootRegionTracker@6621477c
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:131)
org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:104)
org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRoot(CatalogTracker.java:277)
org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:523)
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:468)
org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)
java.lang.Thread.run(Thread.java:662)
Thread 165 (LruBlockCache.EvictionThread):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread@3e9d7b56
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:593)
java.lang.Thread.run(Thread.java:662)
Thread 151 (main-EventThread):
State: WAITING
Blocked count: 40
Waited count: 39
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22531380
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
Thread 150 (main-SendThread(localhost:21819)):
State: RUNNABLE
Blocked count: 55
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
Thread 149 (IPC Server handler 4 on 39664):
State: WAITING
Blocked count: 3
Waited count: 1968
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
Thread 148 (IPC Server handler 3 on 39664):
State: WAITING
Blocked count: 7
Waited count: 1969
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
Thread 147 (IPC Server handler 2 on 39664):
State: WAITING
Blocked count: 11
Waited count: 1968
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
Thread 146 (IPC Server handler 1 on 39664):
State: WAITING
Blocked count: 6
Waited count: 1969
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
Thread 145 (IPC Server handler 0 on 39664):
State: WAITING
Blocked count: 3
Waited count: 1970
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
Thread 133 (IPC Server listener on 39664):
State: RUNNABLE
Blocked count: 1
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:580)
Thread 144 (IPC Server Responder):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:754)
Thread 143 (IPC Reader 9 on port 39664):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 142 (IPC Reader 8 on port 39664):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 141 (IPC Reader 7 on port 39664):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 140 (IPC Reader 6 on port 39664):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 139 (IPC Reader 5 on port 39664):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 138 (IPC Reader 4 on port 39664):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 137 (IPC Reader 3 on port 39664):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 136 (IPC Reader 2 on port 39664):
State: RUNNABLE
Blocked count: 1
Waited count: 1
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 135 (IPC Reader 1 on port 39664):
State: RUNNABLE
Blocked count: 1
Waited count: 1
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 134 (IPC Reader 0 on port 39664):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 124 (LeaseChecker):
State: TIMED_WAITING
Blocked count: 0
Waited count: 407
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1252)
java.lang.Thread.run(Thread.java:662)
Thread 121 (ProcessThread:-1):
State: WAITING
Blocked count: 0
Waited count: 269
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f5d84e0
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:103)
Thread 120 (SyncThread:0):
State: WAITING
Blocked count: 2
Waited count: 252
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b25b27c
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:94)
Thread 119 (SessionTracker):
State: TIMED_WAITING
Blocked count: 0
Waited count: 207
Stack:
java.lang.Object.wait(Native Method)
org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:142)
Thread 118 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:21819):
State: RUNNABLE
Blocked count: 38
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:232)
Thread 116 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@2e6f947b):
State: TIMED_WAITING
Blocked count: 0
Waited count: 407
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
java.lang.Thread.run(Thread.java:662)
Thread 113 (IPC Server handler 2 on 39899):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 112 (IPC Server handler 1 on 39899):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 111 (IPC Server handler 0 on 39899):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 106 (IPC Server listener on 39899):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
Thread 108 (IPC Server Responder):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
Thread 102 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@45d1c3cd):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
java.lang.Thread.run(Thread.java:662)
Thread 109 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6]):
State: TIMED_WAITING
Blocked count: 137
Waited count: 283
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
java.lang.Thread.run(Thread.java:662)
Thread 107 (pool-4-thread-1):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 105 (Timer-3):
State: TIMED_WAITING
Blocked count: 0
Waited count: 15
Stack:
java.lang.Object.wait(Native Method)
java.util.TimerThread.mainLoop(Timer.java:509)
java.util.TimerThread.run(Timer.java:462)
Thread 104 (1353906974@qtp-1197660496-1 - Acceptor0 SelectChannelConnector@localhost:59858):
State: RUNNABLE
Blocked count: 1
Waited count: 1
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Thread 103 (1185716172@qtp-1197660496-0):
State: TIMED_WAITING
Blocked count: 0
Waited count: 8
Stack:
java.lang.Object.wait(Native Method)
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
Thread 101 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6):
State: TIMED_WAITING
Blocked count: 0
Waited count: 1
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
java.lang.Thread.run(Thread.java:662)
Thread 98 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5):
State: TIMED_WAITING
Blocked count: 0
Waited count: 1
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
java.lang.Thread.run(Thread.java:662)
Thread 90 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@605b28c9):
State: TIMED_WAITING
Blocked count: 0
Waited count: 408
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
java.lang.Thread.run(Thread.java:662)
Thread 86 (IPC Server handler 2 on 56098):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 85 (IPC Server handler 1 on 56098):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 84 (IPC Server handler 0 on 56098):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 80 (IPC Server listener on 56098):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
Thread 82 (IPC Server Responder):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
Thread 76 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@2b68989e):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
java.lang.Thread.run(Thread.java:662)
Thread 83 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4]):
State: TIMED_WAITING
Blocked count: 149
Waited count: 290
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
java.lang.Thread.run(Thread.java:662)
Thread 81 (pool-3-thread-1):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 79 (Timer-2):
State: TIMED_WAITING
Blocked count: 0
Waited count: 15
Stack:
java.lang.Object.wait(Native Method)
java.util.TimerThread.mainLoop(Timer.java:509)
java.util.TimerThread.run(Timer.java:462)
Thread 78 (348878159@qtp-164967086-1 - Acceptor0 SelectChannelConnector@localhost:36427):
State: RUNNABLE
Blocked count: 2
Waited count: 1
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Thread 77 (1905729203@qtp-164967086-0):
State: TIMED_WAITING
Blocked count: 0
Waited count: 8
Stack:
java.lang.Object.wait(Native Method)
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
Thread 75 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4):
State: TIMED_WAITING
Blocked count: 0
Waited count: 1
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
java.lang.Thread.run(Thread.java:662)
Thread 72 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3):
State: TIMED_WAITING
Blocked count: 0
Waited count: 1
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
java.lang.Thread.run(Thread.java:662)
Thread 61 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@6709da93):
State: TIMED_WAITING
Blocked count: 0
Waited count: 408
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
java.lang.Thread.run(Thread.java:662)
Thread 60 (IPC Server handler 2 on 40388):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 59 (IPC Server handler 1 on 40388):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 58 (IPC Server handler 0 on 40388):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 53 (IPC Server listener on 40388):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
Thread 55 (IPC Server Responder):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
Thread 49 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@48baa31b):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
java.lang.Thread.run(Thread.java:662)
Thread 57 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2]):
State: TIMED_WAITING
Blocked count: 148
Waited count: 283
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
java.lang.Thread.run(Thread.java:662)
Thread 54 (pool-2-thread-1):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 52 (Timer-1):
State: TIMED_WAITING
Blocked count: 0
Waited count: 15
Stack:
java.lang.Object.wait(Native Method)
java.util.TimerThread.mainLoop(Timer.java:509)
java.util.TimerThread.run(Timer.java:462)
Thread 51 (1031841975@qtp-1068922666-1 - Acceptor0 SelectChannelConnector@localhost:56535):
State: RUNNABLE
Blocked count: 1
Waited count: 1
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Thread 50 (1191055321@qtp-1068922666-0):
State: TIMED_WAITING
Blocked count: 0
Waited count: 8
Stack:
java.lang.Object.wait(Native Method)
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
Thread 48 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2):
State: TIMED_WAITING
Blocked count: 0
Waited count: 1
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
java.lang.Thread.run(Thread.java:662)
Thread 45 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1):
State: TIMED_WAITING
Blocked count: 0
Waited count: 1
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
java.lang.Thread.run(Thread.java:662)
Thread 31 (IPC Server handler 9 on 52227):
State: WAITING
Blocked count: 0
Waited count: 64
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 30 (IPC Server handler 8 on 52227):
State: WAITING
Blocked count: 3
Waited count: 65
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 29 (IPC Server handler 7 on 52227):
State: WAITING
Blocked count: 2
Waited count: 66
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 28 (IPC Server handler 6 on 52227):
State: WAITING
Blocked count: 1
Waited count: 64
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 27 (IPC Server handler 5 on 52227):
State: WAITING
Blocked count: 4
Waited count: 68
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 26 (IPC Server handler 4 on 52227):
State: WAITING
Blocked count: 0
Waited count: 64
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 25 (IPC Server handler 3 on 52227):
State: WAITING
Blocked count: 0
Waited count: 65
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 24 (IPC Server handler 2 on 52227):
State: WAITING
Blocked count: 1
Waited count: 66
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 23 (IPC Server handler 1 on 52227):
State: WAITING
Blocked count: 0
Waited count: 64
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 22 (IPC Server handler 0 on 52227):
State: WAITING
Blocked count: 3
Waited count: 65
Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 15 (IPC Server listener on 52227):
State: RUNNABLE
Blocked count: 43
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
Thread 17 (IPC Server Responder):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
Thread 21 (Timer-0):
State: TIMED_WAITING
Blocked count: 1
Waited count: 17
Stack:
java.lang.Object.wait(Native Method)
java.util.TimerThread.mainLoop(Timer.java:509)
java.util.TimerThread.run(Timer.java:462)
Thread 20 (245022965@qtp-589734556-1 - Acceptor0 SelectChannelConnector@localhost:33079):
State: RUNNABLE
Blocked count: 2
Waited count: 1
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Thread 19 (1821050251@qtp-589734556-0):
State: TIMED_WAITING
Blocked count: 0
Waited count: 8
Stack:
java.lang.Object.wait(Native Method)
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
Thread 16 (pool-1-thread-1):
State: RUNNABLE
Blocked count: 45
Waited count: 44
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 14 (org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@620fa83):
State: TIMED_WAITING
Blocked count: 1
Waited count: 147
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65)
java.lang.Thread.run(Thread.java:662)
Thread 13 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@d5d4de6):
State: TIMED_WAITING
Blocked count: 2
Waited count: 147
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2689)
java.lang.Thread.run(Thread.java:662)
Thread 12 (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@73d32e45):
State: TIMED_WAITING
Blocked count: 0
Waited count: 220
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:368)
java.lang.Thread.run(Thread.java:662)
Thread 11 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@19be4777):
State: TIMED_WAITING
Blocked count: 0
Waited count: 89
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2666)
java.lang.Thread.run(Thread.java:662)
Thread 10 (org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@658f7386):
State: TIMED_WAITING
Blocked count: 0
Waited count: 2
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
java.lang.Thread.run(Thread.java:662)
Thread 4 (Signal Dispatcher):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
Thread 3 (Finalizer):
State: WAITING
Blocked count: 9
Waited count: 10
Waiting on java.lang.ref.ReferenceQueue$Lock@3d1a70a7
Stack:
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
Thread 2 (Reference Handler):
State: WAITING
Blocked count: 11
Waited count: 12
Waiting on java.lang.ref.Reference$Lock@270d75a3
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
Thread 1 (main):
State: RUNNABLE
Blocked count: 104
Waited count: 97
Stack:
sun.management.ThreadImpl.getThreadInfo1(Native Method)
sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
org.junit.runners.ParentRunner.run(ParentRunner.java:236)
org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
{noformat}
Fix Version/s: 0.92.0
> TestAdmin hangs randomly in trunk
> ---------------------------------
>
> Key: HBASE-4724
> URL: https://issues.apache.org/jira/browse/HBASE-4724
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 20111102_4724_TestAdmin.patch, 20111102_4724_TestAdmin.v2.patch
>
>
> fom the logs in my env
> {noformat}
> 2011-11-01 15:48:40,744 WARN [Master:0;localhost,39664,1320187706355] master.AssignmentManager(1471): Failed assignment of -ROOT-,,0.70236052 to localhost,44046,1320187706849, trying to assign elsewhere instead; retry=1
> org.apache.hadoop.hbase.ipc.HBaseRPC$VersionMismatch: Protocol org.apache.hadoop.hbase.ipc.HRegionInterface version mismatch. (client = 28, server = 29)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:185)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:300)
> {noformat}
> Anyway, after this the logs finishes with:
> {noformat}
> 2011-11-01 15:54:35,132 INFO [Master:0;localhost,39664,1320187706355.oldLogCleaner] hbase.Chore(80): Master:0;localhost,39664,1320187706355.oldLogCleaner exiting
> Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on Master:0;localhost,39664,1320187706355
> {noformat}
> it's in
> {noformat}
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> {noformat}
> So that's at least why adding a timeout wont help and may be why it does not end at all. Adding a maximum retry to Threads#threadDumpingIsAlive could help.
> I also wonder if the root cause of the non ending is my modif on the wal, with some threads surprised to have updates that were not written in the wal. Here is the full stack dump:
> {noformat}
> Thread 354 (IPC Client (47) connection to localhost/127.0.0.1:52227 from nkeywal):
> State: TIMED_WAITING
> Blocked count: 360
> Waited count: 359
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:702)
> org.apache.hadoop.ipc.Client$Connection.run(Client.java:744)
> Thread 272 (Master:0;localhost,39664,1320187706355-EventThread):
> State: WAITING
> Blocked count: 0
> Waited count: 4
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@107b954b
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 271 (Master:0;localhost,39664,1320187706355-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 152 (Master:0;localhost,39664,1320187706355):
> State: WAITING
> Blocked count: 217
> Waited count: 174
> Waiting on org.apache.hadoop.hbase.zookeeper.RootRegionTracker@6621477c
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:131)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:104)
> org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRoot(CatalogTracker.java:277)
> org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:523)
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:468)
> org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)
> java.lang.Thread.run(Thread.java:662)
> Thread 165 (LruBlockCache.EvictionThread):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread@3e9d7b56
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:593)
> java.lang.Thread.run(Thread.java:662)
> Thread 151 (main-EventThread):
> State: WAITING
> Blocked count: 40
> Waited count: 39
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22531380
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 150 (main-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 55
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 149 (IPC Server handler 4 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 148 (IPC Server handler 3 on 39664):
> State: WAITING
> Blocked count: 7
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 147 (IPC Server handler 2 on 39664):
> State: WAITING
> Blocked count: 11
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 146 (IPC Server handler 1 on 39664):
> State: WAITING
> Blocked count: 6
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 145 (IPC Server handler 0 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1970
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 133 (IPC Server listener on 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:580)
> Thread 144 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:754)
> Thread 143 (IPC Reader 9 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 142 (IPC Reader 8 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 141 (IPC Reader 7 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 140 (IPC Reader 6 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 139 (IPC Reader 5 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 138 (IPC Reader 4 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 137 (IPC Reader 3 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 136 (IPC Reader 2 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 135 (IPC Reader 1 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 134 (IPC Reader 0 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 124 (LeaseChecker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1252)
> java.lang.Thread.run(Thread.java:662)
> Thread 121 (ProcessThread:-1):
> State: WAITING
> Blocked count: 0
> Waited count: 269
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f5d84e0
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:103)
> Thread 120 (SyncThread:0):
> State: WAITING
> Blocked count: 2
> Waited count: 252
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b25b27c
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:94)
> Thread 119 (SessionTracker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 207
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:142)
> Thread 118 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:21819):
> State: RUNNABLE
> Blocked count: 38
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:232)
> Thread 116 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@2e6f947b):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 113 (IPC Server handler 2 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 112 (IPC Server handler 1 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 111 (IPC Server handler 0 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 106 (IPC Server listener on 39899):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 108 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 102 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@45d1c3cd):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 109 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6]):
> State: TIMED_WAITING
> Blocked count: 137
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 107 (pool-4-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 105 (Timer-3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 104 (1353906974@qtp-1197660496-1 - Acceptor0 SelectChannelConnector@localhost:59858):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 103 (1185716172@qtp-1197660496-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 101 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 98 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 90 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@605b28c9):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 86 (IPC Server handler 2 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 85 (IPC Server handler 1 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 84 (IPC Server handler 0 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 80 (IPC Server listener on 56098):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 82 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 76 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@2b68989e):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 83 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4]):
> State: TIMED_WAITING
> Blocked count: 149
> Waited count: 290
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 81 (pool-3-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 79 (Timer-2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 78 (348878159@qtp-164967086-1 - Acceptor0 SelectChannelConnector@localhost:36427):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 77 (1905729203@qtp-164967086-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 75 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 72 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 61 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@6709da93):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 60 (IPC Server handler 2 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 59 (IPC Server handler 1 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 58 (IPC Server handler 0 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 53 (IPC Server listener on 40388):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 55 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 49 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@48baa31b):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 57 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2]):
> State: TIMED_WAITING
> Blocked count: 148
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 54 (pool-2-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 52 (Timer-1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 51 (1031841975@qtp-1068922666-1 - Acceptor0 SelectChannelConnector@localhost:56535):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 50 (1191055321@qtp-1068922666-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 48 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 45 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 31 (IPC Server handler 9 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 30 (IPC Server handler 8 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 29 (IPC Server handler 7 on 52227):
> State: WAITING
> Blocked count: 2
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 28 (IPC Server handler 6 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 27 (IPC Server handler 5 on 52227):
> State: WAITING
> Blocked count: 4
> Waited count: 68
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 26 (IPC Server handler 4 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 25 (IPC Server handler 3 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 24 (IPC Server handler 2 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 23 (IPC Server handler 1 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 22 (IPC Server handler 0 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 15 (IPC Server listener on 52227):
> State: RUNNABLE
> Blocked count: 43
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 17 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 21 (Timer-0):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 17
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 20 (245022965@qtp-589734556-1 - Acceptor0 SelectChannelConnector@localhost:33079):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 19 (1821050251@qtp-589734556-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 16 (pool-1-thread-1):
> State: RUNNABLE
> Blocked count: 45
> Waited count: 44
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 14 (org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@620fa83):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65)
> java.lang.Thread.run(Thread.java:662)
> Thread 13 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@d5d4de6):
> State: TIMED_WAITING
> Blocked count: 2
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2689)
> java.lang.Thread.run(Thread.java:662)
> Thread 12 (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@73d32e45):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 220
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:368)
> java.lang.Thread.run(Thread.java:662)
> Thread 11 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@19be4777):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 89
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2666)
> java.lang.Thread.run(Thread.java:662)
> Thread 10 (org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@658f7386):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 2
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
> java.lang.Thread.run(Thread.java:662)
> Thread 4 (Signal Dispatcher):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> Thread 3 (Finalizer):
> State: WAITING
> Blocked count: 9
> Waited count: 10
> Waiting on java.lang.ref.ReferenceQueue$Lock@3d1a70a7
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> Thread 2 (Reference Handler):
> State: WAITING
> Blocked count: 11
> Waited count: 12
> Waiting on java.lang.ref.Reference$Lock@270d75a3
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> Thread 1 (main):
> State: RUNNABLE
> Blocked count: 104
> Waited count: 97
> Stack:
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> java.lang.reflect.Method.invoke(Method.java:597)
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
> org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4724) TestAdmin hangs randomly in trunk
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13148583#comment-13148583 ]
Hadoop QA commented on HBASE-4724:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12501935/20111102_4724_TestAdmin.v2.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/233//console
This message is automatically generated.
> TestAdmin hangs randomly in trunk
> ---------------------------------
>
> Key: HBASE-4724
> URL: https://issues.apache.org/jira/browse/HBASE-4724
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Fix For: 0.92.0
>
> Attachments: 20111102_4724_TestAdmin.patch, 20111102_4724_TestAdmin.v2.patch
>
>
> fom the logs in my env
> {noformat}
> 2011-11-01 15:48:40,744 WARN [Master:0;localhost,39664,1320187706355] master.AssignmentManager(1471): Failed assignment of -ROOT-,,0.70236052 to localhost,44046,1320187706849, trying to assign elsewhere instead; retry=1
> org.apache.hadoop.hbase.ipc.HBaseRPC$VersionMismatch: Protocol org.apache.hadoop.hbase.ipc.HRegionInterface version mismatch. (client = 28, server = 29)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:185)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:300)
> {noformat}
> Anyway, after this the logs finishes with:
> {noformat}
> 2011-11-01 15:54:35,132 INFO [Master:0;localhost,39664,1320187706355.oldLogCleaner] hbase.Chore(80): Master:0;localhost,39664,1320187706355.oldLogCleaner exiting
> Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on Master:0;localhost,39664,1320187706355
> {noformat}
> it's in
> {noformat}
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> {noformat}
> So that's at least why adding a timeout wont help and may be why it does not end at all. Adding a maximum retry to Threads#threadDumpingIsAlive could help.
> I also wonder if the root cause of the non ending is my modif on the wal, with some threads surprised to have updates that were not written in the wal. Here is the full stack dump:
> {noformat}
> Thread 354 (IPC Client (47) connection to localhost/127.0.0.1:52227 from nkeywal):
> State: TIMED_WAITING
> Blocked count: 360
> Waited count: 359
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:702)
> org.apache.hadoop.ipc.Client$Connection.run(Client.java:744)
> Thread 272 (Master:0;localhost,39664,1320187706355-EventThread):
> State: WAITING
> Blocked count: 0
> Waited count: 4
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@107b954b
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 271 (Master:0;localhost,39664,1320187706355-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 152 (Master:0;localhost,39664,1320187706355):
> State: WAITING
> Blocked count: 217
> Waited count: 174
> Waiting on org.apache.hadoop.hbase.zookeeper.RootRegionTracker@6621477c
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:131)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:104)
> org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRoot(CatalogTracker.java:277)
> org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:523)
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:468)
> org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)
> java.lang.Thread.run(Thread.java:662)
> Thread 165 (LruBlockCache.EvictionThread):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread@3e9d7b56
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:593)
> java.lang.Thread.run(Thread.java:662)
> Thread 151 (main-EventThread):
> State: WAITING
> Blocked count: 40
> Waited count: 39
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22531380
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 150 (main-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 55
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 149 (IPC Server handler 4 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 148 (IPC Server handler 3 on 39664):
> State: WAITING
> Blocked count: 7
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 147 (IPC Server handler 2 on 39664):
> State: WAITING
> Blocked count: 11
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 146 (IPC Server handler 1 on 39664):
> State: WAITING
> Blocked count: 6
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 145 (IPC Server handler 0 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1970
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 133 (IPC Server listener on 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:580)
> Thread 144 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:754)
> Thread 143 (IPC Reader 9 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 142 (IPC Reader 8 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 141 (IPC Reader 7 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 140 (IPC Reader 6 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 139 (IPC Reader 5 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 138 (IPC Reader 4 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 137 (IPC Reader 3 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 136 (IPC Reader 2 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 135 (IPC Reader 1 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 134 (IPC Reader 0 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 124 (LeaseChecker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1252)
> java.lang.Thread.run(Thread.java:662)
> Thread 121 (ProcessThread:-1):
> State: WAITING
> Blocked count: 0
> Waited count: 269
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f5d84e0
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:103)
> Thread 120 (SyncThread:0):
> State: WAITING
> Blocked count: 2
> Waited count: 252
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b25b27c
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:94)
> Thread 119 (SessionTracker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 207
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:142)
> Thread 118 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:21819):
> State: RUNNABLE
> Blocked count: 38
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:232)
> Thread 116 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@2e6f947b):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 113 (IPC Server handler 2 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 112 (IPC Server handler 1 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 111 (IPC Server handler 0 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 106 (IPC Server listener on 39899):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 108 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 102 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@45d1c3cd):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 109 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6]):
> State: TIMED_WAITING
> Blocked count: 137
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 107 (pool-4-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 105 (Timer-3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 104 (1353906974@qtp-1197660496-1 - Acceptor0 SelectChannelConnector@localhost:59858):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 103 (1185716172@qtp-1197660496-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 101 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 98 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 90 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@605b28c9):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 86 (IPC Server handler 2 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 85 (IPC Server handler 1 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 84 (IPC Server handler 0 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 80 (IPC Server listener on 56098):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 82 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 76 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@2b68989e):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 83 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4]):
> State: TIMED_WAITING
> Blocked count: 149
> Waited count: 290
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 81 (pool-3-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 79 (Timer-2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 78 (348878159@qtp-164967086-1 - Acceptor0 SelectChannelConnector@localhost:36427):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 77 (1905729203@qtp-164967086-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 75 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 72 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 61 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@6709da93):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 60 (IPC Server handler 2 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 59 (IPC Server handler 1 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 58 (IPC Server handler 0 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 53 (IPC Server listener on 40388):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 55 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 49 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@48baa31b):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 57 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2]):
> State: TIMED_WAITING
> Blocked count: 148
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 54 (pool-2-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 52 (Timer-1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 51 (1031841975@qtp-1068922666-1 - Acceptor0 SelectChannelConnector@localhost:56535):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 50 (1191055321@qtp-1068922666-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 48 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 45 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 31 (IPC Server handler 9 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 30 (IPC Server handler 8 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 29 (IPC Server handler 7 on 52227):
> State: WAITING
> Blocked count: 2
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 28 (IPC Server handler 6 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 27 (IPC Server handler 5 on 52227):
> State: WAITING
> Blocked count: 4
> Waited count: 68
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 26 (IPC Server handler 4 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 25 (IPC Server handler 3 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 24 (IPC Server handler 2 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 23 (IPC Server handler 1 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 22 (IPC Server handler 0 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 15 (IPC Server listener on 52227):
> State: RUNNABLE
> Blocked count: 43
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 17 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 21 (Timer-0):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 17
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 20 (245022965@qtp-589734556-1 - Acceptor0 SelectChannelConnector@localhost:33079):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 19 (1821050251@qtp-589734556-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 16 (pool-1-thread-1):
> State: RUNNABLE
> Blocked count: 45
> Waited count: 44
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 14 (org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@620fa83):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65)
> java.lang.Thread.run(Thread.java:662)
> Thread 13 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@d5d4de6):
> State: TIMED_WAITING
> Blocked count: 2
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2689)
> java.lang.Thread.run(Thread.java:662)
> Thread 12 (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@73d32e45):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 220
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:368)
> java.lang.Thread.run(Thread.java:662)
> Thread 11 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@19be4777):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 89
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2666)
> java.lang.Thread.run(Thread.java:662)
> Thread 10 (org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@658f7386):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 2
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
> java.lang.Thread.run(Thread.java:662)
> Thread 4 (Signal Dispatcher):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> Thread 3 (Finalizer):
> State: WAITING
> Blocked count: 9
> Waited count: 10
> Waiting on java.lang.ref.ReferenceQueue$Lock@3d1a70a7
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> Thread 2 (Reference Handler):
> State: WAITING
> Blocked count: 11
> Waited count: 12
> Waiting on java.lang.ref.Reference$Lock@270d75a3
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> Thread 1 (main):
> State: RUNNABLE
> Blocked count: 104
> Waited count: 97
> Stack:
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> java.lang.reflect.Method.invoke(Method.java:597)
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
> org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4724) TestAdmin hangs randomly in trunk
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142272#comment-13142272 ]
nkeywal commented on HBASE-4724:
--------------------------------
The timer I changed on the admin are all with a condition; so instead of checking once per second they check 5 times per second. This should not change the final behaviour, and I took care of not changing the final timemout. testEnableDisableAddColumnDeleteColumn has not been directly impacted by 4703, there is no timer there.
What's the line that's throwing this exception?
> TestAdmin hangs randomly in trunk
> ---------------------------------
>
> Key: HBASE-4724
> URL: https://issues.apache.org/jira/browse/HBASE-4724
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 20111102_4724_TestAdmin.patch, 20111102_4724_TestAdmin.v2.patch
>
>
> fom the logs in my env
> {noformat}
> 2011-11-01 15:48:40,744 WARN [Master:0;localhost,39664,1320187706355] master.AssignmentManager(1471): Failed assignment of -ROOT-,,0.70236052 to localhost,44046,1320187706849, trying to assign elsewhere instead; retry=1
> org.apache.hadoop.hbase.ipc.HBaseRPC$VersionMismatch: Protocol org.apache.hadoop.hbase.ipc.HRegionInterface version mismatch. (client = 28, server = 29)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:185)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:300)
> {noformat}
> Anyway, after this the logs finishes with:
> {noformat}
> 2011-11-01 15:54:35,132 INFO [Master:0;localhost,39664,1320187706355.oldLogCleaner] hbase.Chore(80): Master:0;localhost,39664,1320187706355.oldLogCleaner exiting
> Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on Master:0;localhost,39664,1320187706355
> {noformat}
> it's in
> {noformat}
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> {noformat}
> So that's at least why adding a timeout wont help and may be why it does not end at all. Adding a maximum retry to Threads#threadDumpingIsAlive could help.
> I also wonder if the root cause of the non ending is my modif on the wal, with some threads surprised to have updates that were not written in the wal. Here is the full stack dump:
> {noformat}
> Thread 354 (IPC Client (47) connection to localhost/127.0.0.1:52227 from nkeywal):
> State: TIMED_WAITING
> Blocked count: 360
> Waited count: 359
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:702)
> org.apache.hadoop.ipc.Client$Connection.run(Client.java:744)
> Thread 272 (Master:0;localhost,39664,1320187706355-EventThread):
> State: WAITING
> Blocked count: 0
> Waited count: 4
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@107b954b
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 271 (Master:0;localhost,39664,1320187706355-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 152 (Master:0;localhost,39664,1320187706355):
> State: WAITING
> Blocked count: 217
> Waited count: 174
> Waiting on org.apache.hadoop.hbase.zookeeper.RootRegionTracker@6621477c
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:131)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:104)
> org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRoot(CatalogTracker.java:277)
> org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:523)
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:468)
> org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)
> java.lang.Thread.run(Thread.java:662)
> Thread 165 (LruBlockCache.EvictionThread):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread@3e9d7b56
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:593)
> java.lang.Thread.run(Thread.java:662)
> Thread 151 (main-EventThread):
> State: WAITING
> Blocked count: 40
> Waited count: 39
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22531380
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 150 (main-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 55
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 149 (IPC Server handler 4 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 148 (IPC Server handler 3 on 39664):
> State: WAITING
> Blocked count: 7
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 147 (IPC Server handler 2 on 39664):
> State: WAITING
> Blocked count: 11
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 146 (IPC Server handler 1 on 39664):
> State: WAITING
> Blocked count: 6
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 145 (IPC Server handler 0 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1970
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 133 (IPC Server listener on 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:580)
> Thread 144 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:754)
> Thread 143 (IPC Reader 9 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 142 (IPC Reader 8 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 141 (IPC Reader 7 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 140 (IPC Reader 6 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 139 (IPC Reader 5 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 138 (IPC Reader 4 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 137 (IPC Reader 3 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 136 (IPC Reader 2 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 135 (IPC Reader 1 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 134 (IPC Reader 0 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 124 (LeaseChecker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1252)
> java.lang.Thread.run(Thread.java:662)
> Thread 121 (ProcessThread:-1):
> State: WAITING
> Blocked count: 0
> Waited count: 269
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f5d84e0
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:103)
> Thread 120 (SyncThread:0):
> State: WAITING
> Blocked count: 2
> Waited count: 252
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b25b27c
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:94)
> Thread 119 (SessionTracker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 207
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:142)
> Thread 118 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:21819):
> State: RUNNABLE
> Blocked count: 38
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:232)
> Thread 116 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@2e6f947b):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 113 (IPC Server handler 2 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 112 (IPC Server handler 1 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 111 (IPC Server handler 0 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 106 (IPC Server listener on 39899):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 108 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 102 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@45d1c3cd):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 109 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6]):
> State: TIMED_WAITING
> Blocked count: 137
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 107 (pool-4-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 105 (Timer-3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 104 (1353906974@qtp-1197660496-1 - Acceptor0 SelectChannelConnector@localhost:59858):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 103 (1185716172@qtp-1197660496-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 101 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 98 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 90 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@605b28c9):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 86 (IPC Server handler 2 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 85 (IPC Server handler 1 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 84 (IPC Server handler 0 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 80 (IPC Server listener on 56098):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 82 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 76 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@2b68989e):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 83 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4]):
> State: TIMED_WAITING
> Blocked count: 149
> Waited count: 290
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 81 (pool-3-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 79 (Timer-2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 78 (348878159@qtp-164967086-1 - Acceptor0 SelectChannelConnector@localhost:36427):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 77 (1905729203@qtp-164967086-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 75 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 72 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 61 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@6709da93):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 60 (IPC Server handler 2 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 59 (IPC Server handler 1 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 58 (IPC Server handler 0 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 53 (IPC Server listener on 40388):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 55 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 49 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@48baa31b):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 57 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2]):
> State: TIMED_WAITING
> Blocked count: 148
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 54 (pool-2-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 52 (Timer-1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 51 (1031841975@qtp-1068922666-1 - Acceptor0 SelectChannelConnector@localhost:56535):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 50 (1191055321@qtp-1068922666-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 48 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 45 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 31 (IPC Server handler 9 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 30 (IPC Server handler 8 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 29 (IPC Server handler 7 on 52227):
> State: WAITING
> Blocked count: 2
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 28 (IPC Server handler 6 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 27 (IPC Server handler 5 on 52227):
> State: WAITING
> Blocked count: 4
> Waited count: 68
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 26 (IPC Server handler 4 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 25 (IPC Server handler 3 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 24 (IPC Server handler 2 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 23 (IPC Server handler 1 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 22 (IPC Server handler 0 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 15 (IPC Server listener on 52227):
> State: RUNNABLE
> Blocked count: 43
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 17 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 21 (Timer-0):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 17
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 20 (245022965@qtp-589734556-1 - Acceptor0 SelectChannelConnector@localhost:33079):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 19 (1821050251@qtp-589734556-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 16 (pool-1-thread-1):
> State: RUNNABLE
> Blocked count: 45
> Waited count: 44
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 14 (org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@620fa83):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65)
> java.lang.Thread.run(Thread.java:662)
> Thread 13 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@d5d4de6):
> State: TIMED_WAITING
> Blocked count: 2
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2689)
> java.lang.Thread.run(Thread.java:662)
> Thread 12 (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@73d32e45):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 220
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:368)
> java.lang.Thread.run(Thread.java:662)
> Thread 11 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@19be4777):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 89
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2666)
> java.lang.Thread.run(Thread.java:662)
> Thread 10 (org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@658f7386):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 2
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
> java.lang.Thread.run(Thread.java:662)
> Thread 4 (Signal Dispatcher):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> Thread 3 (Finalizer):
> State: WAITING
> Blocked count: 9
> Waited count: 10
> Waiting on java.lang.ref.ReferenceQueue$Lock@3d1a70a7
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> Thread 2 (Reference Handler):
> State: WAITING
> Blocked count: 11
> Waited count: 12
> Waiting on java.lang.ref.Reference$Lock@270d75a3
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> Thread 1 (main):
> State: RUNNABLE
> Blocked count: 104
> Waited count: 97
> Stack:
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> java.lang.reflect.Method.invoke(Method.java:597)
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
> org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4724) TestAdmin hangs randomly in trunk
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142308#comment-13142308 ]
nkeywal commented on HBASE-4724:
--------------------------------
It happened once out of may be 20 tries.
For the null connection, there is at least a leak in
{noformat}
@Before
public void setUp() throws Exception {
this.admin = new HBaseAdmin(TEST_UTIL.getConfiguration());
}
{noformat}
But I guess it's not the only one, because I saw quite a lot of lines in the logs from Jenkins.
> TestAdmin hangs randomly in trunk
> ---------------------------------
>
> Key: HBASE-4724
> URL: https://issues.apache.org/jira/browse/HBASE-4724
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 20111102_4724_TestAdmin.patch, 20111102_4724_TestAdmin.v2.patch
>
>
> fom the logs in my env
> {noformat}
> 2011-11-01 15:48:40,744 WARN [Master:0;localhost,39664,1320187706355] master.AssignmentManager(1471): Failed assignment of -ROOT-,,0.70236052 to localhost,44046,1320187706849, trying to assign elsewhere instead; retry=1
> org.apache.hadoop.hbase.ipc.HBaseRPC$VersionMismatch: Protocol org.apache.hadoop.hbase.ipc.HRegionInterface version mismatch. (client = 28, server = 29)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:185)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:300)
> {noformat}
> Anyway, after this the logs finishes with:
> {noformat}
> 2011-11-01 15:54:35,132 INFO [Master:0;localhost,39664,1320187706355.oldLogCleaner] hbase.Chore(80): Master:0;localhost,39664,1320187706355.oldLogCleaner exiting
> Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on Master:0;localhost,39664,1320187706355
> {noformat}
> it's in
> {noformat}
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> {noformat}
> So that's at least why adding a timeout wont help and may be why it does not end at all. Adding a maximum retry to Threads#threadDumpingIsAlive could help.
> I also wonder if the root cause of the non ending is my modif on the wal, with some threads surprised to have updates that were not written in the wal. Here is the full stack dump:
> {noformat}
> Thread 354 (IPC Client (47) connection to localhost/127.0.0.1:52227 from nkeywal):
> State: TIMED_WAITING
> Blocked count: 360
> Waited count: 359
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:702)
> org.apache.hadoop.ipc.Client$Connection.run(Client.java:744)
> Thread 272 (Master:0;localhost,39664,1320187706355-EventThread):
> State: WAITING
> Blocked count: 0
> Waited count: 4
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@107b954b
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 271 (Master:0;localhost,39664,1320187706355-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 152 (Master:0;localhost,39664,1320187706355):
> State: WAITING
> Blocked count: 217
> Waited count: 174
> Waiting on org.apache.hadoop.hbase.zookeeper.RootRegionTracker@6621477c
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:131)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:104)
> org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRoot(CatalogTracker.java:277)
> org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:523)
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:468)
> org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)
> java.lang.Thread.run(Thread.java:662)
> Thread 165 (LruBlockCache.EvictionThread):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread@3e9d7b56
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:593)
> java.lang.Thread.run(Thread.java:662)
> Thread 151 (main-EventThread):
> State: WAITING
> Blocked count: 40
> Waited count: 39
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22531380
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 150 (main-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 55
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 149 (IPC Server handler 4 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 148 (IPC Server handler 3 on 39664):
> State: WAITING
> Blocked count: 7
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 147 (IPC Server handler 2 on 39664):
> State: WAITING
> Blocked count: 11
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 146 (IPC Server handler 1 on 39664):
> State: WAITING
> Blocked count: 6
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 145 (IPC Server handler 0 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1970
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 133 (IPC Server listener on 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:580)
> Thread 144 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:754)
> Thread 143 (IPC Reader 9 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 142 (IPC Reader 8 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 141 (IPC Reader 7 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 140 (IPC Reader 6 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 139 (IPC Reader 5 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 138 (IPC Reader 4 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 137 (IPC Reader 3 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 136 (IPC Reader 2 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 135 (IPC Reader 1 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 134 (IPC Reader 0 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 124 (LeaseChecker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1252)
> java.lang.Thread.run(Thread.java:662)
> Thread 121 (ProcessThread:-1):
> State: WAITING
> Blocked count: 0
> Waited count: 269
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f5d84e0
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:103)
> Thread 120 (SyncThread:0):
> State: WAITING
> Blocked count: 2
> Waited count: 252
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b25b27c
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:94)
> Thread 119 (SessionTracker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 207
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:142)
> Thread 118 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:21819):
> State: RUNNABLE
> Blocked count: 38
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:232)
> Thread 116 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@2e6f947b):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 113 (IPC Server handler 2 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 112 (IPC Server handler 1 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 111 (IPC Server handler 0 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 106 (IPC Server listener on 39899):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 108 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 102 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@45d1c3cd):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 109 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6]):
> State: TIMED_WAITING
> Blocked count: 137
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 107 (pool-4-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 105 (Timer-3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 104 (1353906974@qtp-1197660496-1 - Acceptor0 SelectChannelConnector@localhost:59858):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 103 (1185716172@qtp-1197660496-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 101 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 98 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 90 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@605b28c9):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 86 (IPC Server handler 2 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 85 (IPC Server handler 1 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 84 (IPC Server handler 0 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 80 (IPC Server listener on 56098):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 82 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 76 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@2b68989e):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 83 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4]):
> State: TIMED_WAITING
> Blocked count: 149
> Waited count: 290
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 81 (pool-3-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 79 (Timer-2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 78 (348878159@qtp-164967086-1 - Acceptor0 SelectChannelConnector@localhost:36427):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 77 (1905729203@qtp-164967086-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 75 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 72 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 61 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@6709da93):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 60 (IPC Server handler 2 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 59 (IPC Server handler 1 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 58 (IPC Server handler 0 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 53 (IPC Server listener on 40388):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 55 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 49 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@48baa31b):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 57 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2]):
> State: TIMED_WAITING
> Blocked count: 148
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 54 (pool-2-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 52 (Timer-1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 51 (1031841975@qtp-1068922666-1 - Acceptor0 SelectChannelConnector@localhost:56535):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 50 (1191055321@qtp-1068922666-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 48 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 45 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 31 (IPC Server handler 9 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 30 (IPC Server handler 8 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 29 (IPC Server handler 7 on 52227):
> State: WAITING
> Blocked count: 2
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 28 (IPC Server handler 6 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 27 (IPC Server handler 5 on 52227):
> State: WAITING
> Blocked count: 4
> Waited count: 68
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 26 (IPC Server handler 4 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 25 (IPC Server handler 3 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 24 (IPC Server handler 2 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 23 (IPC Server handler 1 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 22 (IPC Server handler 0 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 15 (IPC Server listener on 52227):
> State: RUNNABLE
> Blocked count: 43
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 17 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 21 (Timer-0):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 17
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 20 (245022965@qtp-589734556-1 - Acceptor0 SelectChannelConnector@localhost:33079):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 19 (1821050251@qtp-589734556-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 16 (pool-1-thread-1):
> State: RUNNABLE
> Blocked count: 45
> Waited count: 44
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 14 (org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@620fa83):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65)
> java.lang.Thread.run(Thread.java:662)
> Thread 13 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@d5d4de6):
> State: TIMED_WAITING
> Blocked count: 2
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2689)
> java.lang.Thread.run(Thread.java:662)
> Thread 12 (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@73d32e45):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 220
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:368)
> java.lang.Thread.run(Thread.java:662)
> Thread 11 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@19be4777):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 89
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2666)
> java.lang.Thread.run(Thread.java:662)
> Thread 10 (org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@658f7386):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 2
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
> java.lang.Thread.run(Thread.java:662)
> Thread 4 (Signal Dispatcher):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> Thread 3 (Finalizer):
> State: WAITING
> Blocked count: 9
> Waited count: 10
> Waiting on java.lang.ref.ReferenceQueue$Lock@3d1a70a7
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> Thread 2 (Reference Handler):
> State: WAITING
> Blocked count: 11
> Waited count: 12
> Waiting on java.lang.ref.Reference$Lock@270d75a3
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> Thread 1 (main):
> State: RUNNABLE
> Blocked count: 104
> Waited count: 97
> Stack:
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> java.lang.reflect.Method.invoke(Method.java:597)
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
> org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4724) TestAdmin hangs randomly in trunk
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4724:
---------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
> TestAdmin hangs randomly in trunk
> ---------------------------------
>
> Key: HBASE-4724
> URL: https://issues.apache.org/jira/browse/HBASE-4724
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Attachments: 20111102_4724_TestAdmin.patch, 20111102_4724_TestAdmin.v2.patch
>
>
> fom the logs in my env
> {noformat}
> 2011-11-01 15:48:40,744 WARN [Master:0;localhost,39664,1320187706355] master.AssignmentManager(1471): Failed assignment of -ROOT-,,0.70236052 to localhost,44046,1320187706849, trying to assign elsewhere instead; retry=1
> org.apache.hadoop.hbase.ipc.HBaseRPC$VersionMismatch: Protocol org.apache.hadoop.hbase.ipc.HRegionInterface version mismatch. (client = 28, server = 29)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:185)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:300)
> {noformat}
> Anyway, after this the logs finishes with:
> {noformat}
> 2011-11-01 15:54:35,132 INFO [Master:0;localhost,39664,1320187706355.oldLogCleaner] hbase.Chore(80): Master:0;localhost,39664,1320187706355.oldLogCleaner exiting
> Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on Master:0;localhost,39664,1320187706355
> {noformat}
> it's in
> {noformat}
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> {noformat}
> So that's at least why adding a timeout wont help and may be why it does not end at all. Adding a maximum retry to Threads#threadDumpingIsAlive could help.
> I also wonder if the root cause of the non ending is my modif on the wal, with some threads surprised to have updates that were not written in the wal. Here is the full stack dump:
> {noformat}
> Thread 354 (IPC Client (47) connection to localhost/127.0.0.1:52227 from nkeywal):
> State: TIMED_WAITING
> Blocked count: 360
> Waited count: 359
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:702)
> org.apache.hadoop.ipc.Client$Connection.run(Client.java:744)
> Thread 272 (Master:0;localhost,39664,1320187706355-EventThread):
> State: WAITING
> Blocked count: 0
> Waited count: 4
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@107b954b
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 271 (Master:0;localhost,39664,1320187706355-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 152 (Master:0;localhost,39664,1320187706355):
> State: WAITING
> Blocked count: 217
> Waited count: 174
> Waiting on org.apache.hadoop.hbase.zookeeper.RootRegionTracker@6621477c
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:131)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:104)
> org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRoot(CatalogTracker.java:277)
> org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:523)
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:468)
> org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)
> java.lang.Thread.run(Thread.java:662)
> Thread 165 (LruBlockCache.EvictionThread):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread@3e9d7b56
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:593)
> java.lang.Thread.run(Thread.java:662)
> Thread 151 (main-EventThread):
> State: WAITING
> Blocked count: 40
> Waited count: 39
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22531380
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 150 (main-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 55
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 149 (IPC Server handler 4 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 148 (IPC Server handler 3 on 39664):
> State: WAITING
> Blocked count: 7
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 147 (IPC Server handler 2 on 39664):
> State: WAITING
> Blocked count: 11
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 146 (IPC Server handler 1 on 39664):
> State: WAITING
> Blocked count: 6
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 145 (IPC Server handler 0 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1970
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 133 (IPC Server listener on 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:580)
> Thread 144 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:754)
> Thread 143 (IPC Reader 9 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 142 (IPC Reader 8 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 141 (IPC Reader 7 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 140 (IPC Reader 6 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 139 (IPC Reader 5 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 138 (IPC Reader 4 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 137 (IPC Reader 3 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 136 (IPC Reader 2 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 135 (IPC Reader 1 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 134 (IPC Reader 0 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 124 (LeaseChecker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1252)
> java.lang.Thread.run(Thread.java:662)
> Thread 121 (ProcessThread:-1):
> State: WAITING
> Blocked count: 0
> Waited count: 269
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f5d84e0
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:103)
> Thread 120 (SyncThread:0):
> State: WAITING
> Blocked count: 2
> Waited count: 252
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b25b27c
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:94)
> Thread 119 (SessionTracker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 207
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:142)
> Thread 118 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:21819):
> State: RUNNABLE
> Blocked count: 38
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:232)
> Thread 116 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@2e6f947b):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 113 (IPC Server handler 2 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 112 (IPC Server handler 1 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 111 (IPC Server handler 0 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 106 (IPC Server listener on 39899):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 108 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 102 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@45d1c3cd):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 109 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6]):
> State: TIMED_WAITING
> Blocked count: 137
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 107 (pool-4-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 105 (Timer-3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 104 (1353906974@qtp-1197660496-1 - Acceptor0 SelectChannelConnector@localhost:59858):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 103 (1185716172@qtp-1197660496-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 101 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 98 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 90 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@605b28c9):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 86 (IPC Server handler 2 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 85 (IPC Server handler 1 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 84 (IPC Server handler 0 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 80 (IPC Server listener on 56098):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 82 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 76 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@2b68989e):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 83 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4]):
> State: TIMED_WAITING
> Blocked count: 149
> Waited count: 290
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 81 (pool-3-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 79 (Timer-2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 78 (348878159@qtp-164967086-1 - Acceptor0 SelectChannelConnector@localhost:36427):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 77 (1905729203@qtp-164967086-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 75 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 72 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 61 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@6709da93):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 60 (IPC Server handler 2 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 59 (IPC Server handler 1 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 58 (IPC Server handler 0 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 53 (IPC Server listener on 40388):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 55 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 49 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@48baa31b):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 57 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2]):
> State: TIMED_WAITING
> Blocked count: 148
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 54 (pool-2-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 52 (Timer-1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 51 (1031841975@qtp-1068922666-1 - Acceptor0 SelectChannelConnector@localhost:56535):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 50 (1191055321@qtp-1068922666-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 48 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 45 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 31 (IPC Server handler 9 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 30 (IPC Server handler 8 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 29 (IPC Server handler 7 on 52227):
> State: WAITING
> Blocked count: 2
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 28 (IPC Server handler 6 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 27 (IPC Server handler 5 on 52227):
> State: WAITING
> Blocked count: 4
> Waited count: 68
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 26 (IPC Server handler 4 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 25 (IPC Server handler 3 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 24 (IPC Server handler 2 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 23 (IPC Server handler 1 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 22 (IPC Server handler 0 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 15 (IPC Server listener on 52227):
> State: RUNNABLE
> Blocked count: 43
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 17 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 21 (Timer-0):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 17
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 20 (245022965@qtp-589734556-1 - Acceptor0 SelectChannelConnector@localhost:33079):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 19 (1821050251@qtp-589734556-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 16 (pool-1-thread-1):
> State: RUNNABLE
> Blocked count: 45
> Waited count: 44
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 14 (org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@620fa83):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65)
> java.lang.Thread.run(Thread.java:662)
> Thread 13 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@d5d4de6):
> State: TIMED_WAITING
> Blocked count: 2
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2689)
> java.lang.Thread.run(Thread.java:662)
> Thread 12 (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@73d32e45):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 220
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:368)
> java.lang.Thread.run(Thread.java:662)
> Thread 11 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@19be4777):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 89
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2666)
> java.lang.Thread.run(Thread.java:662)
> Thread 10 (org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@658f7386):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 2
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
> java.lang.Thread.run(Thread.java:662)
> Thread 4 (Signal Dispatcher):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> Thread 3 (Finalizer):
> State: WAITING
> Blocked count: 9
> Waited count: 10
> Waiting on java.lang.ref.ReferenceQueue$Lock@3d1a70a7
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> Thread 2 (Reference Handler):
> State: WAITING
> Blocked count: 11
> Waited count: 12
> Waiting on java.lang.ref.Reference$Lock@270d75a3
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> Thread 1 (main):
> State: RUNNABLE
> Blocked count: 104
> Waited count: 97
> Stack:
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> java.lang.reflect.Method.invoke(Method.java:597)
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
> org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4724) TestAdmin hangs randomly in trunk
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4724:
-------------------------
Fix Version/s: (was: 0.92.0)
This hasn't happened in a while. Moving out of 0.92.
> TestAdmin hangs randomly in trunk
> ---------------------------------
>
> Key: HBASE-4724
> URL: https://issues.apache.org/jira/browse/HBASE-4724
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Attachments: 20111102_4724_TestAdmin.patch, 20111102_4724_TestAdmin.v2.patch
>
>
> fom the logs in my env
> {noformat}
> 2011-11-01 15:48:40,744 WARN [Master:0;localhost,39664,1320187706355] master.AssignmentManager(1471): Failed assignment of -ROOT-,,0.70236052 to localhost,44046,1320187706849, trying to assign elsewhere instead; retry=1
> org.apache.hadoop.hbase.ipc.HBaseRPC$VersionMismatch: Protocol org.apache.hadoop.hbase.ipc.HRegionInterface version mismatch. (client = 28, server = 29)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:185)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:300)
> {noformat}
> Anyway, after this the logs finishes with:
> {noformat}
> 2011-11-01 15:54:35,132 INFO [Master:0;localhost,39664,1320187706355.oldLogCleaner] hbase.Chore(80): Master:0;localhost,39664,1320187706355.oldLogCleaner exiting
> Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on Master:0;localhost,39664,1320187706355
> {noformat}
> it's in
> {noformat}
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> {noformat}
> So that's at least why adding a timeout wont help and may be why it does not end at all. Adding a maximum retry to Threads#threadDumpingIsAlive could help.
> I also wonder if the root cause of the non ending is my modif on the wal, with some threads surprised to have updates that were not written in the wal. Here is the full stack dump:
> {noformat}
> Thread 354 (IPC Client (47) connection to localhost/127.0.0.1:52227 from nkeywal):
> State: TIMED_WAITING
> Blocked count: 360
> Waited count: 359
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:702)
> org.apache.hadoop.ipc.Client$Connection.run(Client.java:744)
> Thread 272 (Master:0;localhost,39664,1320187706355-EventThread):
> State: WAITING
> Blocked count: 0
> Waited count: 4
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@107b954b
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 271 (Master:0;localhost,39664,1320187706355-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 152 (Master:0;localhost,39664,1320187706355):
> State: WAITING
> Blocked count: 217
> Waited count: 174
> Waiting on org.apache.hadoop.hbase.zookeeper.RootRegionTracker@6621477c
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:131)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:104)
> org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRoot(CatalogTracker.java:277)
> org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:523)
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:468)
> org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)
> java.lang.Thread.run(Thread.java:662)
> Thread 165 (LruBlockCache.EvictionThread):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread@3e9d7b56
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:593)
> java.lang.Thread.run(Thread.java:662)
> Thread 151 (main-EventThread):
> State: WAITING
> Blocked count: 40
> Waited count: 39
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22531380
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 150 (main-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 55
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 149 (IPC Server handler 4 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 148 (IPC Server handler 3 on 39664):
> State: WAITING
> Blocked count: 7
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 147 (IPC Server handler 2 on 39664):
> State: WAITING
> Blocked count: 11
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 146 (IPC Server handler 1 on 39664):
> State: WAITING
> Blocked count: 6
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 145 (IPC Server handler 0 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1970
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 133 (IPC Server listener on 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:580)
> Thread 144 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:754)
> Thread 143 (IPC Reader 9 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 142 (IPC Reader 8 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 141 (IPC Reader 7 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 140 (IPC Reader 6 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 139 (IPC Reader 5 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 138 (IPC Reader 4 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 137 (IPC Reader 3 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 136 (IPC Reader 2 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 135 (IPC Reader 1 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 134 (IPC Reader 0 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 124 (LeaseChecker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1252)
> java.lang.Thread.run(Thread.java:662)
> Thread 121 (ProcessThread:-1):
> State: WAITING
> Blocked count: 0
> Waited count: 269
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f5d84e0
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:103)
> Thread 120 (SyncThread:0):
> State: WAITING
> Blocked count: 2
> Waited count: 252
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b25b27c
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:94)
> Thread 119 (SessionTracker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 207
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:142)
> Thread 118 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:21819):
> State: RUNNABLE
> Blocked count: 38
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:232)
> Thread 116 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@2e6f947b):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 113 (IPC Server handler 2 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 112 (IPC Server handler 1 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 111 (IPC Server handler 0 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 106 (IPC Server listener on 39899):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 108 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 102 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@45d1c3cd):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 109 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6]):
> State: TIMED_WAITING
> Blocked count: 137
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 107 (pool-4-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 105 (Timer-3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 104 (1353906974@qtp-1197660496-1 - Acceptor0 SelectChannelConnector@localhost:59858):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 103 (1185716172@qtp-1197660496-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 101 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 98 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 90 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@605b28c9):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 86 (IPC Server handler 2 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 85 (IPC Server handler 1 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 84 (IPC Server handler 0 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 80 (IPC Server listener on 56098):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 82 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 76 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@2b68989e):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 83 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4]):
> State: TIMED_WAITING
> Blocked count: 149
> Waited count: 290
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 81 (pool-3-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 79 (Timer-2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 78 (348878159@qtp-164967086-1 - Acceptor0 SelectChannelConnector@localhost:36427):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 77 (1905729203@qtp-164967086-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 75 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 72 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 61 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@6709da93):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 60 (IPC Server handler 2 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 59 (IPC Server handler 1 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 58 (IPC Server handler 0 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 53 (IPC Server listener on 40388):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 55 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 49 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@48baa31b):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 57 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2]):
> State: TIMED_WAITING
> Blocked count: 148
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 54 (pool-2-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 52 (Timer-1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 51 (1031841975@qtp-1068922666-1 - Acceptor0 SelectChannelConnector@localhost:56535):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 50 (1191055321@qtp-1068922666-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 48 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 45 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 31 (IPC Server handler 9 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 30 (IPC Server handler 8 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 29 (IPC Server handler 7 on 52227):
> State: WAITING
> Blocked count: 2
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 28 (IPC Server handler 6 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 27 (IPC Server handler 5 on 52227):
> State: WAITING
> Blocked count: 4
> Waited count: 68
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 26 (IPC Server handler 4 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 25 (IPC Server handler 3 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 24 (IPC Server handler 2 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 23 (IPC Server handler 1 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 22 (IPC Server handler 0 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 15 (IPC Server listener on 52227):
> State: RUNNABLE
> Blocked count: 43
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 17 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 21 (Timer-0):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 17
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 20 (245022965@qtp-589734556-1 - Acceptor0 SelectChannelConnector@localhost:33079):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 19 (1821050251@qtp-589734556-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 16 (pool-1-thread-1):
> State: RUNNABLE
> Blocked count: 45
> Waited count: 44
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 14 (org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@620fa83):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65)
> java.lang.Thread.run(Thread.java:662)
> Thread 13 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@d5d4de6):
> State: TIMED_WAITING
> Blocked count: 2
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2689)
> java.lang.Thread.run(Thread.java:662)
> Thread 12 (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@73d32e45):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 220
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:368)
> java.lang.Thread.run(Thread.java:662)
> Thread 11 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@19be4777):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 89
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2666)
> java.lang.Thread.run(Thread.java:662)
> Thread 10 (org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@658f7386):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 2
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
> java.lang.Thread.run(Thread.java:662)
> Thread 4 (Signal Dispatcher):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> Thread 3 (Finalizer):
> State: WAITING
> Blocked count: 9
> Waited count: 10
> Waiting on java.lang.ref.ReferenceQueue$Lock@3d1a70a7
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> Thread 2 (Reference Handler):
> State: WAITING
> Blocked count: 11
> Waited count: 12
> Waiting on java.lang.ref.Reference$Lock@270d75a3
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> Thread 1 (main):
> State: RUNNABLE
> Blocked count: 104
> Waited count: 97
> Stack:
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> java.lang.reflect.Method.invoke(Method.java:597)
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
> org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4724) TestAdmin hangs randomly in trunk
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142269#comment-13142269 ]
stack commented on HBASE-4724:
------------------------------
I made HBASE-4725 for the NPE. Looking at the null connection...
> TestAdmin hangs randomly in trunk
> ---------------------------------
>
> Key: HBASE-4724
> URL: https://issues.apache.org/jira/browse/HBASE-4724
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 20111102_4724_TestAdmin.patch, 20111102_4724_TestAdmin.v2.patch
>
>
> fom the logs in my env
> {noformat}
> 2011-11-01 15:48:40,744 WARN [Master:0;localhost,39664,1320187706355] master.AssignmentManager(1471): Failed assignment of -ROOT-,,0.70236052 to localhost,44046,1320187706849, trying to assign elsewhere instead; retry=1
> org.apache.hadoop.hbase.ipc.HBaseRPC$VersionMismatch: Protocol org.apache.hadoop.hbase.ipc.HRegionInterface version mismatch. (client = 28, server = 29)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:185)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:300)
> {noformat}
> Anyway, after this the logs finishes with:
> {noformat}
> 2011-11-01 15:54:35,132 INFO [Master:0;localhost,39664,1320187706355.oldLogCleaner] hbase.Chore(80): Master:0;localhost,39664,1320187706355.oldLogCleaner exiting
> Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on Master:0;localhost,39664,1320187706355
> {noformat}
> it's in
> {noformat}
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> {noformat}
> So that's at least why adding a timeout wont help and may be why it does not end at all. Adding a maximum retry to Threads#threadDumpingIsAlive could help.
> I also wonder if the root cause of the non ending is my modif on the wal, with some threads surprised to have updates that were not written in the wal. Here is the full stack dump:
> {noformat}
> Thread 354 (IPC Client (47) connection to localhost/127.0.0.1:52227 from nkeywal):
> State: TIMED_WAITING
> Blocked count: 360
> Waited count: 359
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:702)
> org.apache.hadoop.ipc.Client$Connection.run(Client.java:744)
> Thread 272 (Master:0;localhost,39664,1320187706355-EventThread):
> State: WAITING
> Blocked count: 0
> Waited count: 4
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@107b954b
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 271 (Master:0;localhost,39664,1320187706355-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 152 (Master:0;localhost,39664,1320187706355):
> State: WAITING
> Blocked count: 217
> Waited count: 174
> Waiting on org.apache.hadoop.hbase.zookeeper.RootRegionTracker@6621477c
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:131)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:104)
> org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRoot(CatalogTracker.java:277)
> org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:523)
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:468)
> org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)
> java.lang.Thread.run(Thread.java:662)
> Thread 165 (LruBlockCache.EvictionThread):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread@3e9d7b56
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:593)
> java.lang.Thread.run(Thread.java:662)
> Thread 151 (main-EventThread):
> State: WAITING
> Blocked count: 40
> Waited count: 39
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22531380
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 150 (main-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 55
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 149 (IPC Server handler 4 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 148 (IPC Server handler 3 on 39664):
> State: WAITING
> Blocked count: 7
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 147 (IPC Server handler 2 on 39664):
> State: WAITING
> Blocked count: 11
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 146 (IPC Server handler 1 on 39664):
> State: WAITING
> Blocked count: 6
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 145 (IPC Server handler 0 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1970
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 133 (IPC Server listener on 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:580)
> Thread 144 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:754)
> Thread 143 (IPC Reader 9 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 142 (IPC Reader 8 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 141 (IPC Reader 7 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 140 (IPC Reader 6 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 139 (IPC Reader 5 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 138 (IPC Reader 4 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 137 (IPC Reader 3 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 136 (IPC Reader 2 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 135 (IPC Reader 1 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 134 (IPC Reader 0 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 124 (LeaseChecker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1252)
> java.lang.Thread.run(Thread.java:662)
> Thread 121 (ProcessThread:-1):
> State: WAITING
> Blocked count: 0
> Waited count: 269
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f5d84e0
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:103)
> Thread 120 (SyncThread:0):
> State: WAITING
> Blocked count: 2
> Waited count: 252
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b25b27c
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:94)
> Thread 119 (SessionTracker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 207
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:142)
> Thread 118 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:21819):
> State: RUNNABLE
> Blocked count: 38
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:232)
> Thread 116 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@2e6f947b):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 113 (IPC Server handler 2 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 112 (IPC Server handler 1 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 111 (IPC Server handler 0 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 106 (IPC Server listener on 39899):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 108 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 102 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@45d1c3cd):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 109 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6]):
> State: TIMED_WAITING
> Blocked count: 137
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 107 (pool-4-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 105 (Timer-3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 104 (1353906974@qtp-1197660496-1 - Acceptor0 SelectChannelConnector@localhost:59858):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 103 (1185716172@qtp-1197660496-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 101 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 98 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 90 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@605b28c9):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 86 (IPC Server handler 2 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 85 (IPC Server handler 1 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 84 (IPC Server handler 0 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 80 (IPC Server listener on 56098):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 82 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 76 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@2b68989e):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 83 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4]):
> State: TIMED_WAITING
> Blocked count: 149
> Waited count: 290
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 81 (pool-3-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 79 (Timer-2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 78 (348878159@qtp-164967086-1 - Acceptor0 SelectChannelConnector@localhost:36427):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 77 (1905729203@qtp-164967086-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 75 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 72 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 61 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@6709da93):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 60 (IPC Server handler 2 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 59 (IPC Server handler 1 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 58 (IPC Server handler 0 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 53 (IPC Server listener on 40388):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 55 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 49 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@48baa31b):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 57 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2]):
> State: TIMED_WAITING
> Blocked count: 148
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 54 (pool-2-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 52 (Timer-1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 51 (1031841975@qtp-1068922666-1 - Acceptor0 SelectChannelConnector@localhost:56535):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 50 (1191055321@qtp-1068922666-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 48 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 45 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 31 (IPC Server handler 9 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 30 (IPC Server handler 8 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 29 (IPC Server handler 7 on 52227):
> State: WAITING
> Blocked count: 2
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 28 (IPC Server handler 6 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 27 (IPC Server handler 5 on 52227):
> State: WAITING
> Blocked count: 4
> Waited count: 68
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 26 (IPC Server handler 4 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 25 (IPC Server handler 3 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 24 (IPC Server handler 2 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 23 (IPC Server handler 1 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 22 (IPC Server handler 0 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 15 (IPC Server listener on 52227):
> State: RUNNABLE
> Blocked count: 43
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 17 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 21 (Timer-0):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 17
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 20 (245022965@qtp-589734556-1 - Acceptor0 SelectChannelConnector@localhost:33079):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 19 (1821050251@qtp-589734556-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 16 (pool-1-thread-1):
> State: RUNNABLE
> Blocked count: 45
> Waited count: 44
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 14 (org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@620fa83):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65)
> java.lang.Thread.run(Thread.java:662)
> Thread 13 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@d5d4de6):
> State: TIMED_WAITING
> Blocked count: 2
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2689)
> java.lang.Thread.run(Thread.java:662)
> Thread 12 (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@73d32e45):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 220
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:368)
> java.lang.Thread.run(Thread.java:662)
> Thread 11 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@19be4777):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 89
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2666)
> java.lang.Thread.run(Thread.java:662)
> Thread 10 (org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@658f7386):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 2
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
> java.lang.Thread.run(Thread.java:662)
> Thread 4 (Signal Dispatcher):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> Thread 3 (Finalizer):
> State: WAITING
> Blocked count: 9
> Waited count: 10
> Waiting on java.lang.ref.ReferenceQueue$Lock@3d1a70a7
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> Thread 2 (Reference Handler):
> State: WAITING
> Blocked count: 11
> Waited count: 12
> Waiting on java.lang.ref.Reference$Lock@270d75a3
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> Thread 1 (main):
> State: RUNNABLE
> Blocked count: 104
> Waited count: 97
> Stack:
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> java.lang.reflect.Method.invoke(Method.java:597)
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
> org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4724) TestAdmin hangs randomly in trunk
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4724:
---------------------------
Status: Open (was: Patch Available)
> TestAdmin hangs randomly in trunk
> ---------------------------------
>
> Key: HBASE-4724
> URL: https://issues.apache.org/jira/browse/HBASE-4724
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Critical
> Attachments: 20111102_4724_TestAdmin.patch, 20111102_4724_TestAdmin.v2.patch
>
>
> fom the logs in my env
> {noformat}
> 2011-11-01 15:48:40,744 WARN [Master:0;localhost,39664,1320187706355] master.AssignmentManager(1471): Failed assignment of -ROOT-,,0.70236052 to localhost,44046,1320187706849, trying to assign elsewhere instead; retry=1
> org.apache.hadoop.hbase.ipc.HBaseRPC$VersionMismatch: Protocol org.apache.hadoop.hbase.ipc.HRegionInterface version mismatch. (client = 28, server = 29)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:185)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:300)
> {noformat}
> Anyway, after this the logs finishes with:
> {noformat}
> 2011-11-01 15:54:35,132 INFO [Master:0;localhost,39664,1320187706355.oldLogCleaner] hbase.Chore(80): Master:0;localhost,39664,1320187706355.oldLogCleaner exiting
> Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on Master:0;localhost,39664,1320187706355
> {noformat}
> it's in
> {noformat}
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> {noformat}
> So that's at least why adding a timeout wont help and may be why it does not end at all. Adding a maximum retry to Threads#threadDumpingIsAlive could help.
> I also wonder if the root cause of the non ending is my modif on the wal, with some threads surprised to have updates that were not written in the wal. Here is the full stack dump:
> {noformat}
> Thread 354 (IPC Client (47) connection to localhost/127.0.0.1:52227 from nkeywal):
> State: TIMED_WAITING
> Blocked count: 360
> Waited count: 359
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:702)
> org.apache.hadoop.ipc.Client$Connection.run(Client.java:744)
> Thread 272 (Master:0;localhost,39664,1320187706355-EventThread):
> State: WAITING
> Blocked count: 0
> Waited count: 4
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@107b954b
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 271 (Master:0;localhost,39664,1320187706355-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 152 (Master:0;localhost,39664,1320187706355):
> State: WAITING
> Blocked count: 217
> Waited count: 174
> Waiting on org.apache.hadoop.hbase.zookeeper.RootRegionTracker@6621477c
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:131)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:104)
> org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRoot(CatalogTracker.java:277)
> org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:523)
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:468)
> org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)
> java.lang.Thread.run(Thread.java:662)
> Thread 165 (LruBlockCache.EvictionThread):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread@3e9d7b56
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:593)
> java.lang.Thread.run(Thread.java:662)
> Thread 151 (main-EventThread):
> State: WAITING
> Blocked count: 40
> Waited count: 39
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22531380
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 150 (main-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 55
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 149 (IPC Server handler 4 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 148 (IPC Server handler 3 on 39664):
> State: WAITING
> Blocked count: 7
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 147 (IPC Server handler 2 on 39664):
> State: WAITING
> Blocked count: 11
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 146 (IPC Server handler 1 on 39664):
> State: WAITING
> Blocked count: 6
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 145 (IPC Server handler 0 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1970
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 133 (IPC Server listener on 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:580)
> Thread 144 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:754)
> Thread 143 (IPC Reader 9 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 142 (IPC Reader 8 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 141 (IPC Reader 7 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 140 (IPC Reader 6 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 139 (IPC Reader 5 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 138 (IPC Reader 4 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 137 (IPC Reader 3 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 136 (IPC Reader 2 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 135 (IPC Reader 1 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 134 (IPC Reader 0 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 124 (LeaseChecker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1252)
> java.lang.Thread.run(Thread.java:662)
> Thread 121 (ProcessThread:-1):
> State: WAITING
> Blocked count: 0
> Waited count: 269
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f5d84e0
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:103)
> Thread 120 (SyncThread:0):
> State: WAITING
> Blocked count: 2
> Waited count: 252
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b25b27c
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:94)
> Thread 119 (SessionTracker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 207
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:142)
> Thread 118 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:21819):
> State: RUNNABLE
> Blocked count: 38
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:232)
> Thread 116 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@2e6f947b):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 113 (IPC Server handler 2 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 112 (IPC Server handler 1 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 111 (IPC Server handler 0 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 106 (IPC Server listener on 39899):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 108 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 102 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@45d1c3cd):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 109 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6]):
> State: TIMED_WAITING
> Blocked count: 137
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 107 (pool-4-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 105 (Timer-3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 104 (1353906974@qtp-1197660496-1 - Acceptor0 SelectChannelConnector@localhost:59858):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 103 (1185716172@qtp-1197660496-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 101 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 98 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 90 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@605b28c9):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 86 (IPC Server handler 2 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 85 (IPC Server handler 1 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 84 (IPC Server handler 0 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 80 (IPC Server listener on 56098):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 82 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 76 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@2b68989e):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 83 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4]):
> State: TIMED_WAITING
> Blocked count: 149
> Waited count: 290
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 81 (pool-3-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 79 (Timer-2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 78 (348878159@qtp-164967086-1 - Acceptor0 SelectChannelConnector@localhost:36427):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 77 (1905729203@qtp-164967086-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 75 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 72 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 61 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@6709da93):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 60 (IPC Server handler 2 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 59 (IPC Server handler 1 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 58 (IPC Server handler 0 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 53 (IPC Server listener on 40388):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 55 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 49 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@48baa31b):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 57 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2]):
> State: TIMED_WAITING
> Blocked count: 148
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 54 (pool-2-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 52 (Timer-1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 51 (1031841975@qtp-1068922666-1 - Acceptor0 SelectChannelConnector@localhost:56535):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 50 (1191055321@qtp-1068922666-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 48 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 45 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 31 (IPC Server handler 9 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 30 (IPC Server handler 8 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 29 (IPC Server handler 7 on 52227):
> State: WAITING
> Blocked count: 2
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 28 (IPC Server handler 6 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 27 (IPC Server handler 5 on 52227):
> State: WAITING
> Blocked count: 4
> Waited count: 68
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 26 (IPC Server handler 4 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 25 (IPC Server handler 3 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 24 (IPC Server handler 2 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 23 (IPC Server handler 1 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 22 (IPC Server handler 0 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 15 (IPC Server listener on 52227):
> State: RUNNABLE
> Blocked count: 43
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 17 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 21 (Timer-0):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 17
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 20 (245022965@qtp-589734556-1 - Acceptor0 SelectChannelConnector@localhost:33079):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 19 (1821050251@qtp-589734556-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 16 (pool-1-thread-1):
> State: RUNNABLE
> Blocked count: 45
> Waited count: 44
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 14 (org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@620fa83):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65)
> java.lang.Thread.run(Thread.java:662)
> Thread 13 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@d5d4de6):
> State: TIMED_WAITING
> Blocked count: 2
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2689)
> java.lang.Thread.run(Thread.java:662)
> Thread 12 (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@73d32e45):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 220
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:368)
> java.lang.Thread.run(Thread.java:662)
> Thread 11 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@19be4777):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 89
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2666)
> java.lang.Thread.run(Thread.java:662)
> Thread 10 (org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@658f7386):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 2
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
> java.lang.Thread.run(Thread.java:662)
> Thread 4 (Signal Dispatcher):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> Thread 3 (Finalizer):
> State: WAITING
> Blocked count: 9
> Waited count: 10
> Waiting on java.lang.ref.ReferenceQueue$Lock@3d1a70a7
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> Thread 2 (Reference Handler):
> State: WAITING
> Blocked count: 11
> Waited count: 12
> Waiting on java.lang.ref.Reference$Lock@270d75a3
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> Thread 1 (main):
> State: RUNNABLE
> Blocked count: 104
> Waited count: 97
> Stack:
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> java.lang.reflect.Method.invoke(Method.java:597)
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
> org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4724) TestAdmin hangs randomly in trunk
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142249#comment-13142249 ]
nkeywal commented on HBASE-4724:
--------------------------------
I recloned the repo, and now it works all the time.
I have this is in the logs, it seems to be new to me:
{noformat}
2011-11-02 03:36:17,271 DEBUG [main] zookeeper.ZKUtil(1034): hconnection Retrieved 29 byte(s) of data from znode /hbase/root-region-server and set watcher; localhost,39688,1320229746489
2011-11-02 03:36:17,272 DEBUG [Finalizer] client.HConnectionManager$HConnectionImplementation(1715): The connection to null has been closed.
2011-11-02 03:36:17,272 DEBUG [Finalizer] client.HConnectionManager$HConnectionImplementation(1734): The connection to null was closed by the finalize method.
2011-11-02 03:36:17,272 DEBUG [Finalizer] client.HConnectionManager$HConnectionImplementation(1715): The connection to null has been closed.
2011-11-02 03:36:17,272 DEBUG [Finalizer] client.HConnectionManager$HConnectionImplementation(1734): The connection to null was closed by the finalize method.
{noformat}
Could it have a side effect in some cases?
FWIW, I also got this once, but the test case succeeded anyway. It something I saw in the past already.
{noformat}
2011-11-02 02:08:13,311 ERROR [MASTER_OPEN_REGION-localhost,40499,1320224444456-3] executor.EventHandler(171): Caught throwable while processing event RS_ZK_REGION_OPENED
java.lang.NullPointerException
at org.apache.hadoop.hbase.master.AssignmentManager.updateTimers(AssignmentManager.java:1059)
at org.apache.hadoop.hbase.master.AssignmentManager.regionOnline(AssignmentManager.java:1033)
at org.apache.hadoop.hbase.master.handler.OpenedRegionHandler.process(OpenedRegionHandler.java:105)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:168)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
2011-11-02 02:08:13,311 INFO [RS_OPEN_REGION-localhost,47967,1320224444999-2] regionserver.HRegion(402): Setting up tabledescriptor config now ...
{noformat}
> TestAdmin hangs randomly in trunk
> ---------------------------------
>
> Key: HBASE-4724
> URL: https://issues.apache.org/jira/browse/HBASE-4724
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 20111102_4724_TestAdmin.patch, 20111102_4724_TestAdmin.v2.patch
>
>
> fom the logs in my env
> {noformat}
> 2011-11-01 15:48:40,744 WARN [Master:0;localhost,39664,1320187706355] master.AssignmentManager(1471): Failed assignment of -ROOT-,,0.70236052 to localhost,44046,1320187706849, trying to assign elsewhere instead; retry=1
> org.apache.hadoop.hbase.ipc.HBaseRPC$VersionMismatch: Protocol org.apache.hadoop.hbase.ipc.HRegionInterface version mismatch. (client = 28, server = 29)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:185)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:300)
> {noformat}
> Anyway, after this the logs finishes with:
> {noformat}
> 2011-11-01 15:54:35,132 INFO [Master:0;localhost,39664,1320187706355.oldLogCleaner] hbase.Chore(80): Master:0;localhost,39664,1320187706355.oldLogCleaner exiting
> Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on Master:0;localhost,39664,1320187706355
> {noformat}
> it's in
> {noformat}
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> {noformat}
> So that's at least why adding a timeout wont help and may be why it does not end at all. Adding a maximum retry to Threads#threadDumpingIsAlive could help.
> I also wonder if the root cause of the non ending is my modif on the wal, with some threads surprised to have updates that were not written in the wal. Here is the full stack dump:
> {noformat}
> Thread 354 (IPC Client (47) connection to localhost/127.0.0.1:52227 from nkeywal):
> State: TIMED_WAITING
> Blocked count: 360
> Waited count: 359
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:702)
> org.apache.hadoop.ipc.Client$Connection.run(Client.java:744)
> Thread 272 (Master:0;localhost,39664,1320187706355-EventThread):
> State: WAITING
> Blocked count: 0
> Waited count: 4
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@107b954b
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 271 (Master:0;localhost,39664,1320187706355-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 152 (Master:0;localhost,39664,1320187706355):
> State: WAITING
> Blocked count: 217
> Waited count: 174
> Waiting on org.apache.hadoop.hbase.zookeeper.RootRegionTracker@6621477c
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:131)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:104)
> org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRoot(CatalogTracker.java:277)
> org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:523)
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:468)
> org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)
> java.lang.Thread.run(Thread.java:662)
> Thread 165 (LruBlockCache.EvictionThread):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread@3e9d7b56
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:593)
> java.lang.Thread.run(Thread.java:662)
> Thread 151 (main-EventThread):
> State: WAITING
> Blocked count: 40
> Waited count: 39
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22531380
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 150 (main-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 55
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 149 (IPC Server handler 4 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 148 (IPC Server handler 3 on 39664):
> State: WAITING
> Blocked count: 7
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 147 (IPC Server handler 2 on 39664):
> State: WAITING
> Blocked count: 11
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 146 (IPC Server handler 1 on 39664):
> State: WAITING
> Blocked count: 6
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 145 (IPC Server handler 0 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1970
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 133 (IPC Server listener on 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:580)
> Thread 144 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:754)
> Thread 143 (IPC Reader 9 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 142 (IPC Reader 8 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 141 (IPC Reader 7 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 140 (IPC Reader 6 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 139 (IPC Reader 5 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 138 (IPC Reader 4 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 137 (IPC Reader 3 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 136 (IPC Reader 2 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 135 (IPC Reader 1 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 134 (IPC Reader 0 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 124 (LeaseChecker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1252)
> java.lang.Thread.run(Thread.java:662)
> Thread 121 (ProcessThread:-1):
> State: WAITING
> Blocked count: 0
> Waited count: 269
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f5d84e0
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:103)
> Thread 120 (SyncThread:0):
> State: WAITING
> Blocked count: 2
> Waited count: 252
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b25b27c
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:94)
> Thread 119 (SessionTracker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 207
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:142)
> Thread 118 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:21819):
> State: RUNNABLE
> Blocked count: 38
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:232)
> Thread 116 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@2e6f947b):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 113 (IPC Server handler 2 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 112 (IPC Server handler 1 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 111 (IPC Server handler 0 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 106 (IPC Server listener on 39899):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 108 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 102 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@45d1c3cd):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 109 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6]):
> State: TIMED_WAITING
> Blocked count: 137
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 107 (pool-4-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 105 (Timer-3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 104 (1353906974@qtp-1197660496-1 - Acceptor0 SelectChannelConnector@localhost:59858):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 103 (1185716172@qtp-1197660496-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 101 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 98 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 90 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@605b28c9):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 86 (IPC Server handler 2 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 85 (IPC Server handler 1 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 84 (IPC Server handler 0 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 80 (IPC Server listener on 56098):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 82 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 76 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@2b68989e):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 83 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4]):
> State: TIMED_WAITING
> Blocked count: 149
> Waited count: 290
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 81 (pool-3-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 79 (Timer-2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 78 (348878159@qtp-164967086-1 - Acceptor0 SelectChannelConnector@localhost:36427):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 77 (1905729203@qtp-164967086-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 75 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 72 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 61 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@6709da93):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 60 (IPC Server handler 2 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 59 (IPC Server handler 1 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 58 (IPC Server handler 0 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 53 (IPC Server listener on 40388):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 55 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 49 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@48baa31b):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 57 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2]):
> State: TIMED_WAITING
> Blocked count: 148
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 54 (pool-2-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 52 (Timer-1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 51 (1031841975@qtp-1068922666-1 - Acceptor0 SelectChannelConnector@localhost:56535):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 50 (1191055321@qtp-1068922666-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 48 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 45 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 31 (IPC Server handler 9 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 30 (IPC Server handler 8 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 29 (IPC Server handler 7 on 52227):
> State: WAITING
> Blocked count: 2
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 28 (IPC Server handler 6 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 27 (IPC Server handler 5 on 52227):
> State: WAITING
> Blocked count: 4
> Waited count: 68
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 26 (IPC Server handler 4 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 25 (IPC Server handler 3 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 24 (IPC Server handler 2 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 23 (IPC Server handler 1 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 22 (IPC Server handler 0 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 15 (IPC Server listener on 52227):
> State: RUNNABLE
> Blocked count: 43
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 17 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 21 (Timer-0):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 17
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 20 (245022965@qtp-589734556-1 - Acceptor0 SelectChannelConnector@localhost:33079):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 19 (1821050251@qtp-589734556-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 16 (pool-1-thread-1):
> State: RUNNABLE
> Blocked count: 45
> Waited count: 44
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 14 (org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@620fa83):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65)
> java.lang.Thread.run(Thread.java:662)
> Thread 13 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@d5d4de6):
> State: TIMED_WAITING
> Blocked count: 2
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2689)
> java.lang.Thread.run(Thread.java:662)
> Thread 12 (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@73d32e45):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 220
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:368)
> java.lang.Thread.run(Thread.java:662)
> Thread 11 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@19be4777):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 89
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2666)
> java.lang.Thread.run(Thread.java:662)
> Thread 10 (org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@658f7386):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 2
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
> java.lang.Thread.run(Thread.java:662)
> Thread 4 (Signal Dispatcher):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> Thread 3 (Finalizer):
> State: WAITING
> Blocked count: 9
> Waited count: 10
> Waiting on java.lang.ref.ReferenceQueue$Lock@3d1a70a7
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> Thread 2 (Reference Handler):
> State: WAITING
> Blocked count: 11
> Waited count: 12
> Waiting on java.lang.ref.Reference$Lock@270d75a3
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> Thread 1 (main):
> State: RUNNABLE
> Blocked count: 104
> Waited count: 97
> Stack:
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> java.lang.reflect.Method.invoke(Method.java:597)
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
> org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4724) TestAdmin hangs randomly in trunk
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4724:
---------------------------
Attachment: 20111102_4724_TestAdmin.patch
> TestAdmin hangs randomly in trunk
> ---------------------------------
>
> Key: HBASE-4724
> URL: https://issues.apache.org/jira/browse/HBASE-4724
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Priority: Critical
> Attachments: 20111102_4724_TestAdmin.patch
>
>
> fom the logs in my env
> {noformat}
> 2011-11-01 15:48:40,744 WARN [Master:0;localhost,39664,1320187706355] master.AssignmentManager(1471): Failed assignment of -ROOT-,,0.70236052 to localhost,44046,1320187706849, trying to assign elsewhere instead; retry=1
> org.apache.hadoop.hbase.ipc.HBaseRPC$VersionMismatch: Protocol org.apache.hadoop.hbase.ipc.HRegionInterface version mismatch. (client = 28, server = 29)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:185)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:300)
> {noformat}
> Anyway, after this the logs finishes with:
> {noformat}
> 2011-11-01 15:54:35,132 INFO [Master:0;localhost,39664,1320187706355.oldLogCleaner] hbase.Chore(80): Master:0;localhost,39664,1320187706355.oldLogCleaner exiting
> Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on Master:0;localhost,39664,1320187706355
> {noformat}
> it's in
> {noformat}
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> {noformat}
> So that's at least why adding a timeout wont help and may be why it does not end at all. Adding a maximum retry to Threads#threadDumpingIsAlive could help.
> I also wonder if the root cause of the non ending is my modif on the wal, with some threads surprised to have updates that were not written in the wal. Here is the full stack dump:
> {noformat}
> Thread 354 (IPC Client (47) connection to localhost/127.0.0.1:52227 from nkeywal):
> State: TIMED_WAITING
> Blocked count: 360
> Waited count: 359
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:702)
> org.apache.hadoop.ipc.Client$Connection.run(Client.java:744)
> Thread 272 (Master:0;localhost,39664,1320187706355-EventThread):
> State: WAITING
> Blocked count: 0
> Waited count: 4
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@107b954b
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 271 (Master:0;localhost,39664,1320187706355-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 152 (Master:0;localhost,39664,1320187706355):
> State: WAITING
> Blocked count: 217
> Waited count: 174
> Waiting on org.apache.hadoop.hbase.zookeeper.RootRegionTracker@6621477c
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:131)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:104)
> org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRoot(CatalogTracker.java:277)
> org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:523)
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:468)
> org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)
> java.lang.Thread.run(Thread.java:662)
> Thread 165 (LruBlockCache.EvictionThread):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread@3e9d7b56
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:593)
> java.lang.Thread.run(Thread.java:662)
> Thread 151 (main-EventThread):
> State: WAITING
> Blocked count: 40
> Waited count: 39
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22531380
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 150 (main-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 55
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 149 (IPC Server handler 4 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 148 (IPC Server handler 3 on 39664):
> State: WAITING
> Blocked count: 7
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 147 (IPC Server handler 2 on 39664):
> State: WAITING
> Blocked count: 11
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 146 (IPC Server handler 1 on 39664):
> State: WAITING
> Blocked count: 6
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 145 (IPC Server handler 0 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1970
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 133 (IPC Server listener on 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:580)
> Thread 144 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:754)
> Thread 143 (IPC Reader 9 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 142 (IPC Reader 8 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 141 (IPC Reader 7 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 140 (IPC Reader 6 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 139 (IPC Reader 5 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 138 (IPC Reader 4 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 137 (IPC Reader 3 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 136 (IPC Reader 2 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 135 (IPC Reader 1 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 134 (IPC Reader 0 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 124 (LeaseChecker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1252)
> java.lang.Thread.run(Thread.java:662)
> Thread 121 (ProcessThread:-1):
> State: WAITING
> Blocked count: 0
> Waited count: 269
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f5d84e0
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:103)
> Thread 120 (SyncThread:0):
> State: WAITING
> Blocked count: 2
> Waited count: 252
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b25b27c
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:94)
> Thread 119 (SessionTracker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 207
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:142)
> Thread 118 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:21819):
> State: RUNNABLE
> Blocked count: 38
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:232)
> Thread 116 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@2e6f947b):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 113 (IPC Server handler 2 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 112 (IPC Server handler 1 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 111 (IPC Server handler 0 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 106 (IPC Server listener on 39899):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 108 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 102 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@45d1c3cd):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 109 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6]):
> State: TIMED_WAITING
> Blocked count: 137
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 107 (pool-4-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 105 (Timer-3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 104 (1353906974@qtp-1197660496-1 - Acceptor0 SelectChannelConnector@localhost:59858):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 103 (1185716172@qtp-1197660496-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 101 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 98 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 90 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@605b28c9):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 86 (IPC Server handler 2 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 85 (IPC Server handler 1 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 84 (IPC Server handler 0 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 80 (IPC Server listener on 56098):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 82 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 76 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@2b68989e):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 83 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4]):
> State: TIMED_WAITING
> Blocked count: 149
> Waited count: 290
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 81 (pool-3-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 79 (Timer-2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 78 (348878159@qtp-164967086-1 - Acceptor0 SelectChannelConnector@localhost:36427):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 77 (1905729203@qtp-164967086-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 75 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 72 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 61 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@6709da93):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 60 (IPC Server handler 2 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 59 (IPC Server handler 1 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 58 (IPC Server handler 0 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 53 (IPC Server listener on 40388):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 55 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 49 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@48baa31b):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 57 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2]):
> State: TIMED_WAITING
> Blocked count: 148
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 54 (pool-2-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 52 (Timer-1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 51 (1031841975@qtp-1068922666-1 - Acceptor0 SelectChannelConnector@localhost:56535):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 50 (1191055321@qtp-1068922666-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 48 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 45 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 31 (IPC Server handler 9 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 30 (IPC Server handler 8 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 29 (IPC Server handler 7 on 52227):
> State: WAITING
> Blocked count: 2
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 28 (IPC Server handler 6 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 27 (IPC Server handler 5 on 52227):
> State: WAITING
> Blocked count: 4
> Waited count: 68
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 26 (IPC Server handler 4 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 25 (IPC Server handler 3 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 24 (IPC Server handler 2 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 23 (IPC Server handler 1 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 22 (IPC Server handler 0 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 15 (IPC Server listener on 52227):
> State: RUNNABLE
> Blocked count: 43
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 17 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 21 (Timer-0):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 17
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 20 (245022965@qtp-589734556-1 - Acceptor0 SelectChannelConnector@localhost:33079):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 19 (1821050251@qtp-589734556-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 16 (pool-1-thread-1):
> State: RUNNABLE
> Blocked count: 45
> Waited count: 44
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 14 (org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@620fa83):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65)
> java.lang.Thread.run(Thread.java:662)
> Thread 13 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@d5d4de6):
> State: TIMED_WAITING
> Blocked count: 2
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2689)
> java.lang.Thread.run(Thread.java:662)
> Thread 12 (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@73d32e45):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 220
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:368)
> java.lang.Thread.run(Thread.java:662)
> Thread 11 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@19be4777):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 89
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2666)
> java.lang.Thread.run(Thread.java:662)
> Thread 10 (org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@658f7386):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 2
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
> java.lang.Thread.run(Thread.java:662)
> Thread 4 (Signal Dispatcher):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> Thread 3 (Finalizer):
> State: WAITING
> Blocked count: 9
> Waited count: 10
> Waiting on java.lang.ref.ReferenceQueue$Lock@3d1a70a7
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> Thread 2 (Reference Handler):
> State: WAITING
> Blocked count: 11
> Waited count: 12
> Waiting on java.lang.ref.Reference$Lock@270d75a3
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> Thread 1 (main):
> State: RUNNABLE
> Blocked count: 104
> Waited count: 97
> Stack:
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> java.lang.reflect.Method.invoke(Method.java:597)
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
> org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4724) TestAdmin hangs randomly in trunk
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4724:
---------------------------
Status: Patch Available (was: Open)
> TestAdmin hangs randomly in trunk
> ---------------------------------
>
> Key: HBASE-4724
> URL: https://issues.apache.org/jira/browse/HBASE-4724
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Critical
> Attachments: 20111102_4724_TestAdmin.patch, 20111102_4724_TestAdmin.v2.patch
>
>
> fom the logs in my env
> {noformat}
> 2011-11-01 15:48:40,744 WARN [Master:0;localhost,39664,1320187706355] master.AssignmentManager(1471): Failed assignment of -ROOT-,,0.70236052 to localhost,44046,1320187706849, trying to assign elsewhere instead; retry=1
> org.apache.hadoop.hbase.ipc.HBaseRPC$VersionMismatch: Protocol org.apache.hadoop.hbase.ipc.HRegionInterface version mismatch. (client = 28, server = 29)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:185)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:300)
> {noformat}
> Anyway, after this the logs finishes with:
> {noformat}
> 2011-11-01 15:54:35,132 INFO [Master:0;localhost,39664,1320187706355.oldLogCleaner] hbase.Chore(80): Master:0;localhost,39664,1320187706355.oldLogCleaner exiting
> Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on Master:0;localhost,39664,1320187706355
> {noformat}
> it's in
> {noformat}
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> {noformat}
> So that's at least why adding a timeout wont help and may be why it does not end at all. Adding a maximum retry to Threads#threadDumpingIsAlive could help.
> I also wonder if the root cause of the non ending is my modif on the wal, with some threads surprised to have updates that were not written in the wal. Here is the full stack dump:
> {noformat}
> Thread 354 (IPC Client (47) connection to localhost/127.0.0.1:52227 from nkeywal):
> State: TIMED_WAITING
> Blocked count: 360
> Waited count: 359
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:702)
> org.apache.hadoop.ipc.Client$Connection.run(Client.java:744)
> Thread 272 (Master:0;localhost,39664,1320187706355-EventThread):
> State: WAITING
> Blocked count: 0
> Waited count: 4
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@107b954b
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 271 (Master:0;localhost,39664,1320187706355-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 152 (Master:0;localhost,39664,1320187706355):
> State: WAITING
> Blocked count: 217
> Waited count: 174
> Waiting on org.apache.hadoop.hbase.zookeeper.RootRegionTracker@6621477c
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:131)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:104)
> org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRoot(CatalogTracker.java:277)
> org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:523)
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:468)
> org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)
> java.lang.Thread.run(Thread.java:662)
> Thread 165 (LruBlockCache.EvictionThread):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread@3e9d7b56
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:593)
> java.lang.Thread.run(Thread.java:662)
> Thread 151 (main-EventThread):
> State: WAITING
> Blocked count: 40
> Waited count: 39
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22531380
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 150 (main-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 55
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 149 (IPC Server handler 4 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 148 (IPC Server handler 3 on 39664):
> State: WAITING
> Blocked count: 7
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 147 (IPC Server handler 2 on 39664):
> State: WAITING
> Blocked count: 11
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 146 (IPC Server handler 1 on 39664):
> State: WAITING
> Blocked count: 6
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 145 (IPC Server handler 0 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1970
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 133 (IPC Server listener on 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:580)
> Thread 144 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:754)
> Thread 143 (IPC Reader 9 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 142 (IPC Reader 8 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 141 (IPC Reader 7 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 140 (IPC Reader 6 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 139 (IPC Reader 5 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 138 (IPC Reader 4 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 137 (IPC Reader 3 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 136 (IPC Reader 2 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 135 (IPC Reader 1 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 134 (IPC Reader 0 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 124 (LeaseChecker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1252)
> java.lang.Thread.run(Thread.java:662)
> Thread 121 (ProcessThread:-1):
> State: WAITING
> Blocked count: 0
> Waited count: 269
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f5d84e0
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:103)
> Thread 120 (SyncThread:0):
> State: WAITING
> Blocked count: 2
> Waited count: 252
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b25b27c
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:94)
> Thread 119 (SessionTracker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 207
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:142)
> Thread 118 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:21819):
> State: RUNNABLE
> Blocked count: 38
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:232)
> Thread 116 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@2e6f947b):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 113 (IPC Server handler 2 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 112 (IPC Server handler 1 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 111 (IPC Server handler 0 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 106 (IPC Server listener on 39899):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 108 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 102 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@45d1c3cd):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 109 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6]):
> State: TIMED_WAITING
> Blocked count: 137
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 107 (pool-4-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 105 (Timer-3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 104 (1353906974@qtp-1197660496-1 - Acceptor0 SelectChannelConnector@localhost:59858):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 103 (1185716172@qtp-1197660496-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 101 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 98 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 90 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@605b28c9):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 86 (IPC Server handler 2 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 85 (IPC Server handler 1 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 84 (IPC Server handler 0 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 80 (IPC Server listener on 56098):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 82 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 76 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@2b68989e):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 83 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4]):
> State: TIMED_WAITING
> Blocked count: 149
> Waited count: 290
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 81 (pool-3-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 79 (Timer-2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 78 (348878159@qtp-164967086-1 - Acceptor0 SelectChannelConnector@localhost:36427):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 77 (1905729203@qtp-164967086-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 75 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 72 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 61 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@6709da93):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 60 (IPC Server handler 2 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 59 (IPC Server handler 1 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 58 (IPC Server handler 0 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 53 (IPC Server listener on 40388):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 55 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 49 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@48baa31b):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 57 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2]):
> State: TIMED_WAITING
> Blocked count: 148
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 54 (pool-2-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 52 (Timer-1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 51 (1031841975@qtp-1068922666-1 - Acceptor0 SelectChannelConnector@localhost:56535):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 50 (1191055321@qtp-1068922666-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 48 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 45 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 31 (IPC Server handler 9 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 30 (IPC Server handler 8 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 29 (IPC Server handler 7 on 52227):
> State: WAITING
> Blocked count: 2
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 28 (IPC Server handler 6 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 27 (IPC Server handler 5 on 52227):
> State: WAITING
> Blocked count: 4
> Waited count: 68
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 26 (IPC Server handler 4 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 25 (IPC Server handler 3 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 24 (IPC Server handler 2 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 23 (IPC Server handler 1 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 22 (IPC Server handler 0 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 15 (IPC Server listener on 52227):
> State: RUNNABLE
> Blocked count: 43
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 17 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 21 (Timer-0):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 17
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 20 (245022965@qtp-589734556-1 - Acceptor0 SelectChannelConnector@localhost:33079):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 19 (1821050251@qtp-589734556-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 16 (pool-1-thread-1):
> State: RUNNABLE
> Blocked count: 45
> Waited count: 44
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 14 (org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@620fa83):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65)
> java.lang.Thread.run(Thread.java:662)
> Thread 13 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@d5d4de6):
> State: TIMED_WAITING
> Blocked count: 2
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2689)
> java.lang.Thread.run(Thread.java:662)
> Thread 12 (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@73d32e45):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 220
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:368)
> java.lang.Thread.run(Thread.java:662)
> Thread 11 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@19be4777):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 89
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2666)
> java.lang.Thread.run(Thread.java:662)
> Thread 10 (org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@658f7386):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 2
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
> java.lang.Thread.run(Thread.java:662)
> Thread 4 (Signal Dispatcher):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> Thread 3 (Finalizer):
> State: WAITING
> Blocked count: 9
> Waited count: 10
> Waiting on java.lang.ref.ReferenceQueue$Lock@3d1a70a7
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> Thread 2 (Reference Handler):
> State: WAITING
> Blocked count: 11
> Waited count: 12
> Waiting on java.lang.ref.Reference$Lock@270d75a3
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> Thread 1 (main):
> State: RUNNABLE
> Blocked count: 104
> Waited count: 97
> Stack:
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> java.lang.reflect.Method.invoke(Method.java:597)
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
> org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4724) TestAdmin hangs randomly in trunk
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142013#comment-13142013 ]
Hadoop QA commented on HBASE-4724:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12501929/20111102_4724_TestAdmin.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/134//console
This message is automatically generated.
> TestAdmin hangs randomly in trunk
> ---------------------------------
>
> Key: HBASE-4724
> URL: https://issues.apache.org/jira/browse/HBASE-4724
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Critical
> Attachments: 20111102_4724_TestAdmin.patch
>
>
> fom the logs in my env
> {noformat}
> 2011-11-01 15:48:40,744 WARN [Master:0;localhost,39664,1320187706355] master.AssignmentManager(1471): Failed assignment of -ROOT-,,0.70236052 to localhost,44046,1320187706849, trying to assign elsewhere instead; retry=1
> org.apache.hadoop.hbase.ipc.HBaseRPC$VersionMismatch: Protocol org.apache.hadoop.hbase.ipc.HRegionInterface version mismatch. (client = 28, server = 29)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:185)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:300)
> {noformat}
> Anyway, after this the logs finishes with:
> {noformat}
> 2011-11-01 15:54:35,132 INFO [Master:0;localhost,39664,1320187706355.oldLogCleaner] hbase.Chore(80): Master:0;localhost,39664,1320187706355.oldLogCleaner exiting
> Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on Master:0;localhost,39664,1320187706355
> {noformat}
> it's in
> {noformat}
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> {noformat}
> So that's at least why adding a timeout wont help and may be why it does not end at all. Adding a maximum retry to Threads#threadDumpingIsAlive could help.
> I also wonder if the root cause of the non ending is my modif on the wal, with some threads surprised to have updates that were not written in the wal. Here is the full stack dump:
> {noformat}
> Thread 354 (IPC Client (47) connection to localhost/127.0.0.1:52227 from nkeywal):
> State: TIMED_WAITING
> Blocked count: 360
> Waited count: 359
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:702)
> org.apache.hadoop.ipc.Client$Connection.run(Client.java:744)
> Thread 272 (Master:0;localhost,39664,1320187706355-EventThread):
> State: WAITING
> Blocked count: 0
> Waited count: 4
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@107b954b
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 271 (Master:0;localhost,39664,1320187706355-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 152 (Master:0;localhost,39664,1320187706355):
> State: WAITING
> Blocked count: 217
> Waited count: 174
> Waiting on org.apache.hadoop.hbase.zookeeper.RootRegionTracker@6621477c
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:131)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:104)
> org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRoot(CatalogTracker.java:277)
> org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:523)
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:468)
> org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)
> java.lang.Thread.run(Thread.java:662)
> Thread 165 (LruBlockCache.EvictionThread):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread@3e9d7b56
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:593)
> java.lang.Thread.run(Thread.java:662)
> Thread 151 (main-EventThread):
> State: WAITING
> Blocked count: 40
> Waited count: 39
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22531380
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 150 (main-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 55
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 149 (IPC Server handler 4 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 148 (IPC Server handler 3 on 39664):
> State: WAITING
> Blocked count: 7
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 147 (IPC Server handler 2 on 39664):
> State: WAITING
> Blocked count: 11
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 146 (IPC Server handler 1 on 39664):
> State: WAITING
> Blocked count: 6
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 145 (IPC Server handler 0 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1970
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 133 (IPC Server listener on 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:580)
> Thread 144 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:754)
> Thread 143 (IPC Reader 9 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 142 (IPC Reader 8 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 141 (IPC Reader 7 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 140 (IPC Reader 6 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 139 (IPC Reader 5 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 138 (IPC Reader 4 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 137 (IPC Reader 3 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 136 (IPC Reader 2 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 135 (IPC Reader 1 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 134 (IPC Reader 0 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 124 (LeaseChecker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1252)
> java.lang.Thread.run(Thread.java:662)
> Thread 121 (ProcessThread:-1):
> State: WAITING
> Blocked count: 0
> Waited count: 269
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f5d84e0
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:103)
> Thread 120 (SyncThread:0):
> State: WAITING
> Blocked count: 2
> Waited count: 252
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b25b27c
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:94)
> Thread 119 (SessionTracker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 207
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:142)
> Thread 118 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:21819):
> State: RUNNABLE
> Blocked count: 38
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:232)
> Thread 116 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@2e6f947b):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 113 (IPC Server handler 2 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 112 (IPC Server handler 1 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 111 (IPC Server handler 0 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 106 (IPC Server listener on 39899):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 108 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 102 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@45d1c3cd):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 109 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6]):
> State: TIMED_WAITING
> Blocked count: 137
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 107 (pool-4-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 105 (Timer-3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 104 (1353906974@qtp-1197660496-1 - Acceptor0 SelectChannelConnector@localhost:59858):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 103 (1185716172@qtp-1197660496-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 101 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 98 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 90 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@605b28c9):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 86 (IPC Server handler 2 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 85 (IPC Server handler 1 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 84 (IPC Server handler 0 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 80 (IPC Server listener on 56098):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 82 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 76 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@2b68989e):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 83 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4]):
> State: TIMED_WAITING
> Blocked count: 149
> Waited count: 290
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 81 (pool-3-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 79 (Timer-2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 78 (348878159@qtp-164967086-1 - Acceptor0 SelectChannelConnector@localhost:36427):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 77 (1905729203@qtp-164967086-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 75 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 72 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 61 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@6709da93):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 60 (IPC Server handler 2 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 59 (IPC Server handler 1 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 58 (IPC Server handler 0 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 53 (IPC Server listener on 40388):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 55 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 49 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@48baa31b):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 57 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2]):
> State: TIMED_WAITING
> Blocked count: 148
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 54 (pool-2-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 52 (Timer-1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 51 (1031841975@qtp-1068922666-1 - Acceptor0 SelectChannelConnector@localhost:56535):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 50 (1191055321@qtp-1068922666-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 48 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 45 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 31 (IPC Server handler 9 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 30 (IPC Server handler 8 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 29 (IPC Server handler 7 on 52227):
> State: WAITING
> Blocked count: 2
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 28 (IPC Server handler 6 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 27 (IPC Server handler 5 on 52227):
> State: WAITING
> Blocked count: 4
> Waited count: 68
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 26 (IPC Server handler 4 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 25 (IPC Server handler 3 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 24 (IPC Server handler 2 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 23 (IPC Server handler 1 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 22 (IPC Server handler 0 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 15 (IPC Server listener on 52227):
> State: RUNNABLE
> Blocked count: 43
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 17 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 21 (Timer-0):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 17
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 20 (245022965@qtp-589734556-1 - Acceptor0 SelectChannelConnector@localhost:33079):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 19 (1821050251@qtp-589734556-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 16 (pool-1-thread-1):
> State: RUNNABLE
> Blocked count: 45
> Waited count: 44
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 14 (org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@620fa83):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65)
> java.lang.Thread.run(Thread.java:662)
> Thread 13 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@d5d4de6):
> State: TIMED_WAITING
> Blocked count: 2
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2689)
> java.lang.Thread.run(Thread.java:662)
> Thread 12 (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@73d32e45):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 220
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:368)
> java.lang.Thread.run(Thread.java:662)
> Thread 11 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@19be4777):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 89
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2666)
> java.lang.Thread.run(Thread.java:662)
> Thread 10 (org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@658f7386):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 2
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
> java.lang.Thread.run(Thread.java:662)
> Thread 4 (Signal Dispatcher):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> Thread 3 (Finalizer):
> State: WAITING
> Blocked count: 9
> Waited count: 10
> Waiting on java.lang.ref.ReferenceQueue$Lock@3d1a70a7
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> Thread 2 (Reference Handler):
> State: WAITING
> Blocked count: 11
> Waited count: 12
> Waiting on java.lang.ref.Reference$Lock@270d75a3
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> Thread 1 (main):
> State: RUNNABLE
> Blocked count: 104
> Waited count: 97
> Stack:
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> java.lang.reflect.Method.invoke(Method.java:597)
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
> org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4724) TestAdmin hangs randomly in trunk
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142215#comment-13142215 ]
stack commented on HBASE-4724:
------------------------------
I applied your restore of wal behavior in v2. Lets see how it plays out on jenkins.
> TestAdmin hangs randomly in trunk
> ---------------------------------
>
> Key: HBASE-4724
> URL: https://issues.apache.org/jira/browse/HBASE-4724
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 20111102_4724_TestAdmin.patch, 20111102_4724_TestAdmin.v2.patch
>
>
> fom the logs in my env
> {noformat}
> 2011-11-01 15:48:40,744 WARN [Master:0;localhost,39664,1320187706355] master.AssignmentManager(1471): Failed assignment of -ROOT-,,0.70236052 to localhost,44046,1320187706849, trying to assign elsewhere instead; retry=1
> org.apache.hadoop.hbase.ipc.HBaseRPC$VersionMismatch: Protocol org.apache.hadoop.hbase.ipc.HRegionInterface version mismatch. (client = 28, server = 29)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:185)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:300)
> {noformat}
> Anyway, after this the logs finishes with:
> {noformat}
> 2011-11-01 15:54:35,132 INFO [Master:0;localhost,39664,1320187706355.oldLogCleaner] hbase.Chore(80): Master:0;localhost,39664,1320187706355.oldLogCleaner exiting
> Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on Master:0;localhost,39664,1320187706355
> {noformat}
> it's in
> {noformat}
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> {noformat}
> So that's at least why adding a timeout wont help and may be why it does not end at all. Adding a maximum retry to Threads#threadDumpingIsAlive could help.
> I also wonder if the root cause of the non ending is my modif on the wal, with some threads surprised to have updates that were not written in the wal. Here is the full stack dump:
> {noformat}
> Thread 354 (IPC Client (47) connection to localhost/127.0.0.1:52227 from nkeywal):
> State: TIMED_WAITING
> Blocked count: 360
> Waited count: 359
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:702)
> org.apache.hadoop.ipc.Client$Connection.run(Client.java:744)
> Thread 272 (Master:0;localhost,39664,1320187706355-EventThread):
> State: WAITING
> Blocked count: 0
> Waited count: 4
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@107b954b
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 271 (Master:0;localhost,39664,1320187706355-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 152 (Master:0;localhost,39664,1320187706355):
> State: WAITING
> Blocked count: 217
> Waited count: 174
> Waiting on org.apache.hadoop.hbase.zookeeper.RootRegionTracker@6621477c
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:131)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:104)
> org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRoot(CatalogTracker.java:277)
> org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:523)
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:468)
> org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)
> java.lang.Thread.run(Thread.java:662)
> Thread 165 (LruBlockCache.EvictionThread):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread@3e9d7b56
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:593)
> java.lang.Thread.run(Thread.java:662)
> Thread 151 (main-EventThread):
> State: WAITING
> Blocked count: 40
> Waited count: 39
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22531380
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 150 (main-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 55
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 149 (IPC Server handler 4 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 148 (IPC Server handler 3 on 39664):
> State: WAITING
> Blocked count: 7
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 147 (IPC Server handler 2 on 39664):
> State: WAITING
> Blocked count: 11
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 146 (IPC Server handler 1 on 39664):
> State: WAITING
> Blocked count: 6
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 145 (IPC Server handler 0 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1970
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 133 (IPC Server listener on 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:580)
> Thread 144 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:754)
> Thread 143 (IPC Reader 9 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 142 (IPC Reader 8 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 141 (IPC Reader 7 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 140 (IPC Reader 6 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 139 (IPC Reader 5 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 138 (IPC Reader 4 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 137 (IPC Reader 3 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 136 (IPC Reader 2 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 135 (IPC Reader 1 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 134 (IPC Reader 0 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 124 (LeaseChecker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1252)
> java.lang.Thread.run(Thread.java:662)
> Thread 121 (ProcessThread:-1):
> State: WAITING
> Blocked count: 0
> Waited count: 269
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f5d84e0
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:103)
> Thread 120 (SyncThread:0):
> State: WAITING
> Blocked count: 2
> Waited count: 252
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b25b27c
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:94)
> Thread 119 (SessionTracker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 207
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:142)
> Thread 118 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:21819):
> State: RUNNABLE
> Blocked count: 38
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:232)
> Thread 116 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@2e6f947b):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 113 (IPC Server handler 2 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 112 (IPC Server handler 1 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 111 (IPC Server handler 0 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 106 (IPC Server listener on 39899):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 108 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 102 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@45d1c3cd):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 109 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6]):
> State: TIMED_WAITING
> Blocked count: 137
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 107 (pool-4-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 105 (Timer-3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 104 (1353906974@qtp-1197660496-1 - Acceptor0 SelectChannelConnector@localhost:59858):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 103 (1185716172@qtp-1197660496-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 101 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 98 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 90 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@605b28c9):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 86 (IPC Server handler 2 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 85 (IPC Server handler 1 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 84 (IPC Server handler 0 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 80 (IPC Server listener on 56098):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 82 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 76 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@2b68989e):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 83 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4]):
> State: TIMED_WAITING
> Blocked count: 149
> Waited count: 290
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 81 (pool-3-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 79 (Timer-2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 78 (348878159@qtp-164967086-1 - Acceptor0 SelectChannelConnector@localhost:36427):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 77 (1905729203@qtp-164967086-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 75 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 72 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 61 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@6709da93):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 60 (IPC Server handler 2 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 59 (IPC Server handler 1 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 58 (IPC Server handler 0 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 53 (IPC Server listener on 40388):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 55 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 49 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@48baa31b):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 57 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2]):
> State: TIMED_WAITING
> Blocked count: 148
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 54 (pool-2-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 52 (Timer-1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 51 (1031841975@qtp-1068922666-1 - Acceptor0 SelectChannelConnector@localhost:56535):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 50 (1191055321@qtp-1068922666-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 48 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 45 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 31 (IPC Server handler 9 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 30 (IPC Server handler 8 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 29 (IPC Server handler 7 on 52227):
> State: WAITING
> Blocked count: 2
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 28 (IPC Server handler 6 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 27 (IPC Server handler 5 on 52227):
> State: WAITING
> Blocked count: 4
> Waited count: 68
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 26 (IPC Server handler 4 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 25 (IPC Server handler 3 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 24 (IPC Server handler 2 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 23 (IPC Server handler 1 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 22 (IPC Server handler 0 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 15 (IPC Server listener on 52227):
> State: RUNNABLE
> Blocked count: 43
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 17 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 21 (Timer-0):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 17
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 20 (245022965@qtp-589734556-1 - Acceptor0 SelectChannelConnector@localhost:33079):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 19 (1821050251@qtp-589734556-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 16 (pool-1-thread-1):
> State: RUNNABLE
> Blocked count: 45
> Waited count: 44
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 14 (org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@620fa83):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65)
> java.lang.Thread.run(Thread.java:662)
> Thread 13 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@d5d4de6):
> State: TIMED_WAITING
> Blocked count: 2
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2689)
> java.lang.Thread.run(Thread.java:662)
> Thread 12 (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@73d32e45):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 220
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:368)
> java.lang.Thread.run(Thread.java:662)
> Thread 11 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@19be4777):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 89
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2666)
> java.lang.Thread.run(Thread.java:662)
> Thread 10 (org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@658f7386):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 2
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
> java.lang.Thread.run(Thread.java:662)
> Thread 4 (Signal Dispatcher):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> Thread 3 (Finalizer):
> State: WAITING
> Blocked count: 9
> Waited count: 10
> Waiting on java.lang.ref.ReferenceQueue$Lock@3d1a70a7
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> Thread 2 (Reference Handler):
> State: WAITING
> Blocked count: 11
> Waited count: 12
> Waiting on java.lang.ref.Reference$Lock@270d75a3
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> Thread 1 (main):
> State: RUNNABLE
> Blocked count: 104
> Waited count: 97
> Stack:
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> java.lang.reflect.Method.invoke(Method.java:597)
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
> org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4724) TestAdmin hangs randomly in trunk
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142299#comment-13142299 ]
stack commented on HBASE-4724:
------------------------------
@N Trying to reproduce but testing the NPE fix at same time.... its not failing for me now.... I'll let it run longer.
> TestAdmin hangs randomly in trunk
> ---------------------------------
>
> Key: HBASE-4724
> URL: https://issues.apache.org/jira/browse/HBASE-4724
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 20111102_4724_TestAdmin.patch, 20111102_4724_TestAdmin.v2.patch
>
>
> fom the logs in my env
> {noformat}
> 2011-11-01 15:48:40,744 WARN [Master:0;localhost,39664,1320187706355] master.AssignmentManager(1471): Failed assignment of -ROOT-,,0.70236052 to localhost,44046,1320187706849, trying to assign elsewhere instead; retry=1
> org.apache.hadoop.hbase.ipc.HBaseRPC$VersionMismatch: Protocol org.apache.hadoop.hbase.ipc.HRegionInterface version mismatch. (client = 28, server = 29)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:185)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:300)
> {noformat}
> Anyway, after this the logs finishes with:
> {noformat}
> 2011-11-01 15:54:35,132 INFO [Master:0;localhost,39664,1320187706355.oldLogCleaner] hbase.Chore(80): Master:0;localhost,39664,1320187706355.oldLogCleaner exiting
> Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on Master:0;localhost,39664,1320187706355
> {noformat}
> it's in
> {noformat}
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> {noformat}
> So that's at least why adding a timeout wont help and may be why it does not end at all. Adding a maximum retry to Threads#threadDumpingIsAlive could help.
> I also wonder if the root cause of the non ending is my modif on the wal, with some threads surprised to have updates that were not written in the wal. Here is the full stack dump:
> {noformat}
> Thread 354 (IPC Client (47) connection to localhost/127.0.0.1:52227 from nkeywal):
> State: TIMED_WAITING
> Blocked count: 360
> Waited count: 359
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:702)
> org.apache.hadoop.ipc.Client$Connection.run(Client.java:744)
> Thread 272 (Master:0;localhost,39664,1320187706355-EventThread):
> State: WAITING
> Blocked count: 0
> Waited count: 4
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@107b954b
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 271 (Master:0;localhost,39664,1320187706355-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 152 (Master:0;localhost,39664,1320187706355):
> State: WAITING
> Blocked count: 217
> Waited count: 174
> Waiting on org.apache.hadoop.hbase.zookeeper.RootRegionTracker@6621477c
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:131)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:104)
> org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRoot(CatalogTracker.java:277)
> org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:523)
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:468)
> org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)
> java.lang.Thread.run(Thread.java:662)
> Thread 165 (LruBlockCache.EvictionThread):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread@3e9d7b56
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:593)
> java.lang.Thread.run(Thread.java:662)
> Thread 151 (main-EventThread):
> State: WAITING
> Blocked count: 40
> Waited count: 39
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22531380
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 150 (main-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 55
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 149 (IPC Server handler 4 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 148 (IPC Server handler 3 on 39664):
> State: WAITING
> Blocked count: 7
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 147 (IPC Server handler 2 on 39664):
> State: WAITING
> Blocked count: 11
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 146 (IPC Server handler 1 on 39664):
> State: WAITING
> Blocked count: 6
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 145 (IPC Server handler 0 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1970
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 133 (IPC Server listener on 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:580)
> Thread 144 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:754)
> Thread 143 (IPC Reader 9 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 142 (IPC Reader 8 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 141 (IPC Reader 7 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 140 (IPC Reader 6 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 139 (IPC Reader 5 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 138 (IPC Reader 4 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 137 (IPC Reader 3 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 136 (IPC Reader 2 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 135 (IPC Reader 1 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 134 (IPC Reader 0 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 124 (LeaseChecker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1252)
> java.lang.Thread.run(Thread.java:662)
> Thread 121 (ProcessThread:-1):
> State: WAITING
> Blocked count: 0
> Waited count: 269
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f5d84e0
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:103)
> Thread 120 (SyncThread:0):
> State: WAITING
> Blocked count: 2
> Waited count: 252
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b25b27c
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:94)
> Thread 119 (SessionTracker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 207
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:142)
> Thread 118 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:21819):
> State: RUNNABLE
> Blocked count: 38
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:232)
> Thread 116 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@2e6f947b):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 113 (IPC Server handler 2 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 112 (IPC Server handler 1 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 111 (IPC Server handler 0 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 106 (IPC Server listener on 39899):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 108 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 102 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@45d1c3cd):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 109 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6]):
> State: TIMED_WAITING
> Blocked count: 137
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 107 (pool-4-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 105 (Timer-3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 104 (1353906974@qtp-1197660496-1 - Acceptor0 SelectChannelConnector@localhost:59858):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 103 (1185716172@qtp-1197660496-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 101 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 98 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 90 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@605b28c9):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 86 (IPC Server handler 2 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 85 (IPC Server handler 1 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 84 (IPC Server handler 0 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 80 (IPC Server listener on 56098):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 82 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 76 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@2b68989e):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 83 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4]):
> State: TIMED_WAITING
> Blocked count: 149
> Waited count: 290
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 81 (pool-3-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 79 (Timer-2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 78 (348878159@qtp-164967086-1 - Acceptor0 SelectChannelConnector@localhost:36427):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 77 (1905729203@qtp-164967086-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 75 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 72 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 61 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@6709da93):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 60 (IPC Server handler 2 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 59 (IPC Server handler 1 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 58 (IPC Server handler 0 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 53 (IPC Server listener on 40388):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 55 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 49 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@48baa31b):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 57 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2]):
> State: TIMED_WAITING
> Blocked count: 148
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 54 (pool-2-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 52 (Timer-1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 51 (1031841975@qtp-1068922666-1 - Acceptor0 SelectChannelConnector@localhost:56535):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 50 (1191055321@qtp-1068922666-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 48 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 45 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 31 (IPC Server handler 9 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 30 (IPC Server handler 8 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 29 (IPC Server handler 7 on 52227):
> State: WAITING
> Blocked count: 2
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 28 (IPC Server handler 6 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 27 (IPC Server handler 5 on 52227):
> State: WAITING
> Blocked count: 4
> Waited count: 68
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 26 (IPC Server handler 4 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 25 (IPC Server handler 3 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 24 (IPC Server handler 2 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 23 (IPC Server handler 1 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 22 (IPC Server handler 0 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 15 (IPC Server listener on 52227):
> State: RUNNABLE
> Blocked count: 43
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 17 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 21 (Timer-0):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 17
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 20 (245022965@qtp-589734556-1 - Acceptor0 SelectChannelConnector@localhost:33079):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 19 (1821050251@qtp-589734556-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 16 (pool-1-thread-1):
> State: RUNNABLE
> Blocked count: 45
> Waited count: 44
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 14 (org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@620fa83):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65)
> java.lang.Thread.run(Thread.java:662)
> Thread 13 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@d5d4de6):
> State: TIMED_WAITING
> Blocked count: 2
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2689)
> java.lang.Thread.run(Thread.java:662)
> Thread 12 (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@73d32e45):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 220
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:368)
> java.lang.Thread.run(Thread.java:662)
> Thread 11 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@19be4777):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 89
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2666)
> java.lang.Thread.run(Thread.java:662)
> Thread 10 (org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@658f7386):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 2
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
> java.lang.Thread.run(Thread.java:662)
> Thread 4 (Signal Dispatcher):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> Thread 3 (Finalizer):
> State: WAITING
> Blocked count: 9
> Waited count: 10
> Waiting on java.lang.ref.ReferenceQueue$Lock@3d1a70a7
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> Thread 2 (Reference Handler):
> State: WAITING
> Blocked count: 11
> Waited count: 12
> Waiting on java.lang.ref.Reference$Lock@270d75a3
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> Thread 1 (main):
> State: RUNNABLE
> Blocked count: 104
> Waited count: 97
> Stack:
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> java.lang.reflect.Method.invoke(Method.java:597)
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
> org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4724) TestAdmin hangs randomly in trunk
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13141971#comment-13141971 ]
nkeywal commented on HBASE-4724:
--------------------------------
I reproduce the issue 100% of the time on trunk, with TestAdmin#testCreateBadTables. I was not reproducing the error at all previously.
> TestAdmin hangs randomly in trunk
> ---------------------------------
>
> Key: HBASE-4724
> URL: https://issues.apache.org/jira/browse/HBASE-4724
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Priority: Critical
>
> fom the logs in my env
> {noformat}
> 2011-11-01 15:48:40,744 WARN [Master:0;localhost,39664,1320187706355] master.AssignmentManager(1471): Failed assignment of -ROOT-,,0.70236052 to localhost,44046,1320187706849, trying to assign elsewhere instead; retry=1
> org.apache.hadoop.hbase.ipc.HBaseRPC$VersionMismatch: Protocol org.apache.hadoop.hbase.ipc.HRegionInterface version mismatch. (client = 28, server = 29)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:185)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:300)
> {noformat}
> Anyway, after this the logs finishes with:
> {noformat}
> 2011-11-01 15:54:35,132 INFO [Master:0;localhost,39664,1320187706355.oldLogCleaner] hbase.Chore(80): Master:0;localhost,39664,1320187706355.oldLogCleaner exiting
> Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on Master:0;localhost,39664,1320187706355
> {noformat}
> it's in
> {noformat}
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> {noformat}
> So that's at least why adding a timeout wont help and may be why it does not end at all. Adding a maximum retry to Threads#threadDumpingIsAlive could help.
> I also wonder if the root cause of the non ending is my modif on the wal, with some threads surprised to have updates that were not written in the wal. Here is the full stack dump:
> {noformat}
> Thread 354 (IPC Client (47) connection to localhost/127.0.0.1:52227 from nkeywal):
> State: TIMED_WAITING
> Blocked count: 360
> Waited count: 359
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:702)
> org.apache.hadoop.ipc.Client$Connection.run(Client.java:744)
> Thread 272 (Master:0;localhost,39664,1320187706355-EventThread):
> State: WAITING
> Blocked count: 0
> Waited count: 4
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@107b954b
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 271 (Master:0;localhost,39664,1320187706355-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 152 (Master:0;localhost,39664,1320187706355):
> State: WAITING
> Blocked count: 217
> Waited count: 174
> Waiting on org.apache.hadoop.hbase.zookeeper.RootRegionTracker@6621477c
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:131)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:104)
> org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRoot(CatalogTracker.java:277)
> org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:523)
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:468)
> org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)
> java.lang.Thread.run(Thread.java:662)
> Thread 165 (LruBlockCache.EvictionThread):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread@3e9d7b56
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:593)
> java.lang.Thread.run(Thread.java:662)
> Thread 151 (main-EventThread):
> State: WAITING
> Blocked count: 40
> Waited count: 39
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22531380
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 150 (main-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 55
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 149 (IPC Server handler 4 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 148 (IPC Server handler 3 on 39664):
> State: WAITING
> Blocked count: 7
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 147 (IPC Server handler 2 on 39664):
> State: WAITING
> Blocked count: 11
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 146 (IPC Server handler 1 on 39664):
> State: WAITING
> Blocked count: 6
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 145 (IPC Server handler 0 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1970
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 133 (IPC Server listener on 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:580)
> Thread 144 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:754)
> Thread 143 (IPC Reader 9 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 142 (IPC Reader 8 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 141 (IPC Reader 7 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 140 (IPC Reader 6 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 139 (IPC Reader 5 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 138 (IPC Reader 4 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 137 (IPC Reader 3 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 136 (IPC Reader 2 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 135 (IPC Reader 1 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 134 (IPC Reader 0 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 124 (LeaseChecker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1252)
> java.lang.Thread.run(Thread.java:662)
> Thread 121 (ProcessThread:-1):
> State: WAITING
> Blocked count: 0
> Waited count: 269
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f5d84e0
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:103)
> Thread 120 (SyncThread:0):
> State: WAITING
> Blocked count: 2
> Waited count: 252
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b25b27c
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:94)
> Thread 119 (SessionTracker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 207
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:142)
> Thread 118 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:21819):
> State: RUNNABLE
> Blocked count: 38
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:232)
> Thread 116 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@2e6f947b):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 113 (IPC Server handler 2 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 112 (IPC Server handler 1 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 111 (IPC Server handler 0 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 106 (IPC Server listener on 39899):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 108 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 102 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@45d1c3cd):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 109 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6]):
> State: TIMED_WAITING
> Blocked count: 137
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 107 (pool-4-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 105 (Timer-3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 104 (1353906974@qtp-1197660496-1 - Acceptor0 SelectChannelConnector@localhost:59858):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 103 (1185716172@qtp-1197660496-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 101 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 98 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 90 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@605b28c9):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 86 (IPC Server handler 2 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 85 (IPC Server handler 1 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 84 (IPC Server handler 0 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 80 (IPC Server listener on 56098):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 82 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 76 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@2b68989e):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 83 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4]):
> State: TIMED_WAITING
> Blocked count: 149
> Waited count: 290
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 81 (pool-3-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 79 (Timer-2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 78 (348878159@qtp-164967086-1 - Acceptor0 SelectChannelConnector@localhost:36427):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 77 (1905729203@qtp-164967086-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 75 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 72 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 61 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@6709da93):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 60 (IPC Server handler 2 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 59 (IPC Server handler 1 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 58 (IPC Server handler 0 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 53 (IPC Server listener on 40388):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 55 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 49 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@48baa31b):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 57 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2]):
> State: TIMED_WAITING
> Blocked count: 148
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 54 (pool-2-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 52 (Timer-1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 51 (1031841975@qtp-1068922666-1 - Acceptor0 SelectChannelConnector@localhost:56535):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 50 (1191055321@qtp-1068922666-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 48 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 45 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 31 (IPC Server handler 9 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 30 (IPC Server handler 8 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 29 (IPC Server handler 7 on 52227):
> State: WAITING
> Blocked count: 2
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 28 (IPC Server handler 6 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 27 (IPC Server handler 5 on 52227):
> State: WAITING
> Blocked count: 4
> Waited count: 68
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 26 (IPC Server handler 4 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 25 (IPC Server handler 3 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 24 (IPC Server handler 2 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 23 (IPC Server handler 1 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 22 (IPC Server handler 0 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 15 (IPC Server listener on 52227):
> State: RUNNABLE
> Blocked count: 43
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 17 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 21 (Timer-0):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 17
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 20 (245022965@qtp-589734556-1 - Acceptor0 SelectChannelConnector@localhost:33079):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 19 (1821050251@qtp-589734556-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 16 (pool-1-thread-1):
> State: RUNNABLE
> Blocked count: 45
> Waited count: 44
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 14 (org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@620fa83):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65)
> java.lang.Thread.run(Thread.java:662)
> Thread 13 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@d5d4de6):
> State: TIMED_WAITING
> Blocked count: 2
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2689)
> java.lang.Thread.run(Thread.java:662)
> Thread 12 (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@73d32e45):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 220
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:368)
> java.lang.Thread.run(Thread.java:662)
> Thread 11 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@19be4777):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 89
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2666)
> java.lang.Thread.run(Thread.java:662)
> Thread 10 (org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@658f7386):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 2
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
> java.lang.Thread.run(Thread.java:662)
> Thread 4 (Signal Dispatcher):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> Thread 3 (Finalizer):
> State: WAITING
> Blocked count: 9
> Waited count: 10
> Waiting on java.lang.ref.ReferenceQueue$Lock@3d1a70a7
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> Thread 2 (Reference Handler):
> State: WAITING
> Blocked count: 11
> Waited count: 12
> Waiting on java.lang.ref.Reference$Lock@270d75a3
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> Thread 1 (main):
> State: RUNNABLE
> Blocked count: 104
> Waited count: 97
> Stack:
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> java.lang.reflect.Method.invoke(Method.java:597)
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
> org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4724) TestAdmin hangs randomly in trunk
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142320#comment-13142320 ]
stack commented on HBASE-4724:
------------------------------
Ok. Want to make new issue to fix the leak?
It just failed for me with this:
{code}
Tests in error:
testHundredsOfTable(org.apache.hadoop.hbase.client.TestAdmin): Call to sv4r11s38/10.4.11.38:46152 failed on socket timeout exception: java.net.SocketTimeoutException: 1500 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.4.11.38:52051 remote=sv4r11s38/10.4.11.38:46152]
{code}
Is 1500ms not enough? Did you change that?
On the message:
{code}
2011-11-02 03:36:17,272 DEBUG [Finalizer] client.HConnectionManager$HConnectionImplementation(1715): The connection to null has been closed.
{code}
... yeah, it looks like this test is making lots of instances of HBaseAdmin... ones it should be cleaning up but it does look like the message is harmless... a close of a closed connection. I made HBASE-4728 for this.
> TestAdmin hangs randomly in trunk
> ---------------------------------
>
> Key: HBASE-4724
> URL: https://issues.apache.org/jira/browse/HBASE-4724
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 20111102_4724_TestAdmin.patch, 20111102_4724_TestAdmin.v2.patch
>
>
> fom the logs in my env
> {noformat}
> 2011-11-01 15:48:40,744 WARN [Master:0;localhost,39664,1320187706355] master.AssignmentManager(1471): Failed assignment of -ROOT-,,0.70236052 to localhost,44046,1320187706849, trying to assign elsewhere instead; retry=1
> org.apache.hadoop.hbase.ipc.HBaseRPC$VersionMismatch: Protocol org.apache.hadoop.hbase.ipc.HRegionInterface version mismatch. (client = 28, server = 29)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:185)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:300)
> {noformat}
> Anyway, after this the logs finishes with:
> {noformat}
> 2011-11-01 15:54:35,132 INFO [Master:0;localhost,39664,1320187706355.oldLogCleaner] hbase.Chore(80): Master:0;localhost,39664,1320187706355.oldLogCleaner exiting
> Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on Master:0;localhost,39664,1320187706355
> {noformat}
> it's in
> {noformat}
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> {noformat}
> So that's at least why adding a timeout wont help and may be why it does not end at all. Adding a maximum retry to Threads#threadDumpingIsAlive could help.
> I also wonder if the root cause of the non ending is my modif on the wal, with some threads surprised to have updates that were not written in the wal. Here is the full stack dump:
> {noformat}
> Thread 354 (IPC Client (47) connection to localhost/127.0.0.1:52227 from nkeywal):
> State: TIMED_WAITING
> Blocked count: 360
> Waited count: 359
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:702)
> org.apache.hadoop.ipc.Client$Connection.run(Client.java:744)
> Thread 272 (Master:0;localhost,39664,1320187706355-EventThread):
> State: WAITING
> Blocked count: 0
> Waited count: 4
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@107b954b
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 271 (Master:0;localhost,39664,1320187706355-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 152 (Master:0;localhost,39664,1320187706355):
> State: WAITING
> Blocked count: 217
> Waited count: 174
> Waiting on org.apache.hadoop.hbase.zookeeper.RootRegionTracker@6621477c
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:131)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:104)
> org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRoot(CatalogTracker.java:277)
> org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:523)
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:468)
> org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)
> java.lang.Thread.run(Thread.java:662)
> Thread 165 (LruBlockCache.EvictionThread):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread@3e9d7b56
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:593)
> java.lang.Thread.run(Thread.java:662)
> Thread 151 (main-EventThread):
> State: WAITING
> Blocked count: 40
> Waited count: 39
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22531380
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 150 (main-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 55
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 149 (IPC Server handler 4 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 148 (IPC Server handler 3 on 39664):
> State: WAITING
> Blocked count: 7
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 147 (IPC Server handler 2 on 39664):
> State: WAITING
> Blocked count: 11
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 146 (IPC Server handler 1 on 39664):
> State: WAITING
> Blocked count: 6
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 145 (IPC Server handler 0 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1970
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 133 (IPC Server listener on 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:580)
> Thread 144 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:754)
> Thread 143 (IPC Reader 9 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 142 (IPC Reader 8 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 141 (IPC Reader 7 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 140 (IPC Reader 6 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 139 (IPC Reader 5 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 138 (IPC Reader 4 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 137 (IPC Reader 3 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 136 (IPC Reader 2 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 135 (IPC Reader 1 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 134 (IPC Reader 0 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 124 (LeaseChecker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1252)
> java.lang.Thread.run(Thread.java:662)
> Thread 121 (ProcessThread:-1):
> State: WAITING
> Blocked count: 0
> Waited count: 269
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f5d84e0
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:103)
> Thread 120 (SyncThread:0):
> State: WAITING
> Blocked count: 2
> Waited count: 252
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b25b27c
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:94)
> Thread 119 (SessionTracker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 207
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:142)
> Thread 118 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:21819):
> State: RUNNABLE
> Blocked count: 38
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:232)
> Thread 116 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@2e6f947b):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 113 (IPC Server handler 2 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 112 (IPC Server handler 1 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 111 (IPC Server handler 0 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 106 (IPC Server listener on 39899):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 108 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 102 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@45d1c3cd):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 109 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6]):
> State: TIMED_WAITING
> Blocked count: 137
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 107 (pool-4-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 105 (Timer-3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 104 (1353906974@qtp-1197660496-1 - Acceptor0 SelectChannelConnector@localhost:59858):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 103 (1185716172@qtp-1197660496-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 101 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 98 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 90 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@605b28c9):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 86 (IPC Server handler 2 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 85 (IPC Server handler 1 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 84 (IPC Server handler 0 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 80 (IPC Server listener on 56098):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 82 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 76 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@2b68989e):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 83 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4]):
> State: TIMED_WAITING
> Blocked count: 149
> Waited count: 290
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 81 (pool-3-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 79 (Timer-2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 78 (348878159@qtp-164967086-1 - Acceptor0 SelectChannelConnector@localhost:36427):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 77 (1905729203@qtp-164967086-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 75 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 72 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 61 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@6709da93):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 60 (IPC Server handler 2 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 59 (IPC Server handler 1 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 58 (IPC Server handler 0 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 53 (IPC Server listener on 40388):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 55 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 49 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@48baa31b):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 57 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2]):
> State: TIMED_WAITING
> Blocked count: 148
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 54 (pool-2-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 52 (Timer-1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 51 (1031841975@qtp-1068922666-1 - Acceptor0 SelectChannelConnector@localhost:56535):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 50 (1191055321@qtp-1068922666-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 48 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 45 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 31 (IPC Server handler 9 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 30 (IPC Server handler 8 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 29 (IPC Server handler 7 on 52227):
> State: WAITING
> Blocked count: 2
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 28 (IPC Server handler 6 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 27 (IPC Server handler 5 on 52227):
> State: WAITING
> Blocked count: 4
> Waited count: 68
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 26 (IPC Server handler 4 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 25 (IPC Server handler 3 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 24 (IPC Server handler 2 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 23 (IPC Server handler 1 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 22 (IPC Server handler 0 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 15 (IPC Server listener on 52227):
> State: RUNNABLE
> Blocked count: 43
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 17 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 21 (Timer-0):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 17
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 20 (245022965@qtp-589734556-1 - Acceptor0 SelectChannelConnector@localhost:33079):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 19 (1821050251@qtp-589734556-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 16 (pool-1-thread-1):
> State: RUNNABLE
> Blocked count: 45
> Waited count: 44
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 14 (org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@620fa83):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65)
> java.lang.Thread.run(Thread.java:662)
> Thread 13 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@d5d4de6):
> State: TIMED_WAITING
> Blocked count: 2
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2689)
> java.lang.Thread.run(Thread.java:662)
> Thread 12 (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@73d32e45):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 220
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:368)
> java.lang.Thread.run(Thread.java:662)
> Thread 11 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@19be4777):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 89
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2666)
> java.lang.Thread.run(Thread.java:662)
> Thread 10 (org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@658f7386):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 2
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
> java.lang.Thread.run(Thread.java:662)
> Thread 4 (Signal Dispatcher):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> Thread 3 (Finalizer):
> State: WAITING
> Blocked count: 9
> Waited count: 10
> Waiting on java.lang.ref.ReferenceQueue$Lock@3d1a70a7
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> Thread 2 (Reference Handler):
> State: WAITING
> Blocked count: 11
> Waited count: 12
> Waiting on java.lang.ref.Reference$Lock@270d75a3
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> Thread 1 (main):
> State: RUNNABLE
> Blocked count: 104
> Waited count: 97
> Stack:
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> java.lang.reflect.Method.invoke(Method.java:597)
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
> org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4724) TestAdmin hangs randomly in trunk
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4724:
---------------------------
Assignee: nkeywal
Status: Patch Available (was: Open)
> TestAdmin hangs randomly in trunk
> ---------------------------------
>
> Key: HBASE-4724
> URL: https://issues.apache.org/jira/browse/HBASE-4724
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Critical
> Attachments: 20111102_4724_TestAdmin.patch
>
>
> fom the logs in my env
> {noformat}
> 2011-11-01 15:48:40,744 WARN [Master:0;localhost,39664,1320187706355] master.AssignmentManager(1471): Failed assignment of -ROOT-,,0.70236052 to localhost,44046,1320187706849, trying to assign elsewhere instead; retry=1
> org.apache.hadoop.hbase.ipc.HBaseRPC$VersionMismatch: Protocol org.apache.hadoop.hbase.ipc.HRegionInterface version mismatch. (client = 28, server = 29)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:185)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:300)
> {noformat}
> Anyway, after this the logs finishes with:
> {noformat}
> 2011-11-01 15:54:35,132 INFO [Master:0;localhost,39664,1320187706355.oldLogCleaner] hbase.Chore(80): Master:0;localhost,39664,1320187706355.oldLogCleaner exiting
> Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on Master:0;localhost,39664,1320187706355
> {noformat}
> it's in
> {noformat}
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> {noformat}
> So that's at least why adding a timeout wont help and may be why it does not end at all. Adding a maximum retry to Threads#threadDumpingIsAlive could help.
> I also wonder if the root cause of the non ending is my modif on the wal, with some threads surprised to have updates that were not written in the wal. Here is the full stack dump:
> {noformat}
> Thread 354 (IPC Client (47) connection to localhost/127.0.0.1:52227 from nkeywal):
> State: TIMED_WAITING
> Blocked count: 360
> Waited count: 359
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:702)
> org.apache.hadoop.ipc.Client$Connection.run(Client.java:744)
> Thread 272 (Master:0;localhost,39664,1320187706355-EventThread):
> State: WAITING
> Blocked count: 0
> Waited count: 4
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@107b954b
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 271 (Master:0;localhost,39664,1320187706355-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 152 (Master:0;localhost,39664,1320187706355):
> State: WAITING
> Blocked count: 217
> Waited count: 174
> Waiting on org.apache.hadoop.hbase.zookeeper.RootRegionTracker@6621477c
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:131)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:104)
> org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRoot(CatalogTracker.java:277)
> org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:523)
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:468)
> org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)
> java.lang.Thread.run(Thread.java:662)
> Thread 165 (LruBlockCache.EvictionThread):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread@3e9d7b56
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:593)
> java.lang.Thread.run(Thread.java:662)
> Thread 151 (main-EventThread):
> State: WAITING
> Blocked count: 40
> Waited count: 39
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22531380
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 150 (main-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 55
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 149 (IPC Server handler 4 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 148 (IPC Server handler 3 on 39664):
> State: WAITING
> Blocked count: 7
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 147 (IPC Server handler 2 on 39664):
> State: WAITING
> Blocked count: 11
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 146 (IPC Server handler 1 on 39664):
> State: WAITING
> Blocked count: 6
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 145 (IPC Server handler 0 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1970
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 133 (IPC Server listener on 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:580)
> Thread 144 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:754)
> Thread 143 (IPC Reader 9 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 142 (IPC Reader 8 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 141 (IPC Reader 7 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 140 (IPC Reader 6 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 139 (IPC Reader 5 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 138 (IPC Reader 4 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 137 (IPC Reader 3 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 136 (IPC Reader 2 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 135 (IPC Reader 1 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 134 (IPC Reader 0 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 124 (LeaseChecker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1252)
> java.lang.Thread.run(Thread.java:662)
> Thread 121 (ProcessThread:-1):
> State: WAITING
> Blocked count: 0
> Waited count: 269
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f5d84e0
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:103)
> Thread 120 (SyncThread:0):
> State: WAITING
> Blocked count: 2
> Waited count: 252
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b25b27c
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:94)
> Thread 119 (SessionTracker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 207
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:142)
> Thread 118 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:21819):
> State: RUNNABLE
> Blocked count: 38
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:232)
> Thread 116 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@2e6f947b):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 113 (IPC Server handler 2 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 112 (IPC Server handler 1 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 111 (IPC Server handler 0 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 106 (IPC Server listener on 39899):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 108 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 102 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@45d1c3cd):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 109 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6]):
> State: TIMED_WAITING
> Blocked count: 137
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 107 (pool-4-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 105 (Timer-3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 104 (1353906974@qtp-1197660496-1 - Acceptor0 SelectChannelConnector@localhost:59858):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 103 (1185716172@qtp-1197660496-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 101 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 98 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 90 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@605b28c9):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 86 (IPC Server handler 2 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 85 (IPC Server handler 1 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 84 (IPC Server handler 0 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 80 (IPC Server listener on 56098):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 82 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 76 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@2b68989e):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 83 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4]):
> State: TIMED_WAITING
> Blocked count: 149
> Waited count: 290
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 81 (pool-3-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 79 (Timer-2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 78 (348878159@qtp-164967086-1 - Acceptor0 SelectChannelConnector@localhost:36427):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 77 (1905729203@qtp-164967086-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 75 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 72 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 61 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@6709da93):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 60 (IPC Server handler 2 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 59 (IPC Server handler 1 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 58 (IPC Server handler 0 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 53 (IPC Server listener on 40388):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 55 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 49 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@48baa31b):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 57 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2]):
> State: TIMED_WAITING
> Blocked count: 148
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 54 (pool-2-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 52 (Timer-1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 51 (1031841975@qtp-1068922666-1 - Acceptor0 SelectChannelConnector@localhost:56535):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 50 (1191055321@qtp-1068922666-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 48 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 45 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 31 (IPC Server handler 9 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 30 (IPC Server handler 8 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 29 (IPC Server handler 7 on 52227):
> State: WAITING
> Blocked count: 2
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 28 (IPC Server handler 6 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 27 (IPC Server handler 5 on 52227):
> State: WAITING
> Blocked count: 4
> Waited count: 68
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 26 (IPC Server handler 4 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 25 (IPC Server handler 3 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 24 (IPC Server handler 2 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 23 (IPC Server handler 1 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 22 (IPC Server handler 0 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 15 (IPC Server listener on 52227):
> State: RUNNABLE
> Blocked count: 43
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 17 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 21 (Timer-0):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 17
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 20 (245022965@qtp-589734556-1 - Acceptor0 SelectChannelConnector@localhost:33079):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 19 (1821050251@qtp-589734556-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 16 (pool-1-thread-1):
> State: RUNNABLE
> Blocked count: 45
> Waited count: 44
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 14 (org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@620fa83):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65)
> java.lang.Thread.run(Thread.java:662)
> Thread 13 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@d5d4de6):
> State: TIMED_WAITING
> Blocked count: 2
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2689)
> java.lang.Thread.run(Thread.java:662)
> Thread 12 (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@73d32e45):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 220
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:368)
> java.lang.Thread.run(Thread.java:662)
> Thread 11 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@19be4777):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 89
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2666)
> java.lang.Thread.run(Thread.java:662)
> Thread 10 (org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@658f7386):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 2
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
> java.lang.Thread.run(Thread.java:662)
> Thread 4 (Signal Dispatcher):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> Thread 3 (Finalizer):
> State: WAITING
> Blocked count: 9
> Waited count: 10
> Waiting on java.lang.ref.ReferenceQueue$Lock@3d1a70a7
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> Thread 2 (Reference Handler):
> State: WAITING
> Blocked count: 11
> Waited count: 12
> Waiting on java.lang.ref.Reference$Lock@270d75a3
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> Thread 1 (main):
> State: RUNNABLE
> Blocked count: 104
> Waited count: 97
> Stack:
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> java.lang.reflect.Method.invoke(Method.java:597)
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
> org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4724) TestAdmin hangs randomly in trunk
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142419#comment-13142419 ]
Hudson commented on HBASE-4724:
-------------------------------
Integrated in HBase-TRUNK #2401 (See [https://builds.apache.org/job/HBase-TRUNK/2401/])
HBASE-4724 TestAdmin hangs randomly in trunk
stack :
Files :
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java
> TestAdmin hangs randomly in trunk
> ---------------------------------
>
> Key: HBASE-4724
> URL: https://issues.apache.org/jira/browse/HBASE-4724
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 20111102_4724_TestAdmin.patch, 20111102_4724_TestAdmin.v2.patch
>
>
> fom the logs in my env
> {noformat}
> 2011-11-01 15:48:40,744 WARN [Master:0;localhost,39664,1320187706355] master.AssignmentManager(1471): Failed assignment of -ROOT-,,0.70236052 to localhost,44046,1320187706849, trying to assign elsewhere instead; retry=1
> org.apache.hadoop.hbase.ipc.HBaseRPC$VersionMismatch: Protocol org.apache.hadoop.hbase.ipc.HRegionInterface version mismatch. (client = 28, server = 29)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:185)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:300)
> {noformat}
> Anyway, after this the logs finishes with:
> {noformat}
> 2011-11-01 15:54:35,132 INFO [Master:0;localhost,39664,1320187706355.oldLogCleaner] hbase.Chore(80): Master:0;localhost,39664,1320187706355.oldLogCleaner exiting
> Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on Master:0;localhost,39664,1320187706355
> {noformat}
> it's in
> {noformat}
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> {noformat}
> So that's at least why adding a timeout wont help and may be why it does not end at all. Adding a maximum retry to Threads#threadDumpingIsAlive could help.
> I also wonder if the root cause of the non ending is my modif on the wal, with some threads surprised to have updates that were not written in the wal. Here is the full stack dump:
> {noformat}
> Thread 354 (IPC Client (47) connection to localhost/127.0.0.1:52227 from nkeywal):
> State: TIMED_WAITING
> Blocked count: 360
> Waited count: 359
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:702)
> org.apache.hadoop.ipc.Client$Connection.run(Client.java:744)
> Thread 272 (Master:0;localhost,39664,1320187706355-EventThread):
> State: WAITING
> Blocked count: 0
> Waited count: 4
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@107b954b
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 271 (Master:0;localhost,39664,1320187706355-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 152 (Master:0;localhost,39664,1320187706355):
> State: WAITING
> Blocked count: 217
> Waited count: 174
> Waiting on org.apache.hadoop.hbase.zookeeper.RootRegionTracker@6621477c
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:131)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:104)
> org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRoot(CatalogTracker.java:277)
> org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:523)
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:468)
> org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)
> java.lang.Thread.run(Thread.java:662)
> Thread 165 (LruBlockCache.EvictionThread):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread@3e9d7b56
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:593)
> java.lang.Thread.run(Thread.java:662)
> Thread 151 (main-EventThread):
> State: WAITING
> Blocked count: 40
> Waited count: 39
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22531380
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 150 (main-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 55
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 149 (IPC Server handler 4 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 148 (IPC Server handler 3 on 39664):
> State: WAITING
> Blocked count: 7
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 147 (IPC Server handler 2 on 39664):
> State: WAITING
> Blocked count: 11
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 146 (IPC Server handler 1 on 39664):
> State: WAITING
> Blocked count: 6
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 145 (IPC Server handler 0 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1970
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 133 (IPC Server listener on 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:580)
> Thread 144 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:754)
> Thread 143 (IPC Reader 9 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 142 (IPC Reader 8 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 141 (IPC Reader 7 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 140 (IPC Reader 6 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 139 (IPC Reader 5 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 138 (IPC Reader 4 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 137 (IPC Reader 3 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 136 (IPC Reader 2 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 135 (IPC Reader 1 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 134 (IPC Reader 0 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 124 (LeaseChecker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1252)
> java.lang.Thread.run(Thread.java:662)
> Thread 121 (ProcessThread:-1):
> State: WAITING
> Blocked count: 0
> Waited count: 269
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f5d84e0
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:103)
> Thread 120 (SyncThread:0):
> State: WAITING
> Blocked count: 2
> Waited count: 252
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b25b27c
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:94)
> Thread 119 (SessionTracker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 207
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:142)
> Thread 118 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:21819):
> State: RUNNABLE
> Blocked count: 38
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:232)
> Thread 116 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@2e6f947b):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 113 (IPC Server handler 2 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 112 (IPC Server handler 1 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 111 (IPC Server handler 0 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 106 (IPC Server listener on 39899):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 108 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 102 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@45d1c3cd):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 109 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6]):
> State: TIMED_WAITING
> Blocked count: 137
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 107 (pool-4-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 105 (Timer-3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 104 (1353906974@qtp-1197660496-1 - Acceptor0 SelectChannelConnector@localhost:59858):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 103 (1185716172@qtp-1197660496-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 101 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 98 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 90 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@605b28c9):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 86 (IPC Server handler 2 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 85 (IPC Server handler 1 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 84 (IPC Server handler 0 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 80 (IPC Server listener on 56098):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 82 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 76 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@2b68989e):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 83 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4]):
> State: TIMED_WAITING
> Blocked count: 149
> Waited count: 290
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 81 (pool-3-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 79 (Timer-2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 78 (348878159@qtp-164967086-1 - Acceptor0 SelectChannelConnector@localhost:36427):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 77 (1905729203@qtp-164967086-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 75 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 72 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 61 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@6709da93):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 60 (IPC Server handler 2 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 59 (IPC Server handler 1 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 58 (IPC Server handler 0 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 53 (IPC Server listener on 40388):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 55 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 49 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@48baa31b):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 57 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2]):
> State: TIMED_WAITING
> Blocked count: 148
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 54 (pool-2-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 52 (Timer-1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 51 (1031841975@qtp-1068922666-1 - Acceptor0 SelectChannelConnector@localhost:56535):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 50 (1191055321@qtp-1068922666-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 48 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 45 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 31 (IPC Server handler 9 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 30 (IPC Server handler 8 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 29 (IPC Server handler 7 on 52227):
> State: WAITING
> Blocked count: 2
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 28 (IPC Server handler 6 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 27 (IPC Server handler 5 on 52227):
> State: WAITING
> Blocked count: 4
> Waited count: 68
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 26 (IPC Server handler 4 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 25 (IPC Server handler 3 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 24 (IPC Server handler 2 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 23 (IPC Server handler 1 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 22 (IPC Server handler 0 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 15 (IPC Server listener on 52227):
> State: RUNNABLE
> Blocked count: 43
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 17 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 21 (Timer-0):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 17
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 20 (245022965@qtp-589734556-1 - Acceptor0 SelectChannelConnector@localhost:33079):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 19 (1821050251@qtp-589734556-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 16 (pool-1-thread-1):
> State: RUNNABLE
> Blocked count: 45
> Waited count: 44
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 14 (org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@620fa83):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65)
> java.lang.Thread.run(Thread.java:662)
> Thread 13 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@d5d4de6):
> State: TIMED_WAITING
> Blocked count: 2
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2689)
> java.lang.Thread.run(Thread.java:662)
> Thread 12 (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@73d32e45):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 220
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:368)
> java.lang.Thread.run(Thread.java:662)
> Thread 11 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@19be4777):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 89
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2666)
> java.lang.Thread.run(Thread.java:662)
> Thread 10 (org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@658f7386):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 2
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
> java.lang.Thread.run(Thread.java:662)
> Thread 4 (Signal Dispatcher):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> Thread 3 (Finalizer):
> State: WAITING
> Blocked count: 9
> Waited count: 10
> Waiting on java.lang.ref.ReferenceQueue$Lock@3d1a70a7
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> Thread 2 (Reference Handler):
> State: WAITING
> Blocked count: 11
> Waited count: 12
> Waiting on java.lang.ref.Reference$Lock@270d75a3
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> Thread 1 (main):
> State: RUNNABLE
> Blocked count: 104
> Waited count: 97
> Stack:
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> java.lang.reflect.Method.invoke(Method.java:597)
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
> org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4724) TestAdmin hangs randomly in trunk
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4724:
---------------------------
Attachment: 20111102_4724_TestAdmin.v2.patch
> TestAdmin hangs randomly in trunk
> ---------------------------------
>
> Key: HBASE-4724
> URL: https://issues.apache.org/jira/browse/HBASE-4724
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Critical
> Attachments: 20111102_4724_TestAdmin.patch, 20111102_4724_TestAdmin.v2.patch
>
>
> fom the logs in my env
> {noformat}
> 2011-11-01 15:48:40,744 WARN [Master:0;localhost,39664,1320187706355] master.AssignmentManager(1471): Failed assignment of -ROOT-,,0.70236052 to localhost,44046,1320187706849, trying to assign elsewhere instead; retry=1
> org.apache.hadoop.hbase.ipc.HBaseRPC$VersionMismatch: Protocol org.apache.hadoop.hbase.ipc.HRegionInterface version mismatch. (client = 28, server = 29)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:185)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:300)
> {noformat}
> Anyway, after this the logs finishes with:
> {noformat}
> 2011-11-01 15:54:35,132 INFO [Master:0;localhost,39664,1320187706355.oldLogCleaner] hbase.Chore(80): Master:0;localhost,39664,1320187706355.oldLogCleaner exiting
> Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on Master:0;localhost,39664,1320187706355
> {noformat}
> it's in
> {noformat}
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> {noformat}
> So that's at least why adding a timeout wont help and may be why it does not end at all. Adding a maximum retry to Threads#threadDumpingIsAlive could help.
> I also wonder if the root cause of the non ending is my modif on the wal, with some threads surprised to have updates that were not written in the wal. Here is the full stack dump:
> {noformat}
> Thread 354 (IPC Client (47) connection to localhost/127.0.0.1:52227 from nkeywal):
> State: TIMED_WAITING
> Blocked count: 360
> Waited count: 359
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:702)
> org.apache.hadoop.ipc.Client$Connection.run(Client.java:744)
> Thread 272 (Master:0;localhost,39664,1320187706355-EventThread):
> State: WAITING
> Blocked count: 0
> Waited count: 4
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@107b954b
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 271 (Master:0;localhost,39664,1320187706355-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 152 (Master:0;localhost,39664,1320187706355):
> State: WAITING
> Blocked count: 217
> Waited count: 174
> Waiting on org.apache.hadoop.hbase.zookeeper.RootRegionTracker@6621477c
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:131)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:104)
> org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRoot(CatalogTracker.java:277)
> org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:523)
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:468)
> org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)
> java.lang.Thread.run(Thread.java:662)
> Thread 165 (LruBlockCache.EvictionThread):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread@3e9d7b56
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:593)
> java.lang.Thread.run(Thread.java:662)
> Thread 151 (main-EventThread):
> State: WAITING
> Blocked count: 40
> Waited count: 39
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22531380
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 150 (main-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 55
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 149 (IPC Server handler 4 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 148 (IPC Server handler 3 on 39664):
> State: WAITING
> Blocked count: 7
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 147 (IPC Server handler 2 on 39664):
> State: WAITING
> Blocked count: 11
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 146 (IPC Server handler 1 on 39664):
> State: WAITING
> Blocked count: 6
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 145 (IPC Server handler 0 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1970
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 133 (IPC Server listener on 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:580)
> Thread 144 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:754)
> Thread 143 (IPC Reader 9 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 142 (IPC Reader 8 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 141 (IPC Reader 7 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 140 (IPC Reader 6 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 139 (IPC Reader 5 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 138 (IPC Reader 4 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 137 (IPC Reader 3 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 136 (IPC Reader 2 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 135 (IPC Reader 1 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 134 (IPC Reader 0 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 124 (LeaseChecker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1252)
> java.lang.Thread.run(Thread.java:662)
> Thread 121 (ProcessThread:-1):
> State: WAITING
> Blocked count: 0
> Waited count: 269
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f5d84e0
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:103)
> Thread 120 (SyncThread:0):
> State: WAITING
> Blocked count: 2
> Waited count: 252
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b25b27c
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:94)
> Thread 119 (SessionTracker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 207
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:142)
> Thread 118 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:21819):
> State: RUNNABLE
> Blocked count: 38
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:232)
> Thread 116 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@2e6f947b):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 113 (IPC Server handler 2 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 112 (IPC Server handler 1 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 111 (IPC Server handler 0 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 106 (IPC Server listener on 39899):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 108 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 102 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@45d1c3cd):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 109 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6]):
> State: TIMED_WAITING
> Blocked count: 137
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 107 (pool-4-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 105 (Timer-3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 104 (1353906974@qtp-1197660496-1 - Acceptor0 SelectChannelConnector@localhost:59858):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 103 (1185716172@qtp-1197660496-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 101 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 98 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 90 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@605b28c9):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 86 (IPC Server handler 2 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 85 (IPC Server handler 1 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 84 (IPC Server handler 0 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 80 (IPC Server listener on 56098):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 82 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 76 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@2b68989e):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 83 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4]):
> State: TIMED_WAITING
> Blocked count: 149
> Waited count: 290
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 81 (pool-3-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 79 (Timer-2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 78 (348878159@qtp-164967086-1 - Acceptor0 SelectChannelConnector@localhost:36427):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 77 (1905729203@qtp-164967086-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 75 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 72 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 61 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@6709da93):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 60 (IPC Server handler 2 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 59 (IPC Server handler 1 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 58 (IPC Server handler 0 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 53 (IPC Server listener on 40388):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 55 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 49 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@48baa31b):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 57 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2]):
> State: TIMED_WAITING
> Blocked count: 148
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 54 (pool-2-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 52 (Timer-1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 51 (1031841975@qtp-1068922666-1 - Acceptor0 SelectChannelConnector@localhost:56535):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 50 (1191055321@qtp-1068922666-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 48 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 45 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 31 (IPC Server handler 9 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 30 (IPC Server handler 8 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 29 (IPC Server handler 7 on 52227):
> State: WAITING
> Blocked count: 2
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 28 (IPC Server handler 6 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 27 (IPC Server handler 5 on 52227):
> State: WAITING
> Blocked count: 4
> Waited count: 68
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 26 (IPC Server handler 4 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 25 (IPC Server handler 3 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 24 (IPC Server handler 2 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 23 (IPC Server handler 1 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 22 (IPC Server handler 0 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 15 (IPC Server listener on 52227):
> State: RUNNABLE
> Blocked count: 43
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 17 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 21 (Timer-0):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 17
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 20 (245022965@qtp-589734556-1 - Acceptor0 SelectChannelConnector@localhost:33079):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 19 (1821050251@qtp-589734556-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 16 (pool-1-thread-1):
> State: RUNNABLE
> Blocked count: 45
> Waited count: 44
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 14 (org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@620fa83):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65)
> java.lang.Thread.run(Thread.java:662)
> Thread 13 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@d5d4de6):
> State: TIMED_WAITING
> Blocked count: 2
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2689)
> java.lang.Thread.run(Thread.java:662)
> Thread 12 (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@73d32e45):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 220
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:368)
> java.lang.Thread.run(Thread.java:662)
> Thread 11 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@19be4777):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 89
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2666)
> java.lang.Thread.run(Thread.java:662)
> Thread 10 (org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@658f7386):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 2
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
> java.lang.Thread.run(Thread.java:662)
> Thread 4 (Signal Dispatcher):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> Thread 3 (Finalizer):
> State: WAITING
> Blocked count: 9
> Waited count: 10
> Waiting on java.lang.ref.ReferenceQueue$Lock@3d1a70a7
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> Thread 2 (Reference Handler):
> State: WAITING
> Blocked count: 11
> Waited count: 12
> Waiting on java.lang.ref.Reference$Lock@270d75a3
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> Thread 1 (main):
> State: RUNNABLE
> Blocked count: 104
> Waited count: 97
> Stack:
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> java.lang.reflect.Method.invoke(Method.java:597)
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
> org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4724) TestAdmin hangs randomly in trunk
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142229#comment-13142229 ]
stack commented on HBASE-4724:
------------------------------
@N I don't have version mismatch when I run TestAdmin. It doesn't fail for me either.
bq. Adding a maximum retry to Threads#threadDumpingIsAlive could help.
Yes. We should do this. No point in going on after we've thread dumped three times I'd say.
> TestAdmin hangs randomly in trunk
> ---------------------------------
>
> Key: HBASE-4724
> URL: https://issues.apache.org/jira/browse/HBASE-4724
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 20111102_4724_TestAdmin.patch, 20111102_4724_TestAdmin.v2.patch
>
>
> fom the logs in my env
> {noformat}
> 2011-11-01 15:48:40,744 WARN [Master:0;localhost,39664,1320187706355] master.AssignmentManager(1471): Failed assignment of -ROOT-,,0.70236052 to localhost,44046,1320187706849, trying to assign elsewhere instead; retry=1
> org.apache.hadoop.hbase.ipc.HBaseRPC$VersionMismatch: Protocol org.apache.hadoop.hbase.ipc.HRegionInterface version mismatch. (client = 28, server = 29)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:185)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:300)
> {noformat}
> Anyway, after this the logs finishes with:
> {noformat}
> 2011-11-01 15:54:35,132 INFO [Master:0;localhost,39664,1320187706355.oldLogCleaner] hbase.Chore(80): Master:0;localhost,39664,1320187706355.oldLogCleaner exiting
> Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on Master:0;localhost,39664,1320187706355
> {noformat}
> it's in
> {noformat}
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> {noformat}
> So that's at least why adding a timeout wont help and may be why it does not end at all. Adding a maximum retry to Threads#threadDumpingIsAlive could help.
> I also wonder if the root cause of the non ending is my modif on the wal, with some threads surprised to have updates that were not written in the wal. Here is the full stack dump:
> {noformat}
> Thread 354 (IPC Client (47) connection to localhost/127.0.0.1:52227 from nkeywal):
> State: TIMED_WAITING
> Blocked count: 360
> Waited count: 359
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:702)
> org.apache.hadoop.ipc.Client$Connection.run(Client.java:744)
> Thread 272 (Master:0;localhost,39664,1320187706355-EventThread):
> State: WAITING
> Blocked count: 0
> Waited count: 4
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@107b954b
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 271 (Master:0;localhost,39664,1320187706355-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 152 (Master:0;localhost,39664,1320187706355):
> State: WAITING
> Blocked count: 217
> Waited count: 174
> Waiting on org.apache.hadoop.hbase.zookeeper.RootRegionTracker@6621477c
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:131)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:104)
> org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRoot(CatalogTracker.java:277)
> org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:523)
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:468)
> org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)
> java.lang.Thread.run(Thread.java:662)
> Thread 165 (LruBlockCache.EvictionThread):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread@3e9d7b56
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:593)
> java.lang.Thread.run(Thread.java:662)
> Thread 151 (main-EventThread):
> State: WAITING
> Blocked count: 40
> Waited count: 39
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22531380
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 150 (main-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 55
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 149 (IPC Server handler 4 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 148 (IPC Server handler 3 on 39664):
> State: WAITING
> Blocked count: 7
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 147 (IPC Server handler 2 on 39664):
> State: WAITING
> Blocked count: 11
> Waited count: 1968
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 146 (IPC Server handler 1 on 39664):
> State: WAITING
> Blocked count: 6
> Waited count: 1969
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 145 (IPC Server handler 0 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1970
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 133 (IPC Server listener on 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:580)
> Thread 144 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:754)
> Thread 143 (IPC Reader 9 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 142 (IPC Reader 8 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 141 (IPC Reader 7 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 140 (IPC Reader 6 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 139 (IPC Reader 5 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 138 (IPC Reader 4 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 137 (IPC Reader 3 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 136 (IPC Reader 2 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 135 (IPC Reader 1 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 134 (IPC Reader 0 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 124 (LeaseChecker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1252)
> java.lang.Thread.run(Thread.java:662)
> Thread 121 (ProcessThread:-1):
> State: WAITING
> Blocked count: 0
> Waited count: 269
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f5d84e0
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:103)
> Thread 120 (SyncThread:0):
> State: WAITING
> Blocked count: 2
> Waited count: 252
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b25b27c
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:94)
> Thread 119 (SessionTracker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 207
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:142)
> Thread 118 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:21819):
> State: RUNNABLE
> Blocked count: 38
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:232)
> Thread 116 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@2e6f947b):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 113 (IPC Server handler 2 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 112 (IPC Server handler 1 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 111 (IPC Server handler 0 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 106 (IPC Server listener on 39899):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 108 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 102 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@45d1c3cd):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 109 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6]):
> State: TIMED_WAITING
> Blocked count: 137
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 107 (pool-4-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 105 (Timer-3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 104 (1353906974@qtp-1197660496-1 - Acceptor0 SelectChannelConnector@localhost:59858):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 103 (1185716172@qtp-1197660496-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 101 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 98 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 90 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@605b28c9):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 86 (IPC Server handler 2 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 85 (IPC Server handler 1 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 84 (IPC Server handler 0 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 80 (IPC Server listener on 56098):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 82 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 76 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@2b68989e):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 83 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4]):
> State: TIMED_WAITING
> Blocked count: 149
> Waited count: 290
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 81 (pool-3-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 79 (Timer-2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 78 (348878159@qtp-164967086-1 - Acceptor0 SelectChannelConnector@localhost:36427):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 77 (1905729203@qtp-164967086-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 75 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 72 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 61 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@6709da93):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 60 (IPC Server handler 2 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 59 (IPC Server handler 1 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 58 (IPC Server handler 0 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 53 (IPC Server listener on 40388):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 55 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 49 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@48baa31b):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 57 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2]):
> State: TIMED_WAITING
> Blocked count: 148
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 54 (pool-2-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 52 (Timer-1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 51 (1031841975@qtp-1068922666-1 - Acceptor0 SelectChannelConnector@localhost:56535):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 50 (1191055321@qtp-1068922666-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 48 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 45 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 31 (IPC Server handler 9 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 30 (IPC Server handler 8 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 29 (IPC Server handler 7 on 52227):
> State: WAITING
> Blocked count: 2
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 28 (IPC Server handler 6 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 27 (IPC Server handler 5 on 52227):
> State: WAITING
> Blocked count: 4
> Waited count: 68
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 26 (IPC Server handler 4 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 25 (IPC Server handler 3 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 24 (IPC Server handler 2 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 66
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 23 (IPC Server handler 1 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 22 (IPC Server handler 0 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 15 (IPC Server listener on 52227):
> State: RUNNABLE
> Blocked count: 43
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 17 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 21 (Timer-0):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 17
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 20 (245022965@qtp-589734556-1 - Acceptor0 SelectChannelConnector@localhost:33079):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 19 (1821050251@qtp-589734556-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 16 (pool-1-thread-1):
> State: RUNNABLE
> Blocked count: 45
> Waited count: 44
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 14 (org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@620fa83):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65)
> java.lang.Thread.run(Thread.java:662)
> Thread 13 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@d5d4de6):
> State: TIMED_WAITING
> Blocked count: 2
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2689)
> java.lang.Thread.run(Thread.java:662)
> Thread 12 (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@73d32e45):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 220
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:368)
> java.lang.Thread.run(Thread.java:662)
> Thread 11 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@19be4777):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 89
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2666)
> java.lang.Thread.run(Thread.java:662)
> Thread 10 (org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@658f7386):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 2
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
> java.lang.Thread.run(Thread.java:662)
> Thread 4 (Signal Dispatcher):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> Thread 3 (Finalizer):
> State: WAITING
> Blocked count: 9
> Waited count: 10
> Waiting on java.lang.ref.ReferenceQueue$Lock@3d1a70a7
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> Thread 2 (Reference Handler):
> State: WAITING
> Blocked count: 11
> Waited count: 12
> Waiting on java.lang.ref.Reference$Lock@270d75a3
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> Thread 1 (main):
> State: RUNNABLE
> Blocked count: 104
> Waited count: 97
> Stack:
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> java.lang.reflect.Method.invoke(Method.java:597)
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
> org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira