You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Geoff Hendrey <gh...@decarta.com> on 2011/09/11 02:50:57 UTC
scanner deadlock?
Hi all -
I'm still dealing with the saga of ScannerTimeoutException,
UnknownScannerException, etc. I rewrote my code, in the hope that simply
a different approach and some different code paths might yield better
results. No change. I tried many variations (caching 1 row vs caching
many rows, changing the regionserver's lease, increasing the number of
allowed zookeeper connections, etc). I created a fresh table on the
thought that maybe there was some problem with the table...no change.
I am dealing with what appears to be some sort of scanner deadlock. I
have a total order partitioned mapreduce job. In the reducer, as long as
I use just one reducer, the task finishes quickly. But as soon as more
than one reducer opens a scanner, the tasks procede in what I would call
a "jittery" lock step. They both are able to do ResultScanner.next() a
few times, but then the call to next() freezes for a long period, and
ScannerTimeoutException. I catch the exception, and get a new
ResultScanner, and the pattern repeats. Jittery lockstep consisting of
being able to get a few successful next() calls, then a lock up.
I have not yet tried upgrading from 90.1 to higher. Nor have I tried
tsuna's async client. Can anyone think of anything else I can try to
resolve this? I've sunk quite a few late nights into this, and would be
very excited to find a solution.
-geoff
Re: scanner deadlock?
Posted by Eric Charles <er...@gmail.com>.
Thx for info.
There are other paying tools for analysis, but the free should make it.
Eclipse MAT is great to see memory objects, but does not highlight the
deadlocks.
Btw, I was recently thought on livelocks
(http://en.wikipedia.org/wiki/Deadlock#Livelock). If you have such one,
no tool except your brain will find them :/
Thx.
On 12/09/11 10:42, Stack wrote:
> What Eric says.
>
> Sun java can find deadlocks too, the dumb ones at least.
>
> And then Todd likes this for the same job: http://www.jcarder.org/
>
> St.Ack
>
>
> On Mon, Sep 12, 2011 at 10:34 AM, Geoff Hendrey<gh...@decarta.com> wrote:
>> Wow, no I had not been using any tool other than eyeballs. I will
>> certainly apply this tool to our jstack.
>>
>> Thanks!!
>>
>> -geoff
>>
>> -----Original Message-----
>> From: Eric Charles [mailto:eric.umg.charles@gmail.com]
>> Sent: Monday, September 12, 2011 10:28 AM
>> To: user@hbase.apache.org
>> Subject: Re: scanner deadlock?
>>
>> Hi,
>>
>> Probably you've already done something similar, but I happily use 'IBM
>> Thread and Monitor Dump Analyzer for Java' to be 100% sure there are no
>> dead lock. The tool gives you a nice short report on your threads.
>>
>> I tried to run it on your dump, but the analyzer does not like
>> mail-wrapped lines.
>>
>> Thx.
>>
>> [1]
>> https://www.ibm.com/developerworks/mydeveloperworks/groups/service/html/
>> communityview?communityUuid=2245aa39-fa5c-4475-b891-14c205f7333c
>>
>> On 12/09/11 09:46, Stack wrote:
>>> On Sun, Sep 11, 2011 at 2:40 PM, Geoff Hendrey<gh...@decarta.com>
>> wrote:
>>>> Hi,
>>>>
>>>> The cluster is definitely loaded. We are running a MR job with 130
>>>> reducers, each using a scanner. Ganglia shows an overall CPU
>> utilization
>>>> of around 40%, with network pushing 800MB/s. The problem may
>> correlate
>>>> with a major_compact I initiated couple days ago. Even today, I still
>>>> see a lot of compacting (continuous) on region servers.
>>>>
>>>> Here is the thread dump from jstack. It does not reveal any explicit
>>>> deadlock. At the time I dumped the jstack, the region server logs are
>>>> reporting "IPC Server handler...ClosedChannelException". As soon as
>> this
>>>> begins occurring, our network throughput, as seen on Ganglia drops
>> from
>>>> 800MB/s down to virtually zero. Then of course there is the cascade
>> of
>>>> failures, eventually leading to the regionservers shutting down.
>>>>
>>>
>>>
>>> Looking at the thread dump, as you say, few are 'BLOCKED' (There are a
>>> few BLOCKED thread trying to go into a next). But most are deep in a
>>> next stuck polling looking to read. If you thread dump a few times
>>> in a row, do any seem to be making progress? (They may be making
>>> progress but slowly, as you say)
>>>
>>> So, is it possible you have a slow datanode in your cluster? Can you
>>> tell who all are connected too... lsof'ing? That might give you a
>>> clue. Perhaps there is a node with sick disks (check dmesg). Or
>>> perhaps the machine's nework card is sick?
>>>
>>> St.Ack
>>>
>>>> I don't see anything unusual in the datanode logs on the same server.
>>>> Just HDFS_READ and HDFS_WRITE for the most part.
>>>>
>>>> jstack here:
>>>> ===================================================================
>>>> 2011-09-11 14:28:52
>>>> Full thread dump OpenJDK 64-Bit Server VM (14.0-b16 mixed mode):
>>>>
>>>> "IPC Client (47) connection to
>>>> doop10.dt.sv4.decarta.com/10.241.8.230:54310 from hroot" daemon
>> prio=10
>>>> tid=0x0000000001492800 nid=0x75cb in Object.wait()
>> [0x0000000047633000]
>>>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> at
>>>> org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:676)
>>>> - locked<0x00002aaaaf680000> (a
>>>> org.apache.hadoop.ipc.Client$Connection)
>>>> at
>> org.apache.hadoop.ipc.Client$Connection.run(Client.java:719)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "ResponseProcessor for block blk_2613854253703670198_5032689" daemon
>>>> prio=10 tid=0x0000000001988800 nid=0x75bb runnable
>> [0x0000000046926000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>>> - locked<0x00002aab69337248> (a sun.nio.ch.Util$1)
>>>> - locked<0x00002aab69337230> (a
>>>> java.util.Collections$UnmodifiableSet)
>>>> - locked<0x00002aaaf3897930> (a
>> sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>>> thTimeout.java:332)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>>> 157)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>>> at
>> java.io.DataInputStream.readFully(DataInputStream.java:195)
>>>> at java.io.DataInputStream.readLong(DataInputStream.java:416)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFie
>>>> lds(DataTransferProtocol.java:120)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(D
>>>> FSClient.java:2638)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "DataStreamer for file
>>>>
>> /hbase-20.6/NAM_CLUSTERKEYS3/60db9e0113c7578ed913dc37e0856954/.tmp/42977
>>>> 03314038441248 block blk_2613854253703670198_5032689" daemon prio=10
>>>> tid=0x00002aae40e8e800 nid=0x75b8 in Object.wait()
>> [0x0000000046e2b000]
>>>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSCli
>>>> ent.java:2483)
>>>> - locked<0x00002aab6551a2d8> (a java.util.LinkedList)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "ResponseProcessor for block blk_8900384789253382699_5032549" daemon
>>>> prio=10 tid=0x00000000013cd000 nid=0x71d0 runnable
>> [0x0000000047431000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>>> - locked<0x00002aaae0569530> (a sun.nio.ch.Util$1)
>>>> - locked<0x00002aaae0569518> (a
>>>> java.util.Collections$UnmodifiableSet)
>>>> - locked<0x00002aaadb4c5c68> (a
>> sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>>> thTimeout.java:332)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>>> 157)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>>> at
>> java.io.DataInputStream.readFully(DataInputStream.java:195)
>>>> at java.io.DataInputStream.readLong(DataInputStream.java:416)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFie
>>>> lds(DataTransferProtocol.java:120)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(D
>>>> FSClient.java:2638)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "DataStreamer for file
>>>>
>> /hbase-20.6/.logs/doop5.dt.sv4.decarta.com,60020,1315768059645/doop5.dt.
>>>> sv4.decarta.com%3A60020.1315776307423 block
>>>> blk_8900384789253382699_5032549" daemon prio=10
>> tid=0x00000000016c3800
>>>> nid=0x71c8 in Object.wait() [0x000000004712e000]
>>>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSCli
>>>> ent.java:2483)
>>>> - locked<0x00002aab092a7f38> (a java.util.LinkedList)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "Attach Listener" daemon prio=10 tid=0x0000000001e32000 nid=0x50f4
>>>> waiting on condition [0x0000000000000000]
>>>> java.lang.Thread.State: RUNNABLE
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "RS_CLOSE_REGION-doop5.dt.sv4.decarta.com,60020,1315768059645-2"
>> daemon
>>>> prio=10 tid=0x00000000018cb800 nid=0x505 waiting on condition
>>>> [0x0000000047330000]
>>>> java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for<0x00002aaabcefb428> (a
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>>> at
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>>> ait(AbstractQueuedSynchronizer.java:2043)
>>>> at
>>>>
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>>>> 86)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
>>>> 1043)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>>> a:1103)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>>> va:603)
>>>> at java.lang.Thread.run(Thread.java:636)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "RS_CLOSE_REGION-doop5.dt.sv4.decarta.com,60020,1315768059645-1"
>> daemon
>>>> prio=10 tid=0x000000000163e000 nid=0x504 waiting on condition
>>>> [0x0000000047532000]
>>>> java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for<0x00002aaabcefb428> (a
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>>> at
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>>> ait(AbstractQueuedSynchronizer.java:2043)
>>>> at
>>>>
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>>>> 86)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
>>>> 1043)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>>> a:1103)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>>> va:603)
>>>> at java.lang.Thread.run(Thread.java:636)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "RS_CLOSE_REGION-doop5.dt.sv4.decarta.com,60020,1315768059645-0"
>> daemon
>>>> prio=10 tid=0x0000000001bff800 nid=0x503 waiting on condition
>>>> [0x0000000046f2c000]
>>>> java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for<0x00002aaabcefb428> (a
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>>> at
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>>> ait(AbstractQueuedSynchronizer.java:2043)
>>>> at
>>>>
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>>>> 86)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
>>>> 1043)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>>> a:1103)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>>> va:603)
>>>> at java.lang.Thread.run(Thread.java:636)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "sendParams-6" daemon prio=10 tid=0x00002aae401f1800 nid=0x6d3b
>> waiting
>>>> on condition [0x000000004702d000]
>>>> java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for<0x00002aaabceea260> (a
>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>> at
>>>>
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>>>> at
>>>>
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Synchro
>>>> nousQueue.java:453)
>>>> at
>>>>
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(Synchronous
>>>> Queue.java:352)
>>>> at
>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:903)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
>>>> 1043)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>>> a:1103)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>>> va:603)
>>>> at java.lang.Thread.run(Thread.java:636)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "RS_OPEN_REGION-doop5.dt.sv4.decarta.com,60020,1315768059645-2"
>> daemon
>>>> prio=10 tid=0x0000000001a7f000 nid=0x5b56 waiting on condition
>>>> [0x0000000046d2a000]
>>>> java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for<0x00002aaabcefb2e8> (a
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>>> at
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>>> ait(AbstractQueuedSynchronizer.java:2043)
>>>> at
>>>>
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>>>> 86)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
>>>> 1043)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>>> a:1103)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>>> va:603)
>>>> at java.lang.Thread.run(Thread.java:636)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "RS_OPEN_REGION-doop5.dt.sv4.decarta.com,60020,1315768059645-1"
>> daemon
>>>> prio=10 tid=0x0000000001af2800 nid=0x5b55 waiting on condition
>>>> [0x0000000042deb000]
>>>> java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for<0x00002aaabcefb2e8> (a
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>>> at
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>>> ait(AbstractQueuedSynchronizer.java:2043)
>>>> at
>>>>
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>>>> 86)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
>>>> 1043)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>>> a:1103)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>>> va:603)
>>>> at java.lang.Thread.run(Thread.java:636)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "RS_OPEN_REGION-doop5.dt.sv4.decarta.com,60020,1315768059645-0"
>> daemon
>>>> prio=10 tid=0x00000000014b3800 nid=0x5b54 waiting on condition
>>>> [0x0000000046c29000]
>>>> java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for<0x00002aaabcefb2e8> (a
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>>> at
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>>> ait(AbstractQueuedSynchronizer.java:2043)
>>>> at
>>>>
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>>>> 86)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
>>>> 1043)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>>> a:1103)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>>> va:603)
>>>> at java.lang.Thread.run(Thread.java:636)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "RS_OPEN_ROOT-doop5.dt.sv4.decarta.com,60020,1315768059645-0" daemon
>>>> prio=10 tid=0x0000000001ad2800 nid=0x556e waiting on condition
>>>> [0x0000000046a27000]
>>>> java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for<0x00002aaabcee8870> (a
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>>> at
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>>> ait(AbstractQueuedSynchronizer.java:2043)
>>>> at
>>>>
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>>>> 86)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
>>>> 1043)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>>> a:1103)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>>> va:603)
>>>> at java.lang.Thread.run(Thread.java:636)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "LRU Statistics #0" prio=10 tid=0x00002aae4048a800 nid=0x5566 waiting
>> on
>>>> condition [0x0000000046825000]
>>>> java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for<0x00002aaabd0695c0> (a
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>>
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>>>> at
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>>> aitNanos(AbstractQueuedSynchronizer.java:2081)
>>>> at java.util.concurrent.DelayQueue.take(DelayQueue.java:193)
>>>> at
>>>>
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(S
>>>> cheduledThreadPoolExecutor.java:688)
>>>> at
>>>>
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(S
>>>> cheduledThreadPoolExecutor.java:681)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
>>>> 1043)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>>> a:1103)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>>> va:603)
>>>> at java.lang.Thread.run(Thread.java:636)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "LruBlockCache.EvictionThread" daemon prio=10 tid=0x00002aae4049f800
>>>> nid=0x5565 in Object.wait() [0x0000000046724000]
>>>> java.lang.Thread.State: WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> at java.lang.Object.wait(Object.java:502)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlo
>>>> ckCache.java:519)
>>>> - locked<0x00002aaabce92f50> (a
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "PRI IPC Server handler 9 on 60020" daemon prio=10
>>>> tid=0x00002aae4057e000 nid=0x5564 waiting on condition
>>>> [0x0000000046623000]
>>>> java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for<0x00002aaabbb307e0> (a
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>>> at
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>>> ait(AbstractQueuedSynchronizer.java:2043)
>>>> at
>>>>
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>>>> 86)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
>>>> 5)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "PRI IPC Server handler 8 on 60020" daemon prio=10
>>>> tid=0x00002aae4057c800 nid=0x5563 waiting on condition
>>>> [0x0000000046522000]
>>>> java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for<0x00002aaabbb307e0> (a
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>>> at
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>>> ait(AbstractQueuedSynchronizer.java:2043)
>>>> at
>>>>
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>>>> 86)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
>>>> 5)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "PRI IPC Server handler 7 on 60020" daemon prio=10
>>>> tid=0x00002aae40559000 nid=0x5562 waiting on condition
>>>> [0x0000000046421000]
>>>> java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for<0x00002aaabbb307e0> (a
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>>> at
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>>> ait(AbstractQueuedSynchronizer.java:2043)
>>>> at
>>>>
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>>>> 86)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
>>>> 5)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "PRI IPC Server handler 6 on 60020" daemon prio=10
>>>> tid=0x00002aae40557800 nid=0x5561 waiting on condition
>>>> [0x0000000046320000]
>>>> java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for<0x00002aaabbb307e0> (a
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>>> at
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>>> ait(AbstractQueuedSynchronizer.java:2043)
>>>> at
>>>>
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>>>> 86)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
>>>> 5)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "PRI IPC Server handler 5 on 60020" daemon prio=10
>>>> tid=0x00002aae40555800 nid=0x5560 waiting on condition
>>>> [0x000000004621f000]
>>>> java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for<0x00002aaabbb307e0> (a
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>>> at
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>>> ait(AbstractQueuedSynchronizer.java:2043)
>>>> at
>>>>
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>>>> 86)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
>>>> 5)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "PRI IPC Server handler 4 on 60020" daemon prio=10
>>>> tid=0x00002aae40553800 nid=0x555f waiting on condition
>>>> [0x000000004611e000]
>>>> java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for<0x00002aaabbb307e0> (a
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>>> at
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>>> ait(AbstractQueuedSynchronizer.java:2043)
>>>> at
>>>>
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>>>> 86)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
>>>> 5)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "PRI IPC Server handler 3 on 60020" daemon prio=10
>>>> tid=0x00002aae4054f000 nid=0x555e waiting on condition
>>>> [0x000000004601d000]
>>>> java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for<0x00002aaabbb307e0> (a
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>>> at
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>>> ait(AbstractQueuedSynchronizer.java:2043)
>>>> at
>>>>
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>>>> 86)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
>>>> 5)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "PRI IPC Server handler 2 on 60020" daemon prio=10
>>>> tid=0x00000000012f8800 nid=0x555d waiting on condition
>>>> [0x0000000045f1c000]
>>>> java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for<0x00002aaabbb307e0> (a
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>>> at
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>>> ait(AbstractQueuedSynchronizer.java:2043)
>>>> at
>>>>
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>>>> 86)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
>>>> 5)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "PRI IPC Server handler 1 on 60020" daemon prio=10
>>>> tid=0x0000000001991000 nid=0x555c waiting on condition
>>>> [0x0000000045e1b000]
>>>> java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for<0x00002aaabbb307e0> (a
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>>> at
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>>> ait(AbstractQueuedSynchronizer.java:2043)
>>>> at
>>>>
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>>>> 86)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
>>>> 5)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "PRI IPC Server handler 0 on 60020" daemon prio=10
>>>> tid=0x000000000198e800 nid=0x555b waiting on condition
>>>> [0x0000000045d1a000]
>>>> java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for<0x00002aaabbb307e0> (a
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>>> at
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>>> ait(AbstractQueuedSynchronizer.java:2043)
>>>> at
>>>>
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>>>> 86)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
>>>> 5)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "IPC Server handler 29 on 60020" daemon prio=10
>> tid=0x000000000198c800
>>>> nid=0x555a runnable [0x0000000045c18000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>>> - locked<0x00002aac053b8450> (a sun.nio.ch.Util$1)
>>>> - locked<0x00002aac053b8438> (a
>>>> java.util.Collections$UnmodifiableSet)
>>>> - locked<0x00002aab1037afc8> (a
>> sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>>> thTimeout.java:332)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>>> 157)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>>> at
>>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>>> - locked<0x00002aac92976a30> (a java.io.BufferedInputStream)
>>>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>>> 50)
>>>> - locked<0x00002aabc43be0a8> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>>> a:237)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>>> - locked<0x00002aabc43be0a8> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>>> - locked<0x00002aabc43be0a8> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>>> a:1899)
>>>> - locked<0x00002aac1a4e0ea8> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>>> )
>>>> - locked<0x00002aac1a4e0ea8> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at java.io.DataInputStream.read(DataInputStream.java:149)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>>> dRangeFileInputStream.java:105)
>>>> - locked<0x00002aac237c37a8> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>>> at
>>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>>> - locked<0x00002aad39485080> (a java.io.BufferedInputStream)
>>>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>>> )
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>>> - locked<0x00002aab0b5ba8a8> (a [B)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>>> 76)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>>>> ner.java:87)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>>> :82)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>>> :262)
>>>> - locked<0x00002aab89f59aa8> (a
>>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>>> :114)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>>> HRegion.java:2344)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2301)
>>>> - locked<0x00002aab016d6988> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2317)
>>>> - locked<0x00002aab016d6988> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>>> va:1832)
>>>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
>> Source)
>>>> at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>>> Impl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:616)
>>>> at
>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>>> 9)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "IPC Server handler 28 on 60020" daemon prio=10
>> tid=0x0000000001864000
>>>> nid=0x5559 runnable [0x0000000045b17000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>>> - locked<0x00002aabaedb0608> (a sun.nio.ch.Util$1)
>>>> - locked<0x00002aabaedb05f0> (a
>>>> java.util.Collections$UnmodifiableSet)
>>>> - locked<0x00002aaca096fa50> (a
>> sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>>> thTimeout.java:332)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>>> 157)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>>> at
>>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>>> - locked<0x00002aac62a70b28> (a java.io.BufferedInputStream)
>>>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>>> 50)
>>>> - locked<0x00002aab856e6220> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>>> a:237)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>>> - locked<0x00002aab856e6220> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>>> - locked<0x00002aab856e6220> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>>> a:1899)
>>>> - locked<0x00002aab0264afe8> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>>> )
>>>> - locked<0x00002aab0264afe8> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at java.io.DataInputStream.read(DataInputStream.java:149)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>>> dRangeFileInputStream.java:105)
>>>> - locked<0x00002aaae070be48> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>>> at
>>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>>> - locked<0x00002aac18961940> (a java.io.BufferedInputStream)
>>>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>>> )
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>>> - locked<0x00002aabad8605d0> (a [B)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>>> 76)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>>>> ner.java:87)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>>> :82)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>>> :262)
>>>> - locked<0x00002aab077e5d80> (a
>>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>>> :114)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>>> HRegion.java:2344)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2301)
>>>> - locked<0x00002aad30d57c18> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2317)
>>>> - locked<0x00002aad30d57c18> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>>> va:1832)
>>>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
>> Source)
>>>> at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>>> Impl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:616)
>>>> at
>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>>> 9)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "IPC Server handler 27 on 60020" daemon prio=10
>> tid=0x0000000001862000
>>>> nid=0x5558 runnable [0x0000000045a16000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>>> - locked<0x00002aab2ea61b70> (a sun.nio.ch.Util$1)
>>>> - locked<0x00002aab2ea61b88> (a
>>>> java.util.Collections$UnmodifiableSet)
>>>> - locked<0x00002aacbb364578> (a
>> sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>>> thTimeout.java:332)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>>> 157)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>>> at
>>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>>> - locked<0x00002aab1ffc0d00> (a java.io.BufferedInputStream)
>>>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>>> 50)
>>>> - locked<0x00002aad890f4640> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>>> a:237)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>>> - locked<0x00002aad890f4640> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>>> - locked<0x00002aad890f4640> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>>> a:1899)
>>>> - locked<0x00002aaba1282b10> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>>> )
>>>> - locked<0x00002aaba1282b10> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at java.io.DataInputStream.read(DataInputStream.java:149)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>>> dRangeFileInputStream.java:105)
>>>> - locked<0x00002aac528d15d0> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>>> at
>>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>>> - locked<0x00002aaaaf68a630> (a java.io.BufferedInputStream)
>>>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>>> )
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>>> - locked<0x00002aabc5c948b8> (a [B)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>>> 76)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
>>>> anner.java:115)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
>>>> va:255)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
>>>> va:394)
>>>> - locked<0x00002aab95832718> (a
>>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>>> :293)
>>>> - locked<0x00002aab95832718> (a
>>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>>> :114)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>>> HRegion.java:2344)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2301)
>>>> - locked<0x00002aab28d7de90> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2317)
>>>> - locked<0x00002aab28d7de90> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>>> va:1832)
>>>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
>> Source)
>>>> at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>>> Impl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:616)
>>>> at
>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>>> 9)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "IPC Server handler 26 on 60020" daemon prio=10
>> tid=0x0000000001860000
>>>> nid=0x5557 runnable [0x0000000045915000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>>> - locked<0x00002aabaaa6b808> (a sun.nio.ch.Util$1)
>>>> - locked<0x00002aabaaa6b7f0> (a
>>>> java.util.Collections$UnmodifiableSet)
>>>> - locked<0x00002aaaf3899ca8> (a
>> sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>>> thTimeout.java:332)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>>> 157)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>>> at
>>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>>> - locked<0x00002aaac1f59118> (a java.io.BufferedInputStream)
>>>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>>> 50)
>>>> - locked<0x00002aaac0e99db0> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>>> a:237)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>>> - locked<0x00002aaac0e99db0> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>>> - locked<0x00002aaac0e99db0> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>>> a:1899)
>>>> - locked<0x00002aabb7221e88> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>>> )
>>>> - locked<0x00002aabb7221e88> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at java.io.DataInputStream.read(DataInputStream.java:149)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>>> dRangeFileInputStream.java:105)
>>>> - locked<0x00002aac28621da0> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>>> at
>>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>>> - locked<0x00002aac18962690> (a java.io.BufferedInputStream)
>>>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>>> )
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>>> - locked<0x00002aab6876b850> (a [B)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.loadBlock(HFile.ja
>>>> va:1447)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.reseekTo(HFile.jav
>>>> a:1325)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(St
>>>> oreFileScanner.java:152)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
>>>> anner.java:110)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
>>>> va:255)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
>>>> va:394)
>>>> - locked<0x00002aaabe3468f8> (a
>>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>>> :293)
>>>> - locked<0x00002aaabe3468f8> (a
>>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>>> :114)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>>> HRegion.java:2344)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2301)
>>>> - locked<0x00002aac8e37dbc8> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2317)
>>>> - locked<0x00002aac8e37dbc8> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>>> va:1832)
>>>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
>> Source)
>>>> at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>>> Impl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:616)
>>>> at
>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>>> 9)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "IPC Server handler 25 on 60020" daemon prio=10
>> tid=0x000000000173d800
>>>> nid=0x5556 runnable [0x0000000045814000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>>> - locked<0x00002aab20178510> (a sun.nio.ch.Util$1)
>>>> - locked<0x00002aab20178528> (a
>>>> java.util.Collections$UnmodifiableSet)
>>>> - locked<0x00002aaad6187af0> (a
>> sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>>> thTimeout.java:332)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>>> 157)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>>> at
>>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>>> - locked<0x00002aac891a6610> (a java.io.BufferedInputStream)
>>>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>>> 50)
>>>> - locked<0x00002aac1de6d9f8> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>>> a:237)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>>> - locked<0x00002aac1de6d9f8> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>>> - locked<0x00002aac1de6d9f8> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>>> a:1899)
>>>> - locked<0x00002aaac041c6c8> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>>> )
>>>> - locked<0x00002aaac041c6c8> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at java.io.DataInputStream.read(DataInputStream.java:149)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>>> dRangeFileInputStream.java:105)
>>>> - locked<0x00002aab25e717e0> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>>> at
>>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>>> - locked<0x00002aac3e6e4e28> (a java.io.BufferedInputStream)
>>>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>>> )
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>>> - locked<0x00002aaad4a425f0> (a [B)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>>> 76)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>>>> ner.java:87)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>>> :82)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>>> :262)
>>>> - locked<0x00002aaabd5eb7e0> (a
>>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>>> :114)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>>> HRegion.java:2344)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2301)
>>>> - locked<0x00002aac8e37dc68> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2317)
>>>> - locked<0x00002aac8e37dc68> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>>> va:1832)
>>>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
>> Source)
>>>> at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>>> Impl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:616)
>>>> at
>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>>> 9)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "IPC Server handler 24 on 60020" daemon prio=10
>> tid=0x000000000173a000
>>>> nid=0x5555 waiting for monitor entry [0x0000000045714000]
>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2317)
>>>> - waiting to lock<0x00002aab016d6618> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>>> va:1832)
>>>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
>> Source)
>>>> at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>>> Impl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:616)
>>>> at
>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>>> 9)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "IPC Server handler 23 on 60020" daemon prio=10
>> tid=0x0000000001444000
>>>> nid=0x5554 runnable [0x0000000045612000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>>> - locked<0x00002aad70044d30> (a sun.nio.ch.Util$1)
>>>> - locked<0x00002aad70044d48> (a
>>>> java.util.Collections$UnmodifiableSet)
>>>> - locked<0x00002aab328f1a40> (a
>> sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>>> thTimeout.java:332)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>>> 157)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>>> at
>>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>>> - locked<0x00002aaac9d48e68> (a java.io.BufferedInputStream)
>>>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>>> 50)
>>>> - locked<0x00002aac1de6dd40> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>>> a:237)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>>> - locked<0x00002aac1de6dd40> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>>> - locked<0x00002aac1de6dd40> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>>> a:1899)
>>>> - locked<0x00002aab39d88148> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>>> )
>>>> - locked<0x00002aab39d88148> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at java.io.DataInputStream.read(DataInputStream.java:149)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>>> dRangeFileInputStream.java:105)
>>>> - locked<0x00002aab7addd0d8> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>>> at
>>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>>> - locked<0x00002aab2811e8e8> (a java.io.BufferedInputStream)
>>>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>>> )
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>>> - locked<0x00002aabe56e8858> (a [B)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>>> 76)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>>>> ner.java:87)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>>> :82)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>>> :262)
>>>> - locked<0x00002aad15b7d360> (a
>>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>>> :114)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>>> HRegion.java:2344)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2301)
>>>> - locked<0x00002aabc7a3d258> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2317)
>>>> - locked<0x00002aabc7a3d258> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>>> va:1832)
>>>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
>> Source)
>>>> at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>>> Impl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:616)
>>>> at
>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>>> 9)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "IPC Server handler 22 on 60020" daemon prio=10
>> tid=0x0000000001442000
>>>> nid=0x5553 runnable [0x0000000045511000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>>> - locked<0x00002aabf6e38450> (a sun.nio.ch.Util$1)
>>>> - locked<0x00002aabf6e38468> (a
>>>> java.util.Collections$UnmodifiableSet)
>>>> - locked<0x00002aabb9e99468> (a
>> sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>>> thTimeout.java:332)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>>> 157)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>>> at
>>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>>> - locked<0x00002aaae13c0340> (a java.io.BufferedInputStream)
>>>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>>> 50)
>>>> - locked<0x00002aad427a91c0> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>>> a:237)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>>> - locked<0x00002aad427a91c0> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>>> - locked<0x00002aad427a91c0> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>>> a:1899)
>>>> - locked<0x00002aab7c7ca0c8> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>>> )
>>>> - locked<0x00002aab7c7ca0c8> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at java.io.DataInputStream.read(DataInputStream.java:149)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>>> dRangeFileInputStream.java:105)
>>>> - locked<0x00002aab3748f1a8> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>>> at
>>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>>> - locked<0x00002aab52d56ed8> (a java.io.BufferedInputStream)
>>>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>>> )
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>>> - locked<0x00002aab60a86ca8> (a [B)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>>> 76)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(St
>>>> oreFileScanner.java:158)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
>>>> anner.java:110)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
>>>> va:255)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
>>>> va:394)
>>>> - locked<0x00002aab706a3e98> (a
>>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>>> :293)
>>>> - locked<0x00002aab706a3e98> (a
>>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>>> :114)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>>> HRegion.java:2344)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2301)
>>>> - locked<0x00002aaae3489758> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2317)
>>>> - locked<0x00002aaae3489758> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>>> va:1832)
>>>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
>> Source)
>>>> at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>>> Impl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:616)
>>>> at
>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>>> 9)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "IPC Server handler 21 on 60020" daemon prio=10
>> tid=0x0000000001792000
>>>> nid=0x5552 waiting for monitor entry [0x0000000045411000]
>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2317)
>>>> - waiting to lock<0x00002aabc7a3d3e8> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>>> va:1832)
>>>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
>> Source)
>>>> at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>>> Impl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:616)
>>>> at
>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>>> 9)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "IPC Server handler 20 on 60020" daemon prio=10
>> tid=0x000000000178e000
>>>> nid=0x5551 runnable [0x000000004530f000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>>> - locked<0x00002aab3745e018> (a sun.nio.ch.Util$1)
>>>> - locked<0x00002aab3745e030> (a
>>>> java.util.Collections$UnmodifiableSet)
>>>> - locked<0x00002aab7d5cc0c0> (a
>> sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>>> thTimeout.java:332)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>>> 157)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>>> at
>>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>>> - locked<0x00002aac035e4990> (a java.io.BufferedInputStream)
>>>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>>> 50)
>>>> - locked<0x00002aad890f42f8> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>>> a:237)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>>> - locked<0x00002aad890f42f8> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>>> - locked<0x00002aad890f42f8> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>>> a:1899)
>>>> - locked<0x00002aab1baffa50> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>>> )
>>>> - locked<0x00002aab1baffa50> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at java.io.DataInputStream.read(DataInputStream.java:149)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>>> dRangeFileInputStream.java:105)
>>>> - locked<0x00002aaadef99fe8> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>>> at
>>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>>> - locked<0x00002aac189625a0> (a java.io.BufferedInputStream)
>>>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>>> )
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>>> - locked<0x00002aabb88c7ce0> (a [B)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>>> 76)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
>>>> anner.java:115)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
>>>> va:255)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
>>>> va:394)
>>>> - locked<0x00002aaadfa95ec0> (a
>>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>>> :293)
>>>> - locked<0x00002aaadfa95ec0> (a
>>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>>> :114)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>>> HRegion.java:2344)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2301)
>>>> - locked<0x00002aabf6832658> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2317)
>>>> - locked<0x00002aabf6832658> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>>> va:1832)
>>>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
>> Source)
>>>> at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>>> Impl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:616)
>>>> at
>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>>> 9)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "IPC Server handler 19 on 60020" daemon prio=10
>> tid=0x0000000001b19800
>>>> nid=0x5550 runnable [0x000000004520e000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>>> - locked<0x00002aab94af3848> (a sun.nio.ch.Util$1)
>>>> - locked<0x00002aab94af3830> (a
>>>> java.util.Collections$UnmodifiableSet)
>>>> - locked<0x00002aac6ed3e1d8> (a
>> sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>>> thTimeout.java:332)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>>> 157)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>>> at
>>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>>> - locked<0x00002aaac1194578> (a java.io.BufferedInputStream)
>>>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>>> 50)
>>>> - locked<0x00002aad890f2600> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>>> a:237)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>>> - locked<0x00002aad890f2600> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>>> - locked<0x00002aad890f2600> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>>> a:1899)
>>>> - locked<0x00002aaae78be750> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>>> )
>>>> - locked<0x00002aaae78be750> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at java.io.DataInputStream.read(DataInputStream.java:149)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>>> dRangeFileInputStream.java:105)
>>>> - locked<0x00002aabba906cc0> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>>> at
>>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>>> - locked<0x00002aac18963ef0> (a java.io.BufferedInputStream)
>>>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>>> )
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>>> - locked<0x00002aaacd2974c0> (a [B)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>>> 76)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
>>>> anner.java:115)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
>>>> va:255)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
>>>> va:394)
>>>> - locked<0x00002aab328ec218> (a
>>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>>> :293)
>>>> - locked<0x00002aab328ec218> (a
>>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>>> :114)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>>> HRegion.java:2344)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2301)
>>>> - locked<0x00002aabbaf1c898> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2317)
>>>> - locked<0x00002aabbaf1c898> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>>> va:1832)
>>>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
>> Source)
>>>> at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>>> Impl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:616)
>>>> at
>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>>> 9)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "IPC Server handler 18 on 60020" daemon prio=10
>> tid=0x0000000001b17800
>>>> nid=0x554f runnable [0x000000004510d000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>>> - locked<0x00002aaae98101b0> (a sun.nio.ch.Util$1)
>>>> - locked<0x00002aaae9810198> (a
>>>> java.util.Collections$UnmodifiableSet)
>>>> - locked<0x00002aaaeba1b040> (a
>> sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>>> thTimeout.java:332)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>>> 157)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>>> at
>>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>>> - locked<0x00002aab1ac7d288> (a java.io.BufferedInputStream)
>>>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>>> 50)
>>>> - locked<0x00002aad890f0db8> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>>> a:237)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>>> - locked<0x00002aad890f0db8> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>>> - locked<0x00002aad890f0db8> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>>> a:1899)
>>>> - locked<0x00002aab8c671d30> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>>> )
>>>> - locked<0x00002aab8c671d30> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at java.io.DataInputStream.read(DataInputStream.java:149)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>>> dRangeFileInputStream.java:105)
>>>> - locked<0x00002aab8c4f3de0> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>>> at
>>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>>> - locked<0x00002aac18963dd0> (a java.io.BufferedInputStream)
>>>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>>> )
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>>> - locked<0x00002aab8d9fd7c8> (a [B)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>>> 76)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
>>>> anner.java:115)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
>>>> va:255)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
>>>> va:394)
>>>> - locked<0x00002aaba69901b8> (a
>>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>>> :293)
>>>> - locked<0x00002aaba69901b8> (a
>>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>>> :114)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>>> HRegion.java:2344)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2301)
>>>> - locked<0x00002aabbaf1e468> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2317)
>>>> - locked<0x00002aabbaf1e468> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>>> va:1832)
>>>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
>> Source)
>>>> at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>>> Impl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:616)
>>>> at
>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>>> 9)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "IPC Server handler 17 on 60020" daemon prio=10
>> tid=0x0000000001b15800
>>>> nid=0x554e runnable [0x000000004500c000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>>> - locked<0x00002aaac2d0d348> (a sun.nio.ch.Util$1)
>>>> - locked<0x00002aaac2d0d330> (a
>>>> java.util.Collections$UnmodifiableSet)
>>>> - locked<0x00002aacbb364b18> (a
>> sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>>> thTimeout.java:332)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>>> 157)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>>> at
>>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>>> - locked<0x00002aaac2750e38> (a java.io.BufferedInputStream)
>>>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>>> 50)
>>>> - locked<0x00002aaac1028128> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>>> a:237)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>>> - locked<0x00002aaac1028128> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>>> - locked<0x00002aaac1028128> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>>> a:1899)
>>>> - locked<0x00002aaae41f13d8> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>>> )
>>>> - locked<0x00002aaae41f13d8> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at java.io.DataInputStream.read(DataInputStream.java:149)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>>> dRangeFileInputStream.java:105)
>>>> - locked<0x00002aab3c08c348> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>>> at
>>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>>> - locked<0x00002aab52d54ef8> (a java.io.BufferedInputStream)
>>>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>>> )
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>>> - locked<0x00002aac0cb261b0> (a [B)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>>> 76)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>>>> ner.java:87)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>>> :82)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>>> :262)
>>>> - locked<0x00002aaabefb9c10> (a
>>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>>> :114)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>>> HRegion.java:2344)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2301)
>>>> - locked<0x00002aad30d56688> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2317)
>>>> - locked<0x00002aad30d56688> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>>> va:1832)
>>>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
>> Source)
>>>> at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>>> Impl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:616)
>>>> at
>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>>> 9)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "IPC Server handler 16 on 60020" daemon prio=10
>> tid=0x00000000013fa000
>>>> nid=0x554d runnable [0x0000000044f0b000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>>> - locked<0x00002aabec61aa08> (a sun.nio.ch.Util$1)
>>>> - locked<0x00002aabec61a9f0> (a
>>>> java.util.Collections$UnmodifiableSet)
>>>> - locked<0x00002aaae7590720> (a
>> sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>>> thTimeout.java:332)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>>> 157)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>>> at
>>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>>> - locked<0x00002aacf2246cb0> (a java.io.BufferedInputStream)
>>>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>>> 50)
>>>> - locked<0x00002aad06dd2c70> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>>> a:237)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>>> - locked<0x00002aad06dd2c70> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>>> - locked<0x00002aad06dd2c70> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>>> a:1899)
>>>> - locked<0x00002aabe38abcd8> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>>> )
>>>> - locked<0x00002aabe38abcd8> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at java.io.DataInputStream.read(DataInputStream.java:149)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>>> dRangeFileInputStream.java:105)
>>>> - locked<0x00002aaad08ca208> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>>> at
>>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>>> - locked<0x00002aac189636e0> (a java.io.BufferedInputStream)
>>>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>>> )
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>>> - locked<0x00002aab84fe43c8> (a [B)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>>> 76)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(St
>>>> oreFileScanner.java:158)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
>>>> anner.java:110)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
>>>> va:255)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
>>>> va:394)
>>>> - locked<0x00002aad072dbd30> (a
>>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>>> :293)
>>>> - locked<0x00002aad072dbd30> (a
>>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>>> :114)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>>> HRegion.java:2344)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2301)
>>>> - locked<0x00002aabdbaa0490> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2317)
>>>> - locked<0x00002aabdbaa0490> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>>> va:1832)
>>>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
>> Source)
>>>> at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>>> Impl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:616)
>>>> at
>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>>> 9)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "IPC Server handler 15 on 60020" daemon prio=10
>> tid=0x00000000013f7800
>>>> nid=0x554c runnable [0x0000000044e0a000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>>> - locked<0x00002aac558cde50> (a sun.nio.ch.Util$1)
>>>> - locked<0x00002aac558cde38> (a
>>>> java.util.Collections$UnmodifiableSet)
>>>> - locked<0x00002aac5cd41558> (a
>> sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>>> thTimeout.java:332)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>>> 157)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>>> at
>>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>>> - locked<0x00002aadcf530468> (a java.io.BufferedInputStream)
>>>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>>> 50)
>>>> - locked<0x00002aad427a97a8> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>>> a:237)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>>> - locked<0x00002aad427a97a8> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>>> - locked<0x00002aad427a97a8> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>>> a:1899)
>>>> - locked<0x00002aab9410bc00> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>>> )
>>>> - locked<0x00002aab9410bc00> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at java.io.DataInputStream.read(DataInputStream.java:149)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>>> dRangeFileInputStream.java:105)
>>>> - locked<0x00002aaca7cfcf70> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>>> at
>>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>>> - locked<0x00002aacf952a6c8> (a java.io.BufferedInputStream)
>>>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>>> )
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>>> - locked<0x00002aaac4472350> (a [B)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>>> 76)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>>>> ner.java:87)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>>> :82)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>>> :297)
>>>> - locked<0x00002aaaefe12070> (a
>>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>>> :114)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>>> HRegion.java:2344)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2301)
>>>> - locked<0x00002aabbaf1d1a8> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2317)
>>>> - locked<0x00002aabbaf1d1a8> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>>> va:1832)
>>>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
>> Source)
>>>> at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>>> Impl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:616)
>>>> at
>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>>> 9)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "IPC Server handler 14 on 60020" daemon prio=10
>> tid=0x00000000013f5800
>>>> nid=0x554b runnable [0x0000000044d09000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>>> - locked<0x00002aab28f91848> (a sun.nio.ch.Util$1)
>>>> - locked<0x00002aab28f91860> (a
>>>> java.util.Collections$UnmodifiableSet)
>>>> - locked<0x00002aaaf5031970> (a
>> sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>>> thTimeout.java:332)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>>> 157)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>>> at
>>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>>> - locked<0x00002aacec8cf590> (a java.io.BufferedInputStream)
>>>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>>> 50)
>>>> - locked<0x00002aaaf789b048> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>>> a:237)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>>> - locked<0x00002aaaf789b048> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>>> - locked<0x00002aaaf789b048> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>>> a:1899)
>>>> - locked<0x00002aab9c1df470> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>>> )
>>>> - locked<0x00002aab9c1df470> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at java.io.DataInputStream.read(DataInputStream.java:149)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>>> dRangeFileInputStream.java:105)
>>>> - locked<0x00002aab7ec60418> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>>> at
>>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>>> - locked<0x00002aab44c84dd8> (a java.io.BufferedInputStream)
>>>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>>> )
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>>> - locked<0x00002aab227a2dc8> (a [B)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>>> 76)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(St
>>>> oreFileScanner.java:158)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
>>>> anner.java:110)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
>>>> va:255)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
>>>> va:394)
>>>> - locked<0x00002aab50d25f28> (a
>>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>>> :293)
>>>> - locked<0x00002aab50d25f28> (a
>>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>>> :114)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>>> HRegion.java:2344)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2301)
>>>> - locked<0x00002aaae9efd9f0> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2317)
>>>> - locked<0x00002aaae9efd9f0> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>>> va:1832)
>>>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
>> Source)
>>>> at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>>> Impl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:616)
>>>> at
>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>>> 9)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "IPC Server handler 13 on 60020" daemon prio=10
>> tid=0x0000000001b61800
>>>> nid=0x554a waiting for monitor entry [0x0000000044c09000]
>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2317)
>>>> - waiting to lock<0x00002aacc684dad0> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>>> va:1832)
>>>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
>> Source)
>>>> at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>>> Impl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:616)
>>>> at
>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>>> 9)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "IPC Server handler 12 on 60020" daemon prio=10
>> tid=0x0000000001b5f800
>>>> nid=0x5549 runnable [0x0000000044b07000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>>> - locked<0x00002aab618bb060> (a sun.nio.ch.Util$1)
>>>> - locked<0x00002aabe1d82718> (a
>>>> java.util.Collections$UnmodifiableSet)
>>>> - locked<0x00002aab76896238> (a
>> sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>>> thTimeout.java:332)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>>> 157)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>>> at
>>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>>> - locked<0x00002aab016dc128> (a java.io.BufferedInputStream)
>>>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>>> 50)
>>>> - locked<0x00002aad03182340> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>>> a:237)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>>> - locked<0x00002aad03182340> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>>> - locked<0x00002aad03182340> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>>> a:1899)
>>>> - locked<0x00002aab8b32e3e0> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>>> )
>>>> - locked<0x00002aab8b32e3e0> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at java.io.DataInputStream.read(DataInputStream.java:149)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>>> dRangeFileInputStream.java:105)
>>>> - locked<0x00002aab8b436170> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>>> at
>>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>>> - locked<0x00002aacd4fb8038> (a java.io.BufferedInputStream)
>>>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>>> )
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>>> - locked<0x00002aab8b4b4908> (a [B)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>>> 76)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>>>> ner.java:87)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>>> :82)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>>> :262)
>>>> - locked<0x00002aab480f55b8> (a
>>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>>> :114)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>>> HRegion.java:2344)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2301)
>>>> - locked<0x00002aaae9efdb30> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2317)
>>>> - locked<0x00002aaae9efdb30> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>>> va:1832)
>>>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
>> Source)
>>>> at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>>> Impl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:616)
>>>> at
>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>>> 9)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "IPC Server handler 11 on 60020" daemon prio=10
>> tid=0x0000000001b5d000
>>>> nid=0x5548 waiting for monitor entry [0x0000000044a07000]
>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2317)
>>>> - waiting to lock<0x00002aacc684dad0> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>>> va:1832)
>>>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
>> Source)
>>>> at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>>> Impl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:616)
>>>> at
>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>>> 9)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "IPC Server handler 10 on 60020" daemon prio=10
>> tid=0x000000000196a800
>>>> nid=0x5547 runnable [0x0000000044905000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>>> - locked<0x00002aad7443dcd8> (a sun.nio.ch.Util$1)
>>>> - locked<0x00002aad7a31cb00> (a
>>>> java.util.Collections$UnmodifiableSet)
>>>> - locked<0x00002aac9067d2d8> (a
>> sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>>> thTimeout.java:332)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>>> 157)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>>> at
>>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>>> - locked<0x00002aad5103b0a0> (a java.io.BufferedInputStream)
>>>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>>> 50)
>>>> - locked<0x00002aadc79059c8> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>>> a:237)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>>> - locked<0x00002aadc79059c8> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>>> - locked<0x00002aadc79059c8> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>>> a:1899)
>>>> - locked<0x00002aabf0f472b0> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>>> )
>>>> - locked<0x00002aabf0f472b0> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at java.io.DataInputStream.read(DataInputStream.java:149)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>>> dRangeFileInputStream.java:105)
>>>> - locked<0x00002aab8b2b29b0> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>>> at
>>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>>> - locked<0x00002aad0477a310> (a java.io.BufferedInputStream)
>>>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>>> )
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>>> - locked<0x00002aabb7b16fd0> (a [B)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>>> 76)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>>>> ner.java:87)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>>> :82)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>>> :262)
>>>> - locked<0x00002aab59165e10> (a
>>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>>> :114)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>>> HRegion.java:2344)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2301)
>>>> - locked<0x00002aac8e37dd58> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2317)
>>>> - locked<0x00002aac8e37dd58> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>>> va:1832)
>>>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
>> Source)
>>>> at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>>> Impl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:616)
>>>> at
>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>>> 9)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "IPC Server handler 9 on 60020" daemon prio=10 tid=0x0000000001968000
>>>> nid=0x5546 waiting for monitor entry [0x0000000044805000]
>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.isFilterDone(
>>>> HRegion.java:2324)
>>>> - waiting to lock<0x00002aacc684dad0> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>>> va:1850)
>>>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
>> Source)
>>>> at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>>> Impl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:616)
>>>> at
>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>>> 9)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "IPC Server handler 8 on 60020" daemon prio=10 tid=0x0000000001967000
>>>> nid=0x5545 runnable [0x0000000044703000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>>> - locked<0x00002aabb0b7b868> (a sun.nio.ch.Util$1)
>>>> - locked<0x00002aabb0b7b880> (a
>>>> java.util.Collections$UnmodifiableSet)
>>>> - locked<0x00002aaac79442c8> (a
>> sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>>> thTimeout.java:332)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>>> 157)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>>> at
>>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>>> - locked<0x00002aad0106a6b0> (a java.io.BufferedInputStream)
>>>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>>> 50)
>>>> - locked<0x00002aac108e6768> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>>> a:237)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>>> - locked<0x00002aac108e6768> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>>> - locked<0x00002aac108e6768> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>>> a:1899)
>>>> - locked<0x00002aac4d4ecd40> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>>> )
>>>> - locked<0x00002aac4d4ecd40> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at java.io.DataInputStream.read(DataInputStream.java:149)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>>> dRangeFileInputStream.java:105)
>>>> - locked<0x00002aaad20c3b08> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>>> at
>>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>>> - locked<0x00002aac536bda88> (a java.io.BufferedInputStream)
>>>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>>> )
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>>> - locked<0x00002aaadfc86320> (a [B)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.loadBlock(HFile.ja
>>>> va:1447)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.reseekTo(HFile.jav
>>>> a:1325)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(St
>>>> oreFileScanner.java:152)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
>>>> anner.java:110)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
>>>> va:255)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
>>>> va:394)
>>>> - locked<0x00002aad072dc1c8> (a
>>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>>> :293)
>>>> - locked<0x00002aad072dc1c8> (a
>>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>>> :114)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>>> HRegion.java:2344)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2301)
>>>> - locked<0x00002aac8e37d808> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2317)
>>>> - locked<0x00002aac8e37d808> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>>> va:1832)
>>>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
>> Source)
>>>> at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>>> Impl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:616)
>>>> at
>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>>> 9)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "IPC Server handler 7 on 60020" daemon prio=10 tid=0x00000000016de800
>>>> nid=0x5544 runnable [0x0000000044602000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>>> - locked<0x00002aab9f4dc588> (a sun.nio.ch.Util$1)
>>>> - locked<0x00002aab9f4dc570> (a
>>>> java.util.Collections$UnmodifiableSet)
>>>> - locked<0x00002aac6ed3e250> (a
>> sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>>> thTimeout.java:332)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>>> 157)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>>> at
>>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>>> - locked<0x00002aad003a0178> (a java.io.BufferedInputStream)
>>>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>>> 50)
>>>> - locked<0x00002aabbd698ec8> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>>> a:237)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>>> - locked<0x00002aabbd698ec8> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>>> - locked<0x00002aabbd698ec8> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>>> a:1899)
>>>> - locked<0x00002aaad91ecfa8> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>>> )
>>>> - locked<0x00002aaad91ecfa8> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at java.io.DataInputStream.read(DataInputStream.java:149)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>>> dRangeFileInputStream.java:105)
>>>> - locked<0x00002aac237c5090> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>>> at
>>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>>> - locked<0x00002aad65d9a7f0> (a java.io.BufferedInputStream)
>>>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>>> )
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>>> - locked<0x00002aaaccc5d6c8> (a [B)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>>> 76)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>>>> ner.java:87)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>>> :82)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>>> :262)
>>>> - locked<0x00002aad15b7d2b8> (a
>>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>>> :114)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>>> HRegion.java:2344)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2301)
>>>> - locked<0x00002aacc684dad0> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2317)
>>>> - locked<0x00002aacc684dad0> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>>> va:1832)
>>>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
>> Source)
>>>> at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>>> Impl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:616)
>>>> at
>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>>> 9)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "IPC Server handler 6 on 60020" daemon prio=10 tid=0x00000000016dd000
>>>> nid=0x5543 runnable [0x0000000044501000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>>> - locked<0x00002aaaf7db82f0> (a sun.nio.ch.Util$1)
>>>> - locked<0x00002aaaf7db82d8> (a
>>>> java.util.Collections$UnmodifiableSet)
>>>> - locked<0x00002aab6f3339a8> (a
>> sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>>> thTimeout.java:332)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>>> 157)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>>> at
>>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>>> - locked<0x00002aab494cdcb8> (a java.io.BufferedInputStream)
>>>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>>> 50)
>>>> - locked<0x00002aad890f5018> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>>> a:237)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>>> - locked<0x00002aad890f5018> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>>> - locked<0x00002aad890f5018> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>>> a:1899)
>>>> - locked<0x00002aaae4d56928> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>>> )
>>>> - locked<0x00002aaae4d56928> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at java.io.DataInputStream.read(DataInputStream.java:149)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>>> dRangeFileInputStream.java:105)
>>>> - locked<0x00002aaae8d98798> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>>> at
>>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>>> - locked<0x00002aaaaf686f40> (a java.io.BufferedInputStream)
>>>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>>> )
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>>> - locked<0x00002aaae77f0ad8> (a [B)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>>> 76)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>>>> ner.java:87)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>>> :82)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>>> :262)
>>>> - locked<0x00002aaae4cd85a0> (a
>>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>>> :114)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>>> HRegion.java:2344)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2301)
>>>> - locked<0x00002aad30d55d28> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2317)
>>>> - locked<0x00002aad30d55d28> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>>> va:1832)
>>>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
>> Source)
>>>> at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>>> Impl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:616)
>>>> at
>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>>> 9)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "IPC Server handler 5 on 60020" daemon prio=10 tid=0x000000000194c800
>>>> nid=0x5542 runnable [0x0000000044400000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>>> - locked<0x00002aaac5bf0ff0> (a sun.nio.ch.Util$1)
>>>> - locked<0x00002aaac5bf10c8> (a
>>>> java.util.Collections$UnmodifiableSet)
>>>> - locked<0x00002aab0492d6d8> (a
>> sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>>> thTimeout.java:332)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>>> 157)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>>> at
>>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>>> at
>>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>>> - locked<0x00002aac451e5d20> (a java.io.BufferedInputStream)
>>>> at java.io.DataInputStream.read(DataInputStream.java:149)
>>>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>>> 89)
>>>> - locked<0x00002aabf9312408> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>>> a:237)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>>> - locked<0x00002aabf9312408> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>>> - locked<0x00002aabf9312408> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>>> a:1899)
>>>> - locked<0x00002aab78dc6068> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>>> )
>>>> - locked<0x00002aab78dc6068> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at java.io.DataInputStream.read(DataInputStream.java:149)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>>> dRangeFileInputStream.java:105)
>>>> - locked<0x00002aab14390138> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>>> at
>>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>>> - locked<0x00002aab7fbb83e8> (a java.io.BufferedInputStream)
>>>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>>> )
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>>> - locked<0x00002aac2cbf46c8> (a [B)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>>> 76)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
>>>> anner.java:115)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
>>>> va:255)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
>>>> va:394)
>>>> - locked<0x00002aad15b7da98> (a
>>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>>> :293)
>>>> - locked<0x00002aad15b7da98> (a
>>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>>> :114)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>>> HRegion.java:2344)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2301)
>>>> - locked<0x00002aabc7a3d3e8> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2317)
>>>> - locked<0x00002aabc7a3d3e8> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>>> va:1832)
>>>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
>> Source)
>>>> at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>>> Impl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:616)
>>>> at
>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>>> 9)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "IPC Server handler 4 on 60020" daemon prio=10 tid=0x000000000194a800
>>>> nid=0x5541 runnable [0x00000000442ff000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>>> - locked<0x00002aabbae85af8> (a sun.nio.ch.Util$1)
>>>> - locked<0x00002aabbae85b10> (a
>>>> java.util.Collections$UnmodifiableSet)
>>>> - locked<0x00002aaae379da00> (a
>> sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>>> thTimeout.java:332)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>>> 157)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>>> at
>>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>>> - locked<0x00002aac7ecf9bb8> (a java.io.BufferedInputStream)
>>>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>>> 50)
>>>> - locked<0x00002aaacf49d528> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>>> a:237)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>>> - locked<0x00002aaacf49d528> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>>> - locked<0x00002aaacf49d528> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>>> a:1899)
>>>> - locked<0x00002aabdc652620> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>>> )
>>>> - locked<0x00002aabdc652620> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at java.io.DataInputStream.read(DataInputStream.java:149)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>>> dRangeFileInputStream.java:105)
>>>> - locked<0x00002aac357f4350> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>>> at
>>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>>> - locked<0x00002aab9fc22a58> (a java.io.BufferedInputStream)
>>>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>>> )
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>>> - locked<0x00002aabdbefbe20> (a [B)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>>> 76)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>>>> ner.java:87)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>>> :82)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>>> :262)
>>>> - locked<0x00002aad97b3b7e8> (a
>>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>>> :114)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>>> HRegion.java:2344)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2301)
>>>> - locked<0x00002aab016d6618> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2317)
>>>> - locked<0x00002aab016d6618> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>>> va:1832)
>>>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
>> Source)
>>>> at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>>> Impl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:616)
>>>> at
>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>>> 9)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "IPC Server handler 3 on 60020" daemon prio=10 tid=0x00000000019c4000
>>>> nid=0x5540 waiting for monitor entry [0x00000000441ff000]
>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2317)
>>>> - waiting to lock<0x00002aab28d7de90> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>>> va:1832)
>>>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
>> Source)
>>>> at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>>> Impl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:616)
>>>> at
>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>>> 9)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "IPC Server handler 2 on 60020" daemon prio=10 tid=0x00000000019c2000
>>>> nid=0x553f runnable [0x00000000440fd000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>>> - locked<0x00002aaaeef929f8> (a sun.nio.ch.Util$1)
>>>> - locked<0x00002aaaeef92a10> (a
>>>> java.util.Collections$UnmodifiableSet)
>>>> - locked<0x00002aaaffd9ef38> (a
>> sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>>> thTimeout.java:332)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>>> 157)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>>> at
>>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>>> - locked<0x00002aab2ac42300> (a java.io.BufferedInputStream)
>>>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>>> 50)
>>>> - locked<0x00002aabc43bef18> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>>> a:237)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>>> - locked<0x00002aabc43bef18> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>>> - locked<0x00002aabc43bef18> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>>> a:1899)
>>>> - locked<0x00002aabd53c1d58> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>>> )
>>>> - locked<0x00002aabd53c1d58> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at java.io.DataInputStream.read(DataInputStream.java:149)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>>> dRangeFileInputStream.java:105)
>>>> - locked<0x00002aabb602a180> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>>> at
>>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>>> - locked<0x00002aab44c85f18> (a java.io.BufferedInputStream)
>>>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>>> )
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>>> - locked<0x00002aac0d2be0b0> (a [B)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>>> 76)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>>>> ner.java:87)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>>> :82)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>>> :262)
>>>> - locked<0x00002aab4eb7a680> (a
>>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>>> :114)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>>> HRegion.java:2344)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2301)
>>>> - locked<0x00002aabc7a3cc68> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2317)
>>>> - locked<0x00002aabc7a3cc68> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>>> va:1832)
>>>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
>> Source)
>>>> at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>>> Impl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:616)
>>>> at
>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>>> 9)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "IPC Server handler 1 on 60020" daemon prio=10 tid=0x0000000001535800
>>>> nid=0x553e runnable [0x0000000043ffc000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>>> - locked<0x00002aad28449198> (a sun.nio.ch.Util$1)
>>>> - locked<0x00002aad284491b0> (a
>>>> java.util.Collections$UnmodifiableSet)
>>>> - locked<0x00002aab3a682700> (a
>> sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>>> thTimeout.java:332)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>>> 157)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>>> at
>>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>>> - locked<0x00002aac5b5cbd28> (a java.io.BufferedInputStream)
>>>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>>> 50)
>>>> - locked<0x00002aabc43bd778> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>>> a:237)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>>> - locked<0x00002aabc43bd778> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>>> - locked<0x00002aabc43bd778> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>>> a:1899)
>>>> - locked<0x00002aac4d4f9b90> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>>> )
>>>> - locked<0x00002aac4d4f9b90> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at java.io.DataInputStream.read(DataInputStream.java:149)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>>> dRangeFileInputStream.java:105)
>>>> - locked<0x00002aac500ac780> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>>> at
>>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>>> - locked<0x00002aabe2211268> (a java.io.BufferedInputStream)
>>>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>>> )
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>>> - locked<0x00002aac537126f0> (a [B)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>>> 76)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>>>> ner.java:87)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>>> :82)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>>> :262)
>>>> - locked<0x00002aadb912fca8> (a
>>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>>> :114)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>>> HRegion.java:2344)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2301)
>>>> - locked<0x00002aadcba8fe98> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2317)
>>>> - locked<0x00002aadcba8fe98> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>>> va:1832)
>>>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
>> Source)
>>>> at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>>> Impl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:616)
>>>> at
>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>>> 9)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "IPC Server handler 0 on 60020" daemon prio=10 tid=0x0000000001534000
>>>> nid=0x553d waiting for monitor entry [0x0000000043efc000]
>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>>> java:2317)
>>>> - waiting to lock<0x00002aabc7a3d258> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>>> va:1832)
>>>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
>> Source)
>>>> at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>>> Impl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:616)
>>>> at
>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>>> 9)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "IPC Server listener on 60020" daemon prio=10 tid=0x00002aae4054e000
>>>> nid=0x553c runnable [0x0000000043dfb000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>>> - locked<0x00002aaabc18bdf8> (a sun.nio.ch.Util$1)
>>>> - locked<0x00002aaabc18bde0> (a
>>>> java.util.Collections$UnmodifiableSet)
>>>> - locked<0x00002aaabbb1ced8> (a
>> sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:41
>>>> 4)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "IPC Server Responder" daemon prio=10 tid=0x00002aae4053d800
>> nid=0x553b
>>>> runnable [0x0000000043cfa000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>>> - locked<0x00002aaabc2ceed8> (a sun.nio.ch.Util$1)
>>>> - locked<0x00002aaabc2ceef0> (a
>>>> java.util.Collections$UnmodifiableSet)
>>>> - locked<0x00002aaabbb33fa8> (a
>> sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:5
>>>> 88)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "Timer-0" daemon prio=10 tid=0x00002aae4053c800 nid=0x553a in
>>>> Object.wait() [0x0000000043bf9000]
>>>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> at java.util.TimerThread.mainLoop(Timer.java:531)
>>>> - locked<0x00002aaabd062850> (a java.util.TaskQueue)
>>>> at java.util.TimerThread.run(Timer.java:484)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "983053578@qtp-1891789590-1 - Acceptor0
>>>> SelectChannelConnector@0.0.0.0:60030" prio=10 tid=0x00002aae40381000
>>>> nid=0x5539 runnable [0x0000000043af8000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>>> - locked<0x00002aaabd07c460> (a sun.nio.ch.Util$1)
>>>> - locked<0x00002aaabd07c478> (a
>>>> java.util.Collections$UnmodifiableSet)
>>>> - locked<0x00002aaabbb26b78> (a
>> sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>>> at
>>>>
>> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.ja
>>>> va:498)
>>>> at
>>>> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
>>>> at
>>>>
>> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnect
>>>> or.java:124)
>>>> at
>>>>
>> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:
>>>> 708)
>>>> at
>>>>
>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java
>>>> :582)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "1148428095@qtp-1891789590-0" prio=10 tid=0x00002aae403d2000
>> nid=0x5538
>>>> in Object.wait() [0x00000000439f7000]
>>>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> at
>>>>
>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java
>>>> :626)
>>>> - locked<0x00002aaabbc4c8f0> (a
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "regionserver60020.leaseChecker" daemon prio=10
>> tid=0x00002aae402ee800
>>>> nid=0x5537 waiting for monitor entry [0x00000000438f6000]
>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.close(HRegion
>>>> .java:2409)
>>>> - waiting to lock<0x00002aadcba8fe98> (a
>>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer$ScannerListener.lease
>>>> Expired(HRegionServer.java:1892)
>>>> at
>>>> org.apache.hadoop.hbase.regionserver.Leases.run(Leases.java:99)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "regionserver60020.majorCompactionChecker" daemon prio=10
>>>> tid=0x00002aae402ec800 nid=0x5536 in Object.wait()
>> [0x00000000437f5000]
>>>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> - waiting on<0x00002aaabc1f7bf0> (a java.lang.Object)
>>>> at
>> org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
>>>> - locked<0x00002aaabc1f7bf0> (a java.lang.Object)
>>>> at org.apache.hadoop.hbase.Chore.run(Chore.java:74)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "regionserver60020.compactor" daemon prio=10 tid=0x00002aae402eb000
>>>> nid=0x5535 runnable [0x00000000436f4000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>>> - locked<0x00002aab26f462e0> (a sun.nio.ch.Util$1)
>>>> - locked<0x00002aab26f462c8> (a
>>>> java.util.Collections$UnmodifiableSet)
>>>> - locked<0x00002aaaeba1ac80> (a
>> sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>>> thTimeout.java:332)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>>> 157)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>>> at
>>>>
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>>> at
>>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>>> - locked<0x00002aab44c85258> (a java.io.BufferedInputStream)
>>>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>>> 50)
>>>> - locked<0x00002aaac6caf140> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>>> a:237)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>>> at
>>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>>> - locked<0x00002aaac6caf140> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>>> - locked<0x00002aaac6caf140> (a
>>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>>> a:1899)
>>>> - locked<0x00002aaabd63fbb0> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>>> )
>>>> - locked<0x00002aaabd63fbb0> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>>> at java.io.DataInputStream.read(DataInputStream.java:149)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>>> dRangeFileInputStream.java:105)
>>>> - locked<0x00002aab21b4e148> (a
>>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>>> at
>>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>>> - locked<0x00002aaaaedc39d0> (a java.io.BufferedInputStream)
>>>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>>> )
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>>> - locked<0x00002aab458cc280> (a [B)
>>>> at
>>>>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>>> 76)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>>>> ner.java:87)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>>> :82)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>>> :297)
>>>> - locked<0x00002aaace390ec0> (a
>>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>>> :326)
>>>> - locked<0x00002aaace390ec0> (a
>>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>>> at
>>>> org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:927)
>>>> at
>>>> org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:733)
>>>> - locked<0x00002aaacd20e840> (a java.lang.Object)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:
>>>> 769)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:
>>>> 714)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplit
>>>> Thread.java:81)
>>>>
>>>> Locked ownable synchronizers:
>>>> -<0x00002aaabce779f8> (a
>>>> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>>>>
>>>> "regionserver60020.cacheFlusher" daemon prio=10
>> tid=0x00002aae402d1800
>>>> nid=0x5534 waiting on condition [0x00000000435f3000]
>>>> java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for<0x00002aaabc241a10> (a
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>>
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>>>> at
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>>> aitNanos(AbstractQueuedSynchronizer.java:2081)
>>>> at java.util.concurrent.DelayQueue.poll(DelayQueue.java:230)
>>>> at java.util.concurrent.DelayQueue.poll(DelayQueue.java:68)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher
>>>> .java:216)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "regionserver60020.logRoller" daemon prio=10 tid=0x00002aae402d1000
>>>> nid=0x5533 in Object.wait() [0x00000000434f2000]
>>>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> at
>>>> org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:78)
>>>> - locked<0x00002aaabcec18e0> (a
>>>> java.util.concurrent.atomic.AtomicBoolean)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "Timer thread for monitoring jvm" daemon prio=10
>> tid=0x00002aae402b4000
>>>> nid=0x5532 in Object.wait() [0x00000000433f1000]
>>>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> at java.util.TimerThread.mainLoop(Timer.java:531)
>>>> - locked<0x00002aaabcedce48> (a java.util.TaskQueue)
>>>> at java.util.TimerThread.run(Timer.java:484)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "Timer thread for monitoring hbase" daemon prio=10
>>>> tid=0x00002aae402b2800 nid=0x5531 in Object.wait()
>> [0x00000000432f0000]
>>>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> at java.util.TimerThread.mainLoop(Timer.java:531)
>>>> - locked<0x00002aaabceca148> (a java.util.TaskQueue)
>>>> at java.util.TimerThread.run(Timer.java:484)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "regionserver60020.logSyncer" daemon prio=10 tid=0x00002aae402b1000
>>>> nid=0x5530 waiting on condition [0x00000000431ef000]
>>>> java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>> at java.lang.Thread.sleep(Native Method)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:96
>>>> 3)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "LeaseChecker" daemon prio=10 tid=0x00002aae4019a800 nid=0x552f
>>>> sleeping[0x00000000430ee000]
>>>> java.lang.Thread.State: TIMED_WAITING (sleeping)
>>>> at java.lang.Thread.sleep(Native Method)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1167)
>>>> at java.lang.Thread.run(Thread.java:636)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "IPC Client (47) connection to
>>>> doop10.dt.sv4.decarta.com/10.241.8.230:60000 from an unknown user"
>>>> daemon prio=10 tid=0x00000000015c9800 nid=0x5528 in Object.wait()
>>>> [0x0000000042cea000]
>>>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.waitForWork(HBaseClie
>>>> nt.java:431)
>>>> - locked<0x00002aaabc24dfc0> (a
>>>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:
>>>> 476)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "DestroyJavaVM" prio=10 tid=0x00002aae4015c800 nid=0x54fa waiting on
>>>> condition [0x0000000000000000]
>>>> java.lang.Thread.State: RUNNABLE
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "regionserver60020-EventThread" daemon prio=10 tid=0x00000000015c5000
>>>> nid=0x5526 waiting on condition [0x0000000042be9000]
>>>> java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for<0x00002aaabc241a60> (a
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>>> at
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>>> ait(AbstractQueuedSynchronizer.java:2043)
>>>> at
>>>>
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>>>> 86)
>>>> at
>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "regionserver60020-SendThread(doop3.dt.sv4.decarta.com:2181)" daemon
>>>> prio=10 tid=0x0000000001486800 nid=0x5525 runnable
>> [0x0000000042ae8000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>>> - locked<0x00002aaabbb3e538> (a sun.nio.ch.Util$1)
>>>> - locked<0x00002aaabbb3e550> (a
>>>> java.util.Collections$UnmodifiableSet)
>>>> - locked<0x00002aaabbb1db08> (a
>> sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>>> at
>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "regionserver60020" prio=10 tid=0x00002aae40187000 nid=0x5523 waiting
>> on
>>>> condition [0x000000004028d000]
>>>> java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for<0x00002aaabc2faea0> (a
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>>
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>>>> at
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>>> aitNanos(AbstractQueuedSynchronizer.java:2081)
>>>> at
>>>>
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:4
>>>> 23)
>>>> at
>>>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.jav
>>>> a:621)
>>>> at java.lang.Thread.run(Thread.java:636)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "Timer thread for monitoring rpc" daemon prio=10
>> tid=0x00002aae40135800
>>>> nid=0x5522 in Object.wait() [0x00000000429e7000]
>>>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> at java.util.TimerThread.mainLoop(Timer.java:531)
>>>> - locked<0x00002aaabbb3cb08> (a java.util.TaskQueue)
>>>> at java.util.TimerThread.run(Timer.java:484)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "IPC Reader 9 on port 60020" prio=10 tid=0x00002aae40133000
>> nid=0x5521
>>>> runnable [0x00000000428e6000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>>> - locked<0x00002aaabbb010a8> (a sun.nio.ch.Util$1)
>>>> - locked<0x00002aaabbb010c0> (a
>>>> java.util.Collections$UnmodifiableSet)
>>>> - locked<0x00002aaabbb1e198> (a
>> sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
>>>> java:305)
>>>> - locked<0x00002aaabbb250e8> (a
>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>>> a:1110)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>>> va:603)
>>>> at java.lang.Thread.run(Thread.java:636)
>>>>
>>>> Locked ownable synchronizers:
>>>> -<0x00002aaabbb4b008> (a
>>>> java.util.concurrent.ThreadPoolExecutor$Worker)
>>>>
>>>> "IPC Reader 8 on port 60020" prio=10 tid=0x00002aae40118000
>> nid=0x5520
>>>> runnable [0x00000000427e5000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>>> - locked<0x00002aaabbb16118> (a sun.nio.ch.Util$1)
>>>> - locked<0x00002aaabbb16130> (a
>>>> java.util.Collections$UnmodifiableSet)
>>>> - locked<0x00002aaabbb144f8> (a
>> sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
>>>> java:305)
>>>> - locked<0x00002aaabbb30308> (a
>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>>> a:1110)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>>> va:603)
>>>> at java.lang.Thread.run(Thread.java:636)
>>>>
>>>> Locked ownable synchronizers:
>>>> -<0x00002aaabbb32888> (a
>>>> java.util.concurrent.ThreadPoolExecutor$Worker)
>>>>
>>>> "IPC Reader 7 on port 60020" prio=10 tid=0x00002aae400fd000
>> nid=0x551f
>>>> runnable [0x00000000426e4000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>>> - locked<0x00002aaabbb2a228> (a sun.nio.ch.Util$1)
>>>> - locked<0x00002aaabbb2a240> (a
>>>> java.util.Collections$UnmodifiableSet)
>>>> - locked<0x00002aaabbb28ab8> (a
>> sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
>>>> java:305)
>>>> - locked<0x00002aaabbb2a6d8> (a
>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>>> a:1110)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>>> va:603)
>>>> at java.lang.Thread.run(Thread.java:636)
>>>>
>>>> Locked ownable synchronizers:
>>>> -<0x00002aaabbb4a068> (a
>>>> java.util.concurrent.ThreadPoolExecutor$Worker)
>>>>
>>>> "IPC Reader 6 on port 60020" prio=10 tid=0x00002aae400e2800
>> nid=0x551e
>>>> runnable [0x00000000425e3000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>>> - locked<0x00002aaabbb27028> (a sun.nio.ch.Util$1)
>>>> - locked<0x00002aaabbb27040> (a
>>>> java.util.Collections$UnmodifiableSet)
>>>> - locked<0x00002aaabbb258b8> (a
>> sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
>>>> java:305)
>>>> - locked<0x00002aaabbb282e8> (a
>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>>> a:1110)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>>> va:603)
>>>> at java.lang.Thread.run(Thread.java:636)
>>>>
>>>> Locked ownable synchronizers:
>>>> -<0x00002aaabbb4fc98> (a
>>>> java.util.concurrent.ThreadPoolExecutor$Worker)
>>>>
>>>> "IPC Reader 5 on port 60020" prio=10 tid=0x00002aae400c7800
>> nid=0x551d
>>>> runnable [0x00000000413e5000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>>> - locked<0x00002aaabbb2ea08> (a sun.nio.ch.Util$1)
>>>> - locked<0x00002aaabbb2ea20> (a
>>>> java.util.Collections$UnmodifiableSet)
>>>> - locked<0x00002aaabbb2d298> (a
>> sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
>>>> java:305)
>>>> - locked<0x00002aaabbb50c38> (a
>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>>> a:1110)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>>> va:603)
>>>> at java.lang.Thread.run(Thread.java:636)
>>>>
>>>> Locked ownable synchronizers:
>>>> -<0x00002aaabbb523a8> (a
>>>> java.util.concurrent.ThreadPoolExecutor$Worker)
>>>>
>>>> "IPC Reader 4 on port 60020" prio=10 tid=0x00002aae400ac800
>> nid=0x551c
>>>> runnable [0x00000000412e4000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>>> - locked<0x00002aaabbb2aea8> (a sun.nio.ch.Util$1)
>>>> - locked<0x00002aaabbb2aec0> (a
>>>> java.util.Collections$UnmodifiableSet)
>>>> - locked<0x00002aaabbb1ca28> (a
>> sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
>>>> java:305)
>>>> - locked<0x00002aaabbb2c168> (a
>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>>> a:1110)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>>> va:603)
>>>> at java.lang.Thread.run(Thread.java:636)
>>>>
>>>> Locked ownable synchronizers:
>>>> -<0x00002aaabbb48128> (a
>>>> java.util.concurrent.ThreadPoolExecutor$Worker)
>>>>
>>>> "IPC Reader 3 on port 60020" prio=10 tid=0x00002aae40091800
>> nid=0x551b
>>>> runnable [0x00000000411e3000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>>> - locked<0x00002aaabbb15c68> (a sun.nio.ch.Util$1)
>>>> - locked<0x00002aaabbb15c80> (a
>>>> java.util.Collections$UnmodifiableSet)
>>>> - locked<0x00002aaabbb12d88> (a
>> sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
>>>> java:305)
>>>> - locked<0x00002aaabbb165c8> (a
>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>>> a:1110)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>>> va:603)
>>>> at java.lang.Thread.run(Thread.java:636)
>>>>
>>>> Locked ownable synchronizers:
>>>> -<0x00002aaabbb30ad8> (a
>>>> java.util.concurrent.ThreadPoolExecutor$Worker)
>>>>
>>>> "IPC Reader 2 on port 60020" prio=10 tid=0x00002aae40077000
>> nid=0x551a
>>>> runnable [0x00000000410e2000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>>> - locked<0x00002aaabbb3d148> (a sun.nio.ch.Util$1)
>>>> - locked<0x00002aaabbb3d160> (a
>>>> java.util.Collections$UnmodifiableSet)
>>>> - locked<0x00002aaabbb33828> (a
>> sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
>>>> java:305)
>>>> - locked<0x00002aaabbb3ebd8> (a
>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>>> a:1110)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>>> va:603)
>>>> at java.lang.Thread.run(Thread.java:636)
>>>>
>>>> Locked ownable synchronizers:
>>>> -<0x00002aaabbb40028> (a
>>>> java.util.concurrent.ThreadPoolExecutor$Worker)
>>>>
>>>> "IPC Reader 1 on port 60020" prio=10 tid=0x00002aae4005d000
>> nid=0x5519
>>>> runnable [0x0000000040fe1000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>>> - locked<0x00002aaabbb15c98> (a sun.nio.ch.Util$1)
>>>> - locked<0x00002aaabbb15cb0> (a
>>>> java.util.Collections$UnmodifiableSet)
>>>> - locked<0x00002aaabbb12e00> (a
>> sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
>>>> java:305)
>>>> - locked<0x00002aaabbb165f0> (a
>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>>> a:1110)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>>> va:603)
>>>> at java.lang.Thread.run(Thread.java:636)
>>>>
>>>> Locked ownable synchronizers:
>>>> -<0x00002aaabbb30b28> (a
>>>> java.util.concurrent.ThreadPoolExecutor$Worker)
>>>>
>>>> "IPC Reader 0 on port 60020" prio=10 tid=0x00002aae40003000
>> nid=0x5518
>>>> runnable [0x0000000040ee0000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>>> - locked<0x00002aaabbb16148> (a sun.nio.ch.Util$1)
>>>> - locked<0x00002aaabbb16160> (a
>>>> java.util.Collections$UnmodifiableSet)
>>>> - locked<0x00002aaabbb14570> (a
>> sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>>>> at
>>>>
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
>>>> java:305)
>>>> - locked<0x00002aaabbb30330> (a
>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>>> a:1110)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>>> va:603)
>>>> at java.lang.Thread.run(Thread.java:636)
>>>>
>>>> Locked ownable synchronizers:
>>>> -<0x00002aaabbb328d8> (a
>>>> java.util.concurrent.ThreadPoolExecutor$Worker)
>>>>
>>>> "main-EventThread" daemon prio=10 tid=0x00002aae3c1b6000 nid=0x5513
>>>> waiting on condition [0x00000000424e2000]
>>>> java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for<0x00002aaabbb16618> (a
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>>> at
>>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>>> ait(AbstractQueuedSynchronizer.java:2043)
>>>> at
>>>>
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>>>> 86)
>>>> at
>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "main-SendThread(doop7.dt.sv4.decarta.com:2181)" daemon prio=10
>>>> tid=0x00002aae3c11b000 nid=0x5512 runnable [0x00000000423e1000]
>>>> java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>>> at
>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>>> at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>>> - locked<0x00002aaabbb16178> (a sun.nio.ch.Util$1)
>>>> - locked<0x00002aaabbb16190> (a
>>>> java.util.Collections$UnmodifiableSet)
>>>> - locked<0x00002aaabbb145e8> (a
>> sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>>> at
>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "Low Memory Detector" daemon prio=10 tid=0x000000000126c800
>> nid=0x5510
>>>> runnable [0x0000000000000000]
>>>> java.lang.Thread.State: RUNNABLE
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "CompilerThread1" daemon prio=10 tid=0x000000000126a800 nid=0x550f
>>>> waiting on condition [0x0000000000000000]
>>>> java.lang.Thread.State: RUNNABLE
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "CompilerThread0" daemon prio=10 tid=0x0000000001266000 nid=0x550e
>>>> waiting on condition [0x0000000000000000]
>>>> java.lang.Thread.State: RUNNABLE
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "Signal Dispatcher" daemon prio=10 tid=0x0000000001264000 nid=0x550d
>>>> runnable [0x0000000000000000]
>>>> java.lang.Thread.State: RUNNABLE
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "Surrogate Locker Thread (CMS)" daemon prio=10 tid=0x0000000001262000
>>>> nid=0x550c waiting on condition [0x0000000000000000]
>>>> java.lang.Thread.State: RUNNABLE
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "Finalizer" daemon prio=10 tid=0x000000000123b800 nid=0x550b in
>>>> Object.wait() [0x00000000422e0000]
>>>> java.lang.Thread.State: WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> at
>> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:133)
>>>> - locked<0x00002aaabbb15cf8> (a
>>>> java.lang.ref.ReferenceQueue$Lock)
>>>> at
>> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:149)
>>>> at
>>>> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "Reference Handler" daemon prio=10 tid=0x0000000001239800 nid=0x550a
>> in
>>>> Object.wait() [0x00000000421df000]
>>>> java.lang.Thread.State: WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> at java.lang.Object.wait(Object.java:502)
>>>> at
>>>> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
>>>> - locked<0x00002aaabbb16d98> (a
>> java.lang.ref.Reference$Lock)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "VM Thread" prio=10 tid=0x0000000001234800 nid=0x5509 runnable
>>>>
>>>> "Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x0000000000fe9000
>>>> nid=0x54fb runnable
>>>>
>>>> "Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x0000000000feb000
>>>> nid=0x54fc runnable
>>>>
>>>> "Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x0000000000fec800
>>>> nid=0x54fd runnable
>>>>
>>>> "Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x0000000000fee800
>>>> nid=0x54fe runnable
>>>>
>>>> "Gang worker#4 (Parallel GC Threads)" prio=10 tid=0x0000000000ff0800
>>>> nid=0x54ff runnable
>>>>
>>>> "Gang worker#5 (Parallel GC Threads)" prio=10 tid=0x0000000000ff2000
>>>> nid=0x5500 runnable
>>>>
>>>> "Gang worker#6 (Parallel GC Threads)" prio=10 tid=0x0000000000ff4000
>>>> nid=0x5501 runnable
>>>>
>>>> "Gang worker#7 (Parallel GC Threads)" prio=10 tid=0x0000000000ff6000
>>>> nid=0x5502 runnable
>>>>
>>>> "Gang worker#8 (Parallel GC Threads)" prio=10 tid=0x0000000000ff7800
>>>> nid=0x5503 runnable
>>>>
>>>> "Gang worker#9 (Parallel GC Threads)" prio=10 tid=0x0000000000ff9800
>>>> nid=0x5504 runnable
>>>>
>>>> "Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00000000010f4000
>>>> nid=0x5508 runnable
>>>> "Gang worker#0 (Parallel CMS Threads)" prio=10 tid=0x00000000010ee000
>>>> nid=0x5505 runnable
>>>>
>>>> "Gang worker#1 (Parallel CMS Threads)" prio=10 tid=0x00000000010f0000
>>>> nid=0x5506 runnable
>>>>
>>>> "Gang worker#2 (Parallel CMS Threads)" prio=10 tid=0x00000000010f2000
>>>> nid=0x5507 runnable
>>>>
>>>> "VM Periodic Task Thread" prio=10 tid=0x000000000126f800 nid=0x5511
>>>> waiting on condition
>>>>
>>>> JNI global references: 906
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of
>>>> Stack
>>>> Sent: Sunday, September 11, 2011 2:27 PM
>>>> To: user@hbase.apache.org
>>>> Subject: Re: scanner deadlock?
>>>>
>>>> 0.90.4 fixes two deadlocks (HBASE-4101 and HBASE-4077). Since then,
>>>> there is HBASE-4367 (Which has a posted patch).
>>>>
>>>> Below sounds like slowness. Can you thread dump the particular
>>>> regionserver and see what its up to? Is there other loading on the
>>>> system at the time? For example, loading on hdfs? Anything in the
>>>> hdfs logs for the datanode running beside the slow regionserver?
>>>>
>>>> St.Ack
>>>>
>>>> On Sat, Sep 10, 2011 at 5:50 PM, Geoff Hendrey<gh...@decarta.com>
>>>> wrote:
>>>>> Hi all -
>>>>>
>>>>>
>>>>>
>>>>> I'm still dealing with the saga of ScannerTimeoutException,
>>>>> UnknownScannerException, etc. I rewrote my code, in the hope that
>>>> simply
>>>>> a different approach and some different code paths might yield
>> better
>>>>> results. No change. I tried many variations (caching 1 row vs
>> caching
>>>>> many rows, changing the regionserver's lease, increasing the number
>> of
>>>>> allowed zookeeper connections, etc). I created a fresh table on the
>>>>> thought that maybe there was some problem with the table...no
>> change.
>>>>>
>>>>>
>>>>>
>>>>> I am dealing with what appears to be some sort of scanner deadlock.
>> I
>>>>> have a total order partitioned mapreduce job. In the reducer, as
>> long
>>>> as
>>>>> I use just one reducer, the task finishes quickly. But as soon as
>> more
>>>>> than one reducer opens a scanner, the tasks procede in what I would
>>>> call
>>>>> a "jittery" lock step. They both are able to do ResultScanner.next()
>> a
>>>>> few times, but then the call to next() freezes for a long period,
>> and
>>>>> ScannerTimeoutException. I catch the exception, and get a new
>>>>> ResultScanner, and the pattern repeats. Jittery lockstep consisting
>> of
>>>>> being able to get a few successful next() calls, then a lock up.
>>>>>
>>>>>
>>>>>
>>>>> I have not yet tried upgrading from 90.1 to higher. Nor have I tried
>>>>> tsuna's async client. Can anyone think of anything else I can try to
>>>>> resolve this? I've sunk quite a few late nights into this, and would
>>>> be
>>>>> very excited to find a solution.
>>>>>
>>>>>
>>>>>
>>>>> -geoff
>>>>>
>>>>>
>>>>
>>
>> --
>> Eric
>> http://about.echarles.net
>>
--
Eric
http://about.echarles.net
RE: scanner deadlock?
Posted by Geoff Hendrey <gh...@decarta.com>.
Hi - Will definitely switch to hotspot jre (as opposed to openjdk).
WRT scanner: I made sure scanner caching was set to 1, but I also played
with various levels of caching (up to 100). No impact on the problem.
WRT Put: write buffer size is 10*1024*1024 (10 MB). Each row is about
750KB, so I expect to see batching of order of 10 rows.
We are going to upgrade to 90.3 or 90.4. I am running my MR job with all
the setting changes recommended by Stack (except new JRE). Afterwards, I
will run again with the new JRE for comparison (want to change a few
things at a time, and test). After that we'll upgrade hbase and run our
tests again. And of course, I will keep the list posted on findings.
Guys: THANK YOU for all the help.
-geoff
-----Original Message-----
From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of
Jean-Daniel Cryans
Sent: Monday, September 12, 2011 11:44 AM
To: user@hbase.apache.org
Subject: Re: scanner deadlock?
> I thought that as long as I specified neither -client nor -server,
that
> Server Class detection would automatically invoke the "-server"
option.
>
>
http://download.oracle.com/javase/6/docs/technotes/guides/vm/server-clas
> s.html
>
> We are running 12-core AMD Opteron which is AMD64, so according to the
> guide above, -server is selected automatically. Please let me know if
> I've misunderstood this. We *definitely* want to be running hotspot!
It's two different JVMs, not a matter of using -client or -server
(which are just different configurations). What you are running is:
http://openjdk.java.net/
What most people run is:
http://www.oracle.com/us/technologies/java/index.html
>
> Regarding GC: we are generating GC logs for namenode, datanode, master
> and regionserver. We do see long GC from time to time. In fact, I
played
> with the mslab option, but didn't find significant improvement. We've
> seen times on the order of a minute in these logs, and have found no
way
> around it (spent countless days and nights experimenting with
different
> GC parameters, mslab, different heap sizes, etc).
Sometimes it's just a matter of how much data you have in flight.
That's why I mentioned scanner pre-caching (set via Scan.setCaching),
because it can potentially load a lot of rows into the RS's heap. More
concurrent scanners means also more data loaded into memory.
Are you also inserting at the same time? What's your write buffer size?
The discussion in this jira could be relevant:
https://issues.apache.org/jira/browse/HBASE-3813
A temporary fix got committed in 0.90.3 to make
ipc.server.max.queue.size configurable.
J-D
RE: scanner deadlock?
Posted by Sandy Pratt <pr...@adobe.com>.
Thanks for the info Andy.
I'm using this:
java version "1.6.0_20"
OpenJDK Runtime Environment (IcedTea6 1.9.8) (rhel-1.22.1.9.8.el5_6-x86_64)
OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
On CentOS 5.6.
I switched from (Sun) 1.6.0_u14 as an experiment a while back and I've never had reason to switch back. For reference, it's a 12-node cluster with ~6.4 billion HBase records, ~8-10 million inserts and a number of Hive queries per day. Not big as far as HBase clusters go, but it does see some use.
Thanks,
Sandy
> -----Original Message-----
> From: Andrew Purtell [mailto:apurtell@apache.org]
> Sent: Monday, September 12, 2011 15:45
> To: user@hbase.apache.org
> Subject: Re: scanner deadlock?
>
> > From: Sandy Pratt <pr...@adobe.com>
> > TLDR: OpenJDK ~= Oracle JDK, so why not use it?
>
>
> This advice is given out of an abundance of caution. Some have been burned
> in production by bad JVM versions in the past. Oracle's 1.6.0_u18 is a
> particularly egregious example, it will segfault all over the place under
> load. 1.6.0_u14 was widely used once. Now u21+ seems a good option given
> the JIT and GC bugfixes that went in. IIRC, with u20 or previous you need to
> enable -XX:+UseMembar to avoid a JIT bug that will cause object monitors to
> miss wakeups every once in a while. I currently use Oracle 1.6.0_u26.
>
> Given the history of JVM issues, we do not automatically trust any given
> version. None of the committers run HBase in production on OpenJDK that I
> know of, so its suitability is unknown. I'm glad to hear it works for you. What
> specific version are you using?
>
> Best regards,
>
>
> Â Â - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein (via
> Tom White)
RE: scanner deadlock?
Posted by Geoff Hendrey <gh...@decarta.com>.
Agree: 1.6.0_u18 was totally broken. We definitely saw that too.
-geoff
-----Original Message-----
From: Andrew Purtell [mailto:apurtell@apache.org]
Sent: Monday, September 12, 2011 3:45 PM
To: user@hbase.apache.org
Subject: Re: scanner deadlock?
> From: Sandy Pratt <pr...@adobe.com>
> TLDR: OpenJDK ~= Oracle JDK, so why not use it?
This advice is given out of an abundance of caution. Some have been burned in production by bad JVM versions in the past. Oracle's 1.6.0_u18 is a particularly egregious example, it will segfault all over the place under load. 1.6.0_u14 was widely used once. Now u21+ seems a good option given the JIT and GC bugfixes that went in. IIRC, with u20 or previous you need to enable -XX:+UseMembar to avoid a JIT bug that will cause object monitors to miss wakeups every once in a while. I currently use Oracle 1.6.0_u26.
Given the history of JVM issues, we do not automatically trust any given version. None of the committers run HBase in production on OpenJDK that I know of, so its suitability is unknown. I'm glad to hear it works for you. What specific version are you using?
Best regards,
  - Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
Re: scanner deadlock?
Posted by Andrew Purtell <ap...@apache.org>.
> From: Sandy Pratt <pr...@adobe.com>
> TLDR: OpenJDK ~= Oracle JDK, so why not use it?
This advice is given out of an abundance of caution. Some have been burned in production by bad JVM versions in the past. Oracle's 1.6.0_u18 is a particularly egregious example, it will segfault all over the place under load. 1.6.0_u14 was widely used once. Now u21+ seems a good option given the JIT and GC bugfixes that went in. IIRC, with u20 or previous you need to enable -XX:+UseMembar to avoid a JIT bug that will cause object monitors to miss wakeups every once in a while. I currently use Oracle 1.6.0_u26.
Given the history of JVM issues, we do not automatically trust any given version. None of the committers run HBase in production on OpenJDK that I know of, so its suitability is unknown. I'm glad to hear it works for you. What specific version are you using?
Best regards,
  - Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
Re: scanner deadlock?
Posted by Stack <st...@duboce.net>.
On Tue, Sep 13, 2011 at 10:56 PM, Geoff Hendrey <gh...@decarta.com> wrote:
> Anything in its logs when regionserver slows down?
>
> ANSWER: Yes. I see ScannerTimeoutException, and unknown scanner, and then ClosedChannelException. Stack trace shows the ClosedChannelException occurs when the server tries to write the response to the scanner. This seems like a bug to me. Once you close the channel you cannot write it, no? If you try to write it after you close it, you will get ClosedChannelException.
>
I meant to say in the accompanying datanode logs.
On the CCE, yes. We're trying to write the response. You have set
down the timeout so client goes away faster. Only way to figure
socket gone is to try to use it. Hence CCE.
> ANSWER: Nope, no long pauses at all. I've periodically run a few greps with a regex to try to find pauses one second or longer, and I haven't seen any of late. HOWEVER, one thing I don't understand at all is why ganglia reports HUGE gc pauses (like 1000 seconds!). But in reality I can *never* find such a pause in the GC log. Is there any known issue with ganglia graphs being on the wrong vertical scale for GC pauses? I know that sounds odd, but I just can't correlate the Ganglia graphs for GC to reality.
>
Why don't you send us some logs that span a slow down. GC, RS, and
DN. Send jstacks too.
St.Ack
Re: scanner deadlock?
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Yeah like Stack said, the ClosedChannelException is how we figure the
client is gone. As you have a 60s timeout on the RPC call the client
_will_ go away (and possibly come right back in through another
handler) when a call takes longer than that. One of my theories was
that in your case if a region server slowed down it would start piling
up calls in the queues, some of them would timeout, so the client
would come right back with the same request, making the whole
situation worse.
When you did set the config that I told you about, you still got the
CCEs? Meaning that the calls were still slow? If so, then the issue is
elsewhere and this proves it.
Like Stack, I think we'll have to be fed more data about your system :)
J-D
On Wed, Sep 14, 2011 at 9:32 AM, Geoff Hendrey <gh...@decarta.com> wrote:
> I've already been able to replicate the problem using just two reducers,
> on a completely fresh table. So it seemed to me when I did that the
> problem was independent of the number of reducers...
>
> -----Original Message-----
> From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of
> Stack
> Sent: Wednesday, September 14, 2011 8:47 AM
> To: user@hbase.apache.org
> Subject: Re: scanner deadlock?
>
> On Wed, Sep 14, 2011 at 8:42 AM, Geoff Hendrey <gh...@decarta.com>
> wrote:
>> 17 MR nodes, 8 reducers per machine = 138 concurrent reducers.
>> (machines are 12-core, and I've found 8 reducers with 1GB allocated
> heap to be a happy medium that doesn't freeze out the data nodes or the
> region servers - or so I think :-).
>>
>
> Are you swapping at all?
>
> What if you restored your config. to something sane -- 100 handlers
> with queue size of 10, default timeout -- with 1/4 of the reducers?
>
> What does this MR job do?
>
> St.Ack
>
RE: scanner deadlock?
Posted by Geoff Hendrey <gh...@decarta.com>.
I've already been able to replicate the problem using just two reducers,
on a completely fresh table. So it seemed to me when I did that the
problem was independent of the number of reducers...
-----Original Message-----
From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of
Stack
Sent: Wednesday, September 14, 2011 8:47 AM
To: user@hbase.apache.org
Subject: Re: scanner deadlock?
On Wed, Sep 14, 2011 at 8:42 AM, Geoff Hendrey <gh...@decarta.com>
wrote:
> 17 MR nodes, 8 reducers per machine = 138 concurrent reducers.
> (machines are 12-core, and I've found 8 reducers with 1GB allocated
heap to be a happy medium that doesn't freeze out the data nodes or the
region servers - or so I think :-).
>
Are you swapping at all?
What if you restored your config. to something sane -- 100 handlers
with queue size of 10, default timeout -- with 1/4 of the reducers?
What does this MR job do?
St.Ack
Re: scanner deadlock?
Posted by Stack <st...@duboce.net>.
On Wed, Sep 14, 2011 at 8:42 AM, Geoff Hendrey <gh...@decarta.com> wrote:
> 17 MR nodes, 8 reducers per machine = 138 concurrent reducers.
> (machines are 12-core, and I've found 8 reducers with 1GB allocated heap to be a happy medium that doesn't freeze out the data nodes or the region servers - or so I think :-).
>
Are you swapping at all?
What if you restored your config. to something sane -- 100 handlers
with queue size of 10, default timeout -- with 1/4 of the reducers?
What does this MR job do?
St.Ack
RE: scanner deadlock?
Posted by Geoff Hendrey <gh...@decarta.com>.
17 MR nodes, 8 reducers per machine = 138 concurrent reducers.
(machines are 12-core, and I've found 8 reducers with 1GB allocated heap to be a happy medium that doesn't freeze out the data nodes or the region servers - or so I think :-).
-geoff
-----Original Message-----
From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of Stack
Sent: Wednesday, September 14, 2011 8:39 AM
To: user@hbase.apache.org
Subject: Re: scanner deadlock?
Also, how many nodes, and how many reducers.
St.Ack
On Wed, Sep 14, 2011 at 8:38 AM, Stack <st...@duboce.net> wrote:
> Yeah. Â Ten handlers and no queue for the RPCs, it'll just reject calls
> when its > handler count.
>
> Also send a listing of your hbase rootdir: hadoop fs -lsr /hbase
>
> St.Ack
>
> On Tue, Sep 13, 2011 at 11:40 PM, Geoff Hendrey <gh...@decarta.com> wrote:
>> As expected, J-D's suggestion basically causes the system to collapse almost immediately. All the region servers show stack traces such as these:
>>
>> 2011-09-13 23:34:43,151 WARN Â [IPC Reader 0 on port 60020] ipc.HBaseServer$Listener(526): IPC Server listener on 60020: readAndProcess threw exception java.io.IOException: Connection reset by peer. Count of bytes read: 0
>> java.io.IOException: Connection reset by peer
>> Â Â Â Â at sun.nio.ch.FileDispatcher.read0(Native Method)
>> Â Â Â Â at sun.nio.ch.SocketDispatcher.read(Unknown Source)
>> Â Â Â Â at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
>> Â Â Â Â at sun.nio.ch.IOUtil.read(Unknown Source)
>> Â Â Â Â at sun.nio.ch.SocketChannelImpl.read(Unknown Source)
>> Â Â Â Â at org.apache.hadoop.hbase.ipc.HBaseServer.channelRead(HBaseServer.java:1359)
>> Â Â Â Â at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:900)
>> Â Â Â Â at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:522)
>> Â Â Â Â at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:316)
>> Â Â Â Â at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>> Â Â Â Â at java.util.concurrent.ThreadPoolExe
>>
>>
>> 2011-09-13 23:34:43,293 WARN Â [IPC Server handler 0 on 60020] ipc.HBaseServer$Handler(1100): IPC Server handler 0 on 60020 caught: java.nio.channels.ClosedChannelException
>> Â Â Â Â at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(Unknown Source)
>> Â Â Â Â at sun.nio.ch.SocketChannelImpl.write(Unknown Source)
>> Â Â Â Â at org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1341)
>> Â Â Â Â at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:727)
>> Â Â Â Â at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:792)
>> Â Â Â Â at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1083)
>>
>> -----Original Message-----
>> From: Geoff Hendrey [mailto:ghendrey@decarta.com]
>> Sent: Tuesday, September 13, 2011 10:56 PM
>> To: user@hbase.apache.org
>> Cc: Andrew Purtell; Tony Wang; Rohit Nigam; Parmod Mehta; James Ladd
>> Subject: RE: scanner deadlock?
>>
>> answers below
>>
>> -----Original Message-----
>> From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of Stack
>> Sent: Tuesday, September 13, 2011 10:41 PM
>> To: user@hbase.apache.org
>> Cc: Andrew Purtell; Tony Wang; Rohit Nigam; Parmod Mehta; James Ladd
>> Subject: Re: scanner deadlock?
>>
>> On Tue, Sep 13, 2011 at 10:25 PM, Geoff Hendrey <gh...@decarta.com> wrote:
>>> I've upgraded to HotSpot 64 Bit Server VM, with HBase 90.4 and all recommended config changes (100 region server handlers, mslab enabled, etc). No change, if anything it dies faster. Count of sockets in CLOSE_WAIT on 50010 increases linearly. I logged netstat from a random node in the cluster, periodically. Then dumped the output into excel using a pivot table to look at a behavior of TCP. Number of connections from the given node to others on 50010 was relatively uniform (no hotspot). Connections on 50010 from given node to *self* was much way higher than to other nodes, but that's probably a good thing. My guess is it's HBase leveraging locality of files for the region server. Just a guess.
>>>
>> Yes. Â You have good locality. Â So maybe you are not bound up on a
>> single network resource.
>>
>> So when you jstack and you see that regionserver has its threads all
>> stuck in next -- are they? -- then we are likely going to the local
>> datanode.
>>
>> ANSWER: yes, I believe they are stuck in next. I can see from logs on the client that the call to next periodically takes very long to return. You had earlier commented that my 5 seconds regionserver least was low. I had set it low intentionally to get a fail-fast on the call to next. If I don't set it low, it will just take whatever the timeout is to fail (60 seconds default).
>>
>>
>> Anything in its logs when regionserver slows down?
>>
>> ANSWER: Yes. I see ScannerTimeoutException, and unknown scanner, and then ClosedChannelException. Stack trace shows the ClosedChannelException occurs when the server tries to write the response to the scanner. This seems like a bug to me. Once you close the channel you cannot write it, no? If you try to write it after you close it, you will get ClosedChannelException.
>>
>>
>>> next step will be to test with JD Cryans suggestion:
>>> " In order to completely rule out at least one thing, can you set ipc.server.max.queue.size to 1 and hbase.regionserver.handler.count to a low number (let's say 10)? If payload is putting too much memory pressure, we'll know."
>>>
>>> ...though I'm not sure what I'm supposed to observe with these settings...but I'll try it and report on the outcome.
>>>
>>
>> Well, you have GC logging already. Â If you look at the output do you
>> see big pauses?
>>
>> ANSWER: Nope, no long pauses at all. I've periodically run a few greps with a regex to try to find pauses one second or longer, and I haven't seen any of late. HOWEVER, one thing I don't understand at all is why ganglia reports HUGE gc pauses (like 1000 seconds!). But in reality I can *never* find such a pause in the GC log. Is there any known issue with ganglia graphs being on the wrong vertical scale for GC pauses? I know that sounds odd, but I just can't correlate the Ganglia graphs for GC to reality.
>>
>> Â I think J-D was thinking that regionservers would be
>> using less memory if you make the queues smaller. Â You could try that.
>> Â Maybe when queues are big, its taking a while to process them and
>> client times out. Â What size are these rows?
>>
>> ANSWER: rows are about 100KB-750KB. Trying J-D's suggestion now.
>>
>> St.Ack
>>
>>
>>> -geoff
>>>
>>> -----Original Message-----
>>> From: Geoff Hendrey [mailto:ghendrey@decarta.com]
>>> Sent: Tuesday, September 13, 2011 4:50 PM
>>> To: user@hbase.apache.org; Andrew Purtell
>>> Cc: Tony Wang; Rohit Nigam; Parmod Mehta; James Ladd
>>> Subject: RE: scanner deadlock?
>>>
>>> 1019 sockets on 50010 in CLOSED_WAIT state.
>>>
>>> -geoff
>>>
>>> -----Original Message-----
>>> From: Andrew Purtell [mailto:apurtell@apache.org]
>>> Sent: Tuesday, September 13, 2011 4:00 PM
>>> To: user@hbase.apache.org
>>> Cc: Tony Wang; Rohit Nigam; Parmod Mehta; James Ladd
>>> Subject: Re: scanner deadlock?
>>>
>>>
>>>
>>>> My current working theory is that
>>>> too many sockets are in CLOSE_WAIT state (leading to
>>>> ClosedChannelException?). We're going to try to adjust some OS
>>>> parameters.
>>>
>>> How many sockets are in that state? netstat -an | grep CLOSE_WAIT | wc -l
>>>
>>> CDH3U1 contains HDFS-1836... https://issues.apache.org/jira/browse/HDFS-1836
>>>
>>> Best regards,
>>>
>>> Â Â Â Â Â Â Â - Andy
>>>
>>> Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
>>>
>>>
>>>>________________________________
>>>>From: Geoff Hendrey <gh...@decarta.com>
>>>>To: user@hbase.apache.org
>>>>Cc: Tony Wang <tw...@decarta.com>; Rohit Nigam <rn...@decarta.com>; Parmod Mehta <pm...@decarta.com>; James Ladd <jl...@decarta.com>
>>>>Sent: Tuesday, September 13, 2011 9:49 AM
>>>>Subject: RE: scanner deadlock?
>>>>
>>>>Thanks Stack -
>>>>
>>>>Answers to all your questions below. My current working theory is that
>>>>too many sockets are in CLOSE_WAIT state (leading to
>>>>ClosedChannelException?). We're going to try to adjust some OS
>>>>parameters.
>>>>
>>>>" I'm asking if regionservers are bottlenecking on a single network
>>>>resource; a particular datanode, dns?"
>>>>
>>>>Gotcha. I'm gathering some tools now to collect and analyze netstat
>>>>output.
>>>>
>>>>" the regionserver is going slow getting data out of
>>>>hdfs. Whats iowait like at the time of slowness? Has it changed from
>>>>when all was running nicely?"
>>>>
>>>>iowait is high (20% above cpu), but not increasing. I'll try to quantify
>>>>that better.
>>>>
>>>>" You talk to hbase in the reducer?   Reducers don't start writing hbase
>>>>until job is 66% complete IIRC.  Perhaps its slowing as soon as it
>>>>starts writing hbase? Is that so?"
>>>>
>>>>My statement about "running fine" applies to after the reducer has
>>>>completed sort. We have metrics produced by the reducer that log the
>>>>results of scans ant Puts. So we know that scans and puts proceed
>>>>without issue for hours.
>>>>
>>>
>>
>
Re: scanner deadlock?
Posted by Stack <st...@duboce.net>.
Also, how many nodes, and how many reducers.
St.Ack
On Wed, Sep 14, 2011 at 8:38 AM, Stack <st...@duboce.net> wrote:
> Yeah. Â Ten handlers and no queue for the RPCs, it'll just reject calls
> when its > handler count.
>
> Also send a listing of your hbase rootdir: hadoop fs -lsr /hbase
>
> St.Ack
>
> On Tue, Sep 13, 2011 at 11:40 PM, Geoff Hendrey <gh...@decarta.com> wrote:
>> As expected, J-D's suggestion basically causes the system to collapse almost immediately. All the region servers show stack traces such as these:
>>
>> 2011-09-13 23:34:43,151 WARN Â [IPC Reader 0 on port 60020] ipc.HBaseServer$Listener(526): IPC Server listener on 60020: readAndProcess threw exception java.io.IOException: Connection reset by peer. Count of bytes read: 0
>> java.io.IOException: Connection reset by peer
>> Â Â Â Â at sun.nio.ch.FileDispatcher.read0(Native Method)
>> Â Â Â Â at sun.nio.ch.SocketDispatcher.read(Unknown Source)
>> Â Â Â Â at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
>> Â Â Â Â at sun.nio.ch.IOUtil.read(Unknown Source)
>> Â Â Â Â at sun.nio.ch.SocketChannelImpl.read(Unknown Source)
>> Â Â Â Â at org.apache.hadoop.hbase.ipc.HBaseServer.channelRead(HBaseServer.java:1359)
>> Â Â Â Â at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:900)
>> Â Â Â Â at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:522)
>> Â Â Â Â at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:316)
>> Â Â Â Â at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>> Â Â Â Â at java.util.concurrent.ThreadPoolExe
>>
>>
>> 2011-09-13 23:34:43,293 WARN Â [IPC Server handler 0 on 60020] ipc.HBaseServer$Handler(1100): IPC Server handler 0 on 60020 caught: java.nio.channels.ClosedChannelException
>> Â Â Â Â at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(Unknown Source)
>> Â Â Â Â at sun.nio.ch.SocketChannelImpl.write(Unknown Source)
>> Â Â Â Â at org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1341)
>> Â Â Â Â at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:727)
>> Â Â Â Â at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:792)
>> Â Â Â Â at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1083)
>>
>> -----Original Message-----
>> From: Geoff Hendrey [mailto:ghendrey@decarta.com]
>> Sent: Tuesday, September 13, 2011 10:56 PM
>> To: user@hbase.apache.org
>> Cc: Andrew Purtell; Tony Wang; Rohit Nigam; Parmod Mehta; James Ladd
>> Subject: RE: scanner deadlock?
>>
>> answers below
>>
>> -----Original Message-----
>> From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of Stack
>> Sent: Tuesday, September 13, 2011 10:41 PM
>> To: user@hbase.apache.org
>> Cc: Andrew Purtell; Tony Wang; Rohit Nigam; Parmod Mehta; James Ladd
>> Subject: Re: scanner deadlock?
>>
>> On Tue, Sep 13, 2011 at 10:25 PM, Geoff Hendrey <gh...@decarta.com> wrote:
>>> I've upgraded to HotSpot 64 Bit Server VM, with HBase 90.4 and all recommended config changes (100 region server handlers, mslab enabled, etc). No change, if anything it dies faster. Count of sockets in CLOSE_WAIT on 50010 increases linearly. I logged netstat from a random node in the cluster, periodically. Then dumped the output into excel using a pivot table to look at a behavior of TCP. Number of connections from the given node to others on 50010 was relatively uniform (no hotspot). Connections on 50010 from given node to *self* was much way higher than to other nodes, but that's probably a good thing. My guess is it's HBase leveraging locality of files for the region server. Just a guess.
>>>
>> Yes. Â You have good locality. Â So maybe you are not bound up on a
>> single network resource.
>>
>> So when you jstack and you see that regionserver has its threads all
>> stuck in next -- are they? -- then we are likely going to the local
>> datanode.
>>
>> ANSWER: yes, I believe they are stuck in next. I can see from logs on the client that the call to next periodically takes very long to return. You had earlier commented that my 5 seconds regionserver least was low. I had set it low intentionally to get a fail-fast on the call to next. If I don't set it low, it will just take whatever the timeout is to fail (60 seconds default).
>>
>>
>> Anything in its logs when regionserver slows down?
>>
>> ANSWER: Yes. I see ScannerTimeoutException, and unknown scanner, and then ClosedChannelException. Stack trace shows the ClosedChannelException occurs when the server tries to write the response to the scanner. This seems like a bug to me. Once you close the channel you cannot write it, no? If you try to write it after you close it, you will get ClosedChannelException.
>>
>>
>>> next step will be to test with JD Cryans suggestion:
>>> " In order to completely rule out at least one thing, can you set ipc.server.max.queue.size to 1 and hbase.regionserver.handler.count to a low number (let's say 10)? If payload is putting too much memory pressure, we'll know."
>>>
>>> ...though I'm not sure what I'm supposed to observe with these settings...but I'll try it and report on the outcome.
>>>
>>
>> Well, you have GC logging already. Â If you look at the output do you
>> see big pauses?
>>
>> ANSWER: Nope, no long pauses at all. I've periodically run a few greps with a regex to try to find pauses one second or longer, and I haven't seen any of late. HOWEVER, one thing I don't understand at all is why ganglia reports HUGE gc pauses (like 1000 seconds!). But in reality I can *never* find such a pause in the GC log. Is there any known issue with ganglia graphs being on the wrong vertical scale for GC pauses? I know that sounds odd, but I just can't correlate the Ganglia graphs for GC to reality.
>>
>> Â I think J-D was thinking that regionservers would be
>> using less memory if you make the queues smaller. Â You could try that.
>> Â Maybe when queues are big, its taking a while to process them and
>> client times out. Â What size are these rows?
>>
>> ANSWER: rows are about 100KB-750KB. Trying J-D's suggestion now.
>>
>> St.Ack
>>
>>
>>> -geoff
>>>
>>> -----Original Message-----
>>> From: Geoff Hendrey [mailto:ghendrey@decarta.com]
>>> Sent: Tuesday, September 13, 2011 4:50 PM
>>> To: user@hbase.apache.org; Andrew Purtell
>>> Cc: Tony Wang; Rohit Nigam; Parmod Mehta; James Ladd
>>> Subject: RE: scanner deadlock?
>>>
>>> 1019 sockets on 50010 in CLOSED_WAIT state.
>>>
>>> -geoff
>>>
>>> -----Original Message-----
>>> From: Andrew Purtell [mailto:apurtell@apache.org]
>>> Sent: Tuesday, September 13, 2011 4:00 PM
>>> To: user@hbase.apache.org
>>> Cc: Tony Wang; Rohit Nigam; Parmod Mehta; James Ladd
>>> Subject: Re: scanner deadlock?
>>>
>>>
>>>
>>>> My current working theory is that
>>>> too many sockets are in CLOSE_WAIT state (leading to
>>>> ClosedChannelException?). We're going to try to adjust some OS
>>>> parameters.
>>>
>>> How many sockets are in that state? netstat -an | grep CLOSE_WAIT | wc -l
>>>
>>> CDH3U1 contains HDFS-1836... https://issues.apache.org/jira/browse/HDFS-1836
>>>
>>> Best regards,
>>>
>>> Â Â Â Â Â Â Â - Andy
>>>
>>> Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
>>>
>>>
>>>>________________________________
>>>>From: Geoff Hendrey <gh...@decarta.com>
>>>>To: user@hbase.apache.org
>>>>Cc: Tony Wang <tw...@decarta.com>; Rohit Nigam <rn...@decarta.com>; Parmod Mehta <pm...@decarta.com>; James Ladd <jl...@decarta.com>
>>>>Sent: Tuesday, September 13, 2011 9:49 AM
>>>>Subject: RE: scanner deadlock?
>>>>
>>>>Thanks Stack -
>>>>
>>>>Answers to all your questions below. My current working theory is that
>>>>too many sockets are in CLOSE_WAIT state (leading to
>>>>ClosedChannelException?). We're going to try to adjust some OS
>>>>parameters.
>>>>
>>>>" I'm asking if regionservers are bottlenecking on a single network
>>>>resource; a particular datanode, dns?"
>>>>
>>>>Gotcha. I'm gathering some tools now to collect and analyze netstat
>>>>output.
>>>>
>>>>" the regionserver is going slow getting data out of
>>>>hdfs. Whats iowait like at the time of slowness? Has it changed from
>>>>when all was running nicely?"
>>>>
>>>>iowait is high (20% above cpu), but not increasing. I'll try to quantify
>>>>that better.
>>>>
>>>>" You talk to hbase in the reducer?   Reducers don't start writing hbase
>>>>until job is 66% complete IIRC.  Perhaps its slowing as soon as it
>>>>starts writing hbase? Is that so?"
>>>>
>>>>My statement about "running fine" applies to after the reducer has
>>>>completed sort. We have metrics produced by the reducer that log the
>>>>results of scans ant Puts. So we know that scans and puts proceed
>>>>without issue for hours.
>>>>
>>>
>>
>
RE: scanner deadlock?
Posted by Geoff Hendrey <gh...@decarta.com>.
Thanks Stack -
We will work on putting together a tar of relevant logs and dump it on pastebin. Thanks a lot.
-geoff
-----Original Message-----
From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of Stack
Sent: Wednesday, September 14, 2011 8:38 AM
To: user@hbase.apache.org
Subject: Re: scanner deadlock?
Yeah. Ten handlers and no queue for the RPCs, it'll just reject calls
when its > handler count.
Also send a listing of your hbase rootdir: hadoop fs -lsr /hbase
St.Ack
On Tue, Sep 13, 2011 at 11:40 PM, Geoff Hendrey <gh...@decarta.com> wrote:
> As expected, J-D's suggestion basically causes the system to collapse almost immediately. All the region servers show stack traces such as these:
>
> 2011-09-13 23:34:43,151 WARN Â [IPC Reader 0 on port 60020] ipc.HBaseServer$Listener(526): IPC Server listener on 60020: readAndProcess threw exception java.io.IOException: Connection reset by peer. Count of bytes read: 0
> java.io.IOException: Connection reset by peer
> Â Â Â Â at sun.nio.ch.FileDispatcher.read0(Native Method)
> Â Â Â Â at sun.nio.ch.SocketDispatcher.read(Unknown Source)
> Â Â Â Â at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
> Â Â Â Â at sun.nio.ch.IOUtil.read(Unknown Source)
> Â Â Â Â at sun.nio.ch.SocketChannelImpl.read(Unknown Source)
> Â Â Â Â at org.apache.hadoop.hbase.ipc.HBaseServer.channelRead(HBaseServer.java:1359)
> Â Â Â Â at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:900)
> Â Â Â Â at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:522)
> Â Â Â Â at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:316)
> Â Â Â Â at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> Â Â Â Â at java.util.concurrent.ThreadPoolExe
>
>
> 2011-09-13 23:34:43,293 WARN Â [IPC Server handler 0 on 60020] ipc.HBaseServer$Handler(1100): IPC Server handler 0 on 60020 caught: java.nio.channels.ClosedChannelException
> Â Â Â Â at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(Unknown Source)
> Â Â Â Â at sun.nio.ch.SocketChannelImpl.write(Unknown Source)
> Â Â Â Â at org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1341)
> Â Â Â Â at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:727)
> Â Â Â Â at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:792)
> Â Â Â Â at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1083)
>
> -----Original Message-----
> From: Geoff Hendrey [mailto:ghendrey@decarta.com]
> Sent: Tuesday, September 13, 2011 10:56 PM
> To: user@hbase.apache.org
> Cc: Andrew Purtell; Tony Wang; Rohit Nigam; Parmod Mehta; James Ladd
> Subject: RE: scanner deadlock?
>
> answers below
>
> -----Original Message-----
> From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of Stack
> Sent: Tuesday, September 13, 2011 10:41 PM
> To: user@hbase.apache.org
> Cc: Andrew Purtell; Tony Wang; Rohit Nigam; Parmod Mehta; James Ladd
> Subject: Re: scanner deadlock?
>
> On Tue, Sep 13, 2011 at 10:25 PM, Geoff Hendrey <gh...@decarta.com> wrote:
>> I've upgraded to HotSpot 64 Bit Server VM, with HBase 90.4 and all recommended config changes (100 region server handlers, mslab enabled, etc). No change, if anything it dies faster. Count of sockets in CLOSE_WAIT on 50010 increases linearly. I logged netstat from a random node in the cluster, periodically. Then dumped the output into excel using a pivot table to look at a behavior of TCP. Number of connections from the given node to others on 50010 was relatively uniform (no hotspot). Connections on 50010 from given node to *self* was much way higher than to other nodes, but that's probably a good thing. My guess is it's HBase leveraging locality of files for the region server. Just a guess.
>>
> Yes. Â You have good locality. Â So maybe you are not bound up on a
> single network resource.
>
> So when you jstack and you see that regionserver has its threads all
> stuck in next -- are they? -- then we are likely going to the local
> datanode.
>
> ANSWER: yes, I believe they are stuck in next. I can see from logs on the client that the call to next periodically takes very long to return. You had earlier commented that my 5 seconds regionserver least was low. I had set it low intentionally to get a fail-fast on the call to next. If I don't set it low, it will just take whatever the timeout is to fail (60 seconds default).
>
>
> Anything in its logs when regionserver slows down?
>
> ANSWER: Yes. I see ScannerTimeoutException, and unknown scanner, and then ClosedChannelException. Stack trace shows the ClosedChannelException occurs when the server tries to write the response to the scanner. This seems like a bug to me. Once you close the channel you cannot write it, no? If you try to write it after you close it, you will get ClosedChannelException.
>
>
>> next step will be to test with JD Cryans suggestion:
>> " In order to completely rule out at least one thing, can you set ipc.server.max.queue.size to 1 and hbase.regionserver.handler.count to a low number (let's say 10)? If payload is putting too much memory pressure, we'll know."
>>
>> ...though I'm not sure what I'm supposed to observe with these settings...but I'll try it and report on the outcome.
>>
>
> Well, you have GC logging already. Â If you look at the output do you
> see big pauses?
>
> ANSWER: Nope, no long pauses at all. I've periodically run a few greps with a regex to try to find pauses one second or longer, and I haven't seen any of late. HOWEVER, one thing I don't understand at all is why ganglia reports HUGE gc pauses (like 1000 seconds!). But in reality I can *never* find such a pause in the GC log. Is there any known issue with ganglia graphs being on the wrong vertical scale for GC pauses? I know that sounds odd, but I just can't correlate the Ganglia graphs for GC to reality.
>
> Â I think J-D was thinking that regionservers would be
> using less memory if you make the queues smaller. Â You could try that.
> Â Maybe when queues are big, its taking a while to process them and
> client times out. Â What size are these rows?
>
> ANSWER: rows are about 100KB-750KB. Trying J-D's suggestion now.
>
> St.Ack
>
>
>> -geoff
>>
>> -----Original Message-----
>> From: Geoff Hendrey [mailto:ghendrey@decarta.com]
>> Sent: Tuesday, September 13, 2011 4:50 PM
>> To: user@hbase.apache.org; Andrew Purtell
>> Cc: Tony Wang; Rohit Nigam; Parmod Mehta; James Ladd
>> Subject: RE: scanner deadlock?
>>
>> 1019 sockets on 50010 in CLOSED_WAIT state.
>>
>> -geoff
>>
>> -----Original Message-----
>> From: Andrew Purtell [mailto:apurtell@apache.org]
>> Sent: Tuesday, September 13, 2011 4:00 PM
>> To: user@hbase.apache.org
>> Cc: Tony Wang; Rohit Nigam; Parmod Mehta; James Ladd
>> Subject: Re: scanner deadlock?
>>
>>
>>
>>> My current working theory is that
>>> too many sockets are in CLOSE_WAIT state (leading to
>>> ClosedChannelException?). We're going to try to adjust some OS
>>> parameters.
>>
>> How many sockets are in that state? netstat -an | grep CLOSE_WAIT | wc -l
>>
>> CDH3U1 contains HDFS-1836... https://issues.apache.org/jira/browse/HDFS-1836
>>
>> Best regards,
>>
>> Â Â Â Â Â Â Â - Andy
>>
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
>>
>>
>>>________________________________
>>>From: Geoff Hendrey <gh...@decarta.com>
>>>To: user@hbase.apache.org
>>>Cc: Tony Wang <tw...@decarta.com>; Rohit Nigam <rn...@decarta.com>; Parmod Mehta <pm...@decarta.com>; James Ladd <jl...@decarta.com>
>>>Sent: Tuesday, September 13, 2011 9:49 AM
>>>Subject: RE: scanner deadlock?
>>>
>>>Thanks Stack -
>>>
>>>Answers to all your questions below. My current working theory is that
>>>too many sockets are in CLOSE_WAIT state (leading to
>>>ClosedChannelException?). We're going to try to adjust some OS
>>>parameters.
>>>
>>>" I'm asking if regionservers are bottlenecking on a single network
>>>resource; a particular datanode, dns?"
>>>
>>>Gotcha. I'm gathering some tools now to collect and analyze netstat
>>>output.
>>>
>>>" the regionserver is going slow getting data out of
>>>hdfs. Whats iowait like at the time of slowness? Has it changed from
>>>when all was running nicely?"
>>>
>>>iowait is high (20% above cpu), but not increasing. I'll try to quantify
>>>that better.
>>>
>>>" You talk to hbase in the reducer?   Reducers don't start writing hbase
>>>until job is 66% complete IIRC.  Perhaps its slowing as soon as it
>>>starts writing hbase? Is that so?"
>>>
>>>My statement about "running fine" applies to after the reducer has
>>>completed sort. We have metrics produced by the reducer that log the
>>>results of scans ant Puts. So we know that scans and puts proceed
>>>without issue for hours.
>>>
>>
>
Re: scanner deadlock?
Posted by Stack <st...@duboce.net>.
Yeah. Ten handlers and no queue for the RPCs, it'll just reject calls
when its > handler count.
Also send a listing of your hbase rootdir: hadoop fs -lsr /hbase
St.Ack
On Tue, Sep 13, 2011 at 11:40 PM, Geoff Hendrey <gh...@decarta.com> wrote:
> As expected, J-D's suggestion basically causes the system to collapse almost immediately. All the region servers show stack traces such as these:
>
> 2011-09-13 23:34:43,151 WARN Â [IPC Reader 0 on port 60020] ipc.HBaseServer$Listener(526): IPC Server listener on 60020: readAndProcess threw exception java.io.IOException: Connection reset by peer. Count of bytes read: 0
> java.io.IOException: Connection reset by peer
> Â Â Â Â at sun.nio.ch.FileDispatcher.read0(Native Method)
> Â Â Â Â at sun.nio.ch.SocketDispatcher.read(Unknown Source)
> Â Â Â Â at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
> Â Â Â Â at sun.nio.ch.IOUtil.read(Unknown Source)
> Â Â Â Â at sun.nio.ch.SocketChannelImpl.read(Unknown Source)
> Â Â Â Â at org.apache.hadoop.hbase.ipc.HBaseServer.channelRead(HBaseServer.java:1359)
> Â Â Â Â at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:900)
> Â Â Â Â at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:522)
> Â Â Â Â at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:316)
> Â Â Â Â at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> Â Â Â Â at java.util.concurrent.ThreadPoolExe
>
>
> 2011-09-13 23:34:43,293 WARN Â [IPC Server handler 0 on 60020] ipc.HBaseServer$Handler(1100): IPC Server handler 0 on 60020 caught: java.nio.channels.ClosedChannelException
> Â Â Â Â at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(Unknown Source)
> Â Â Â Â at sun.nio.ch.SocketChannelImpl.write(Unknown Source)
> Â Â Â Â at org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1341)
> Â Â Â Â at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:727)
> Â Â Â Â at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:792)
> Â Â Â Â at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1083)
>
> -----Original Message-----
> From: Geoff Hendrey [mailto:ghendrey@decarta.com]
> Sent: Tuesday, September 13, 2011 10:56 PM
> To: user@hbase.apache.org
> Cc: Andrew Purtell; Tony Wang; Rohit Nigam; Parmod Mehta; James Ladd
> Subject: RE: scanner deadlock?
>
> answers below
>
> -----Original Message-----
> From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of Stack
> Sent: Tuesday, September 13, 2011 10:41 PM
> To: user@hbase.apache.org
> Cc: Andrew Purtell; Tony Wang; Rohit Nigam; Parmod Mehta; James Ladd
> Subject: Re: scanner deadlock?
>
> On Tue, Sep 13, 2011 at 10:25 PM, Geoff Hendrey <gh...@decarta.com> wrote:
>> I've upgraded to HotSpot 64 Bit Server VM, with HBase 90.4 and all recommended config changes (100 region server handlers, mslab enabled, etc). No change, if anything it dies faster. Count of sockets in CLOSE_WAIT on 50010 increases linearly. I logged netstat from a random node in the cluster, periodically. Then dumped the output into excel using a pivot table to look at a behavior of TCP. Number of connections from the given node to others on 50010 was relatively uniform (no hotspot). Connections on 50010 from given node to *self* was much way higher than to other nodes, but that's probably a good thing. My guess is it's HBase leveraging locality of files for the region server. Just a guess.
>>
> Yes. Â You have good locality. Â So maybe you are not bound up on a
> single network resource.
>
> So when you jstack and you see that regionserver has its threads all
> stuck in next -- are they? -- then we are likely going to the local
> datanode.
>
> ANSWER: yes, I believe they are stuck in next. I can see from logs on the client that the call to next periodically takes very long to return. You had earlier commented that my 5 seconds regionserver least was low. I had set it low intentionally to get a fail-fast on the call to next. If I don't set it low, it will just take whatever the timeout is to fail (60 seconds default).
>
>
> Anything in its logs when regionserver slows down?
>
> ANSWER: Yes. I see ScannerTimeoutException, and unknown scanner, and then ClosedChannelException. Stack trace shows the ClosedChannelException occurs when the server tries to write the response to the scanner. This seems like a bug to me. Once you close the channel you cannot write it, no? If you try to write it after you close it, you will get ClosedChannelException.
>
>
>> next step will be to test with JD Cryans suggestion:
>> " In order to completely rule out at least one thing, can you set ipc.server.max.queue.size to 1 and hbase.regionserver.handler.count to a low number (let's say 10)? If payload is putting too much memory pressure, we'll know."
>>
>> ...though I'm not sure what I'm supposed to observe with these settings...but I'll try it and report on the outcome.
>>
>
> Well, you have GC logging already. Â If you look at the output do you
> see big pauses?
>
> ANSWER: Nope, no long pauses at all. I've periodically run a few greps with a regex to try to find pauses one second or longer, and I haven't seen any of late. HOWEVER, one thing I don't understand at all is why ganglia reports HUGE gc pauses (like 1000 seconds!). But in reality I can *never* find such a pause in the GC log. Is there any known issue with ganglia graphs being on the wrong vertical scale for GC pauses? I know that sounds odd, but I just can't correlate the Ganglia graphs for GC to reality.
>
> Â I think J-D was thinking that regionservers would be
> using less memory if you make the queues smaller. Â You could try that.
> Â Maybe when queues are big, its taking a while to process them and
> client times out. Â What size are these rows?
>
> ANSWER: rows are about 100KB-750KB. Trying J-D's suggestion now.
>
> St.Ack
>
>
>> -geoff
>>
>> -----Original Message-----
>> From: Geoff Hendrey [mailto:ghendrey@decarta.com]
>> Sent: Tuesday, September 13, 2011 4:50 PM
>> To: user@hbase.apache.org; Andrew Purtell
>> Cc: Tony Wang; Rohit Nigam; Parmod Mehta; James Ladd
>> Subject: RE: scanner deadlock?
>>
>> 1019 sockets on 50010 in CLOSED_WAIT state.
>>
>> -geoff
>>
>> -----Original Message-----
>> From: Andrew Purtell [mailto:apurtell@apache.org]
>> Sent: Tuesday, September 13, 2011 4:00 PM
>> To: user@hbase.apache.org
>> Cc: Tony Wang; Rohit Nigam; Parmod Mehta; James Ladd
>> Subject: Re: scanner deadlock?
>>
>>
>>
>>> My current working theory is that
>>> too many sockets are in CLOSE_WAIT state (leading to
>>> ClosedChannelException?). We're going to try to adjust some OS
>>> parameters.
>>
>> How many sockets are in that state? netstat -an | grep CLOSE_WAIT | wc -l
>>
>> CDH3U1 contains HDFS-1836... https://issues.apache.org/jira/browse/HDFS-1836
>>
>> Best regards,
>>
>> Â Â Â Â Â Â Â - Andy
>>
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
>>
>>
>>>________________________________
>>>From: Geoff Hendrey <gh...@decarta.com>
>>>To: user@hbase.apache.org
>>>Cc: Tony Wang <tw...@decarta.com>; Rohit Nigam <rn...@decarta.com>; Parmod Mehta <pm...@decarta.com>; James Ladd <jl...@decarta.com>
>>>Sent: Tuesday, September 13, 2011 9:49 AM
>>>Subject: RE: scanner deadlock?
>>>
>>>Thanks Stack -
>>>
>>>Answers to all your questions below. My current working theory is that
>>>too many sockets are in CLOSE_WAIT state (leading to
>>>ClosedChannelException?). We're going to try to adjust some OS
>>>parameters.
>>>
>>>" I'm asking if regionservers are bottlenecking on a single network
>>>resource; a particular datanode, dns?"
>>>
>>>Gotcha. I'm gathering some tools now to collect and analyze netstat
>>>output.
>>>
>>>" the regionserver is going slow getting data out of
>>>hdfs. Whats iowait like at the time of slowness? Has it changed from
>>>when all was running nicely?"
>>>
>>>iowait is high (20% above cpu), but not increasing. I'll try to quantify
>>>that better.
>>>
>>>" You talk to hbase in the reducer?   Reducers don't start writing hbase
>>>until job is 66% complete IIRC.  Perhaps its slowing as soon as it
>>>starts writing hbase? Is that so?"
>>>
>>>My statement about "running fine" applies to after the reducer has
>>>completed sort. We have metrics produced by the reducer that log the
>>>results of scans ant Puts. So we know that scans and puts proceed
>>>without issue for hours.
>>>
>>
>
RE: scanner deadlock?
Posted by Geoff Hendrey <gh...@decarta.com>.
As expected, J-D's suggestion basically causes the system to collapse almost immediately. All the region servers show stack traces such as these:
2011-09-13 23:34:43,151 WARN [IPC Reader 0 on port 60020] ipc.HBaseServer$Listener(526): IPC Server listener on 60020: readAndProcess threw exception java.io.IOException: Connection reset by peer. Count of bytes read: 0
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(Unknown Source)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
at sun.nio.ch.IOUtil.read(Unknown Source)
at sun.nio.ch.SocketChannelImpl.read(Unknown Source)
at org.apache.hadoop.hbase.ipc.HBaseServer.channelRead(HBaseServer.java:1359)
at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:900)
at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:522)
at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:316)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExe
2011-09-13 23:34:43,293 WARN [IPC Server handler 0 on 60020] ipc.HBaseServer$Handler(1100): IPC Server handler 0 on 60020 caught: java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(Unknown Source)
at sun.nio.ch.SocketChannelImpl.write(Unknown Source)
at org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1341)
at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:727)
at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:792)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1083)
-----Original Message-----
From: Geoff Hendrey [mailto:ghendrey@decarta.com]
Sent: Tuesday, September 13, 2011 10:56 PM
To: user@hbase.apache.org
Cc: Andrew Purtell; Tony Wang; Rohit Nigam; Parmod Mehta; James Ladd
Subject: RE: scanner deadlock?
answers below
-----Original Message-----
From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of Stack
Sent: Tuesday, September 13, 2011 10:41 PM
To: user@hbase.apache.org
Cc: Andrew Purtell; Tony Wang; Rohit Nigam; Parmod Mehta; James Ladd
Subject: Re: scanner deadlock?
On Tue, Sep 13, 2011 at 10:25 PM, Geoff Hendrey <gh...@decarta.com> wrote:
> I've upgraded to HotSpot 64 Bit Server VM, with HBase 90.4 and all recommended config changes (100 region server handlers, mslab enabled, etc). No change, if anything it dies faster. Count of sockets in CLOSE_WAIT on 50010 increases linearly. I logged netstat from a random node in the cluster, periodically. Then dumped the output into excel using a pivot table to look at a behavior of TCP. Number of connections from the given node to others on 50010 was relatively uniform (no hotspot). Connections on 50010 from given node to *self* was much way higher than to other nodes, but that's probably a good thing. My guess is it's HBase leveraging locality of files for the region server. Just a guess.
>
Yes. You have good locality. So maybe you are not bound up on a
single network resource.
So when you jstack and you see that regionserver has its threads all
stuck in next -- are they? -- then we are likely going to the local
datanode.
ANSWER: yes, I believe they are stuck in next. I can see from logs on the client that the call to next periodically takes very long to return. You had earlier commented that my 5 seconds regionserver least was low. I had set it low intentionally to get a fail-fast on the call to next. If I don't set it low, it will just take whatever the timeout is to fail (60 seconds default).
Anything in its logs when regionserver slows down?
ANSWER: Yes. I see ScannerTimeoutException, and unknown scanner, and then ClosedChannelException. Stack trace shows the ClosedChannelException occurs when the server tries to write the response to the scanner. This seems like a bug to me. Once you close the channel you cannot write it, no? If you try to write it after you close it, you will get ClosedChannelException.
> next step will be to test with JD Cryans suggestion:
> " In order to completely rule out at least one thing, can you set ipc.server.max.queue.size to 1 and hbase.regionserver.handler.count to a low number (let's say 10)? If payload is putting too much memory pressure, we'll know."
>
> ...though I'm not sure what I'm supposed to observe with these settings...but I'll try it and report on the outcome.
>
Well, you have GC logging already. If you look at the output do you
see big pauses?
ANSWER: Nope, no long pauses at all. I've periodically run a few greps with a regex to try to find pauses one second or longer, and I haven't seen any of late. HOWEVER, one thing I don't understand at all is why ganglia reports HUGE gc pauses (like 1000 seconds!). But in reality I can *never* find such a pause in the GC log. Is there any known issue with ganglia graphs being on the wrong vertical scale for GC pauses? I know that sounds odd, but I just can't correlate the Ganglia graphs for GC to reality.
I think J-D was thinking that regionservers would be
using less memory if you make the queues smaller. You could try that.
Maybe when queues are big, its taking a while to process them and
client times out. What size are these rows?
ANSWER: rows are about 100KB-750KB. Trying J-D's suggestion now.
St.Ack
> -geoff
>
> -----Original Message-----
> From: Geoff Hendrey [mailto:ghendrey@decarta.com]
> Sent: Tuesday, September 13, 2011 4:50 PM
> To: user@hbase.apache.org; Andrew Purtell
> Cc: Tony Wang; Rohit Nigam; Parmod Mehta; James Ladd
> Subject: RE: scanner deadlock?
>
> 1019 sockets on 50010 in CLOSED_WAIT state.
>
> -geoff
>
> -----Original Message-----
> From: Andrew Purtell [mailto:apurtell@apache.org]
> Sent: Tuesday, September 13, 2011 4:00 PM
> To: user@hbase.apache.org
> Cc: Tony Wang; Rohit Nigam; Parmod Mehta; James Ladd
> Subject: Re: scanner deadlock?
>
>
>
>> My current working theory is that
>> too many sockets are in CLOSE_WAIT state (leading to
>> ClosedChannelException?). We're going to try to adjust some OS
>> parameters.
>
> How many sockets are in that state? netstat -an | grep CLOSE_WAIT | wc -l
>
> CDH3U1 contains HDFS-1836... https://issues.apache.org/jira/browse/HDFS-1836
>
> Best regards,
>
> Â Â Â Â Â Â Â - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
>
>
>>________________________________
>>From: Geoff Hendrey <gh...@decarta.com>
>>To: user@hbase.apache.org
>>Cc: Tony Wang <tw...@decarta.com>; Rohit Nigam <rn...@decarta.com>; Parmod Mehta <pm...@decarta.com>; James Ladd <jl...@decarta.com>
>>Sent: Tuesday, September 13, 2011 9:49 AM
>>Subject: RE: scanner deadlock?
>>
>>Thanks Stack -
>>
>>Answers to all your questions below. My current working theory is that
>>too many sockets are in CLOSE_WAIT state (leading to
>>ClosedChannelException?). We're going to try to adjust some OS
>>parameters.
>>
>>" I'm asking if regionservers are bottlenecking on a single network
>>resource; a particular datanode, dns?"
>>
>>Gotcha. I'm gathering some tools now to collect and analyze netstat
>>output.
>>
>>" the regionserver is going slow getting data out of
>>hdfs. Whats iowait like at the time of slowness? Has it changed from
>>when all was running nicely?"
>>
>>iowait is high (20% above cpu), but not increasing. I'll try to quantify
>>that better.
>>
>>" You talk to hbase in the reducer?   Reducers don't start writing hbase
>>until job is 66% complete IIRC.  Perhaps its slowing as soon as it
>>starts writing hbase? Is that so?"
>>
>>My statement about "running fine" applies to after the reducer has
>>completed sort. We have metrics produced by the reducer that log the
>>results of scans ant Puts. So we know that scans and puts proceed
>>without issue for hours.
>>
>
RE: scanner deadlock?
Posted by Geoff Hendrey <gh...@decarta.com>.
answers below
-----Original Message-----
From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of Stack
Sent: Tuesday, September 13, 2011 10:41 PM
To: user@hbase.apache.org
Cc: Andrew Purtell; Tony Wang; Rohit Nigam; Parmod Mehta; James Ladd
Subject: Re: scanner deadlock?
On Tue, Sep 13, 2011 at 10:25 PM, Geoff Hendrey <gh...@decarta.com> wrote:
> I've upgraded to HotSpot 64 Bit Server VM, with HBase 90.4 and all recommended config changes (100 region server handlers, mslab enabled, etc). No change, if anything it dies faster. Count of sockets in CLOSE_WAIT on 50010 increases linearly. I logged netstat from a random node in the cluster, periodically. Then dumped the output into excel using a pivot table to look at a behavior of TCP. Number of connections from the given node to others on 50010 was relatively uniform (no hotspot). Connections on 50010 from given node to *self* was much way higher than to other nodes, but that's probably a good thing. My guess is it's HBase leveraging locality of files for the region server. Just a guess.
>
Yes. You have good locality. So maybe you are not bound up on a
single network resource.
So when you jstack and you see that regionserver has its threads all
stuck in next -- are they? -- then we are likely going to the local
datanode.
ANSWER: yes, I believe they are stuck in next. I can see from logs on the client that the call to next periodically takes very long to return. You had earlier commented that my 5 seconds regionserver least was low. I had set it low intentionally to get a fail-fast on the call to next. If I don't set it low, it will just take whatever the timeout is to fail (60 seconds default).
Anything in its logs when regionserver slows down?
ANSWER: Yes. I see ScannerTimeoutException, and unknown scanner, and then ClosedChannelException. Stack trace shows the ClosedChannelException occurs when the server tries to write the response to the scanner. This seems like a bug to me. Once you close the channel you cannot write it, no? If you try to write it after you close it, you will get ClosedChannelException.
> next step will be to test with JD Cryans suggestion:
> " In order to completely rule out at least one thing, can you set ipc.server.max.queue.size to 1 and hbase.regionserver.handler.count to a low number (let's say 10)? If payload is putting too much memory pressure, we'll know."
>
> ...though I'm not sure what I'm supposed to observe with these settings...but I'll try it and report on the outcome.
>
Well, you have GC logging already. If you look at the output do you
see big pauses?
ANSWER: Nope, no long pauses at all. I've periodically run a few greps with a regex to try to find pauses one second or longer, and I haven't seen any of late. HOWEVER, one thing I don't understand at all is why ganglia reports HUGE gc pauses (like 1000 seconds!). But in reality I can *never* find such a pause in the GC log. Is there any known issue with ganglia graphs being on the wrong vertical scale for GC pauses? I know that sounds odd, but I just can't correlate the Ganglia graphs for GC to reality.
I think J-D was thinking that regionservers would be
using less memory if you make the queues smaller. You could try that.
Maybe when queues are big, its taking a while to process them and
client times out. What size are these rows?
ANSWER: rows are about 100KB-750KB. Trying J-D's suggestion now.
St.Ack
> -geoff
>
> -----Original Message-----
> From: Geoff Hendrey [mailto:ghendrey@decarta.com]
> Sent: Tuesday, September 13, 2011 4:50 PM
> To: user@hbase.apache.org; Andrew Purtell
> Cc: Tony Wang; Rohit Nigam; Parmod Mehta; James Ladd
> Subject: RE: scanner deadlock?
>
> 1019 sockets on 50010 in CLOSED_WAIT state.
>
> -geoff
>
> -----Original Message-----
> From: Andrew Purtell [mailto:apurtell@apache.org]
> Sent: Tuesday, September 13, 2011 4:00 PM
> To: user@hbase.apache.org
> Cc: Tony Wang; Rohit Nigam; Parmod Mehta; James Ladd
> Subject: Re: scanner deadlock?
>
>
>
>> My current working theory is that
>> too many sockets are in CLOSE_WAIT state (leading to
>> ClosedChannelException?). We're going to try to adjust some OS
>> parameters.
>
> How many sockets are in that state? netstat -an | grep CLOSE_WAIT | wc -l
>
> CDH3U1 contains HDFS-1836... https://issues.apache.org/jira/browse/HDFS-1836
>
> Best regards,
>
> Â Â Â Â Â Â Â - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
>
>
>>________________________________
>>From: Geoff Hendrey <gh...@decarta.com>
>>To: user@hbase.apache.org
>>Cc: Tony Wang <tw...@decarta.com>; Rohit Nigam <rn...@decarta.com>; Parmod Mehta <pm...@decarta.com>; James Ladd <jl...@decarta.com>
>>Sent: Tuesday, September 13, 2011 9:49 AM
>>Subject: RE: scanner deadlock?
>>
>>Thanks Stack -
>>
>>Answers to all your questions below. My current working theory is that
>>too many sockets are in CLOSE_WAIT state (leading to
>>ClosedChannelException?). We're going to try to adjust some OS
>>parameters.
>>
>>" I'm asking if regionservers are bottlenecking on a single network
>>resource; a particular datanode, dns?"
>>
>>Gotcha. I'm gathering some tools now to collect and analyze netstat
>>output.
>>
>>" the regionserver is going slow getting data out of
>>hdfs. Whats iowait like at the time of slowness? Has it changed from
>>when all was running nicely?"
>>
>>iowait is high (20% above cpu), but not increasing. I'll try to quantify
>>that better.
>>
>>" You talk to hbase in the reducer?   Reducers don't start writing hbase
>>until job is 66% complete IIRC.  Perhaps its slowing as soon as it
>>starts writing hbase? Is that so?"
>>
>>My statement about "running fine" applies to after the reducer has
>>completed sort. We have metrics produced by the reducer that log the
>>results of scans ant Puts. So we know that scans and puts proceed
>>without issue for hours.
>>
>
Re: scanner deadlock?
Posted by Stack <st...@duboce.net>.
On Tue, Sep 13, 2011 at 10:25 PM, Geoff Hendrey <gh...@decarta.com> wrote:
> I've upgraded to HotSpot 64 Bit Server VM, with HBase 90.4 and all recommended config changes (100 region server handlers, mslab enabled, etc). No change, if anything it dies faster. Count of sockets in CLOSE_WAIT on 50010 increases linearly. I logged netstat from a random node in the cluster, periodically. Then dumped the output into excel using a pivot table to look at a behavior of TCP. Number of connections from the given node to others on 50010 was relatively uniform (no hotspot). Connections on 50010 from given node to *self* was much way higher than to other nodes, but that's probably a good thing. My guess is it's HBase leveraging locality of files for the region server. Just a guess.
>
Yes. You have good locality. So maybe you are not bound up on a
single network resource.
So when you jstack and you see that regionserver has its threads all
stuck in next -- are they? -- then we are likely going to the local
datanode. Anything in its logs when regionserver slows down?
> next step will be to test with JD Cryans suggestion:
> " In order to completely rule out at least one thing, can you set ipc.server.max.queue.size to 1 and hbase.regionserver.handler.count to a low number (let's say 10)? If payload is putting too much memory pressure, we'll know."
>
> ...though I'm not sure what I'm supposed to observe with these settings...but I'll try it and report on the outcome.
>
Well, you have GC logging already. If you look at the output do you
see big pauses? I think J-D was thinking that regionservers would be
using less memory if you make the queues smaller. You could try that.
Maybe when queues are big, its taking a while to process them and
client times out. What size are these rows?
St.Ack
> -geoff
>
> -----Original Message-----
> From: Geoff Hendrey [mailto:ghendrey@decarta.com]
> Sent: Tuesday, September 13, 2011 4:50 PM
> To: user@hbase.apache.org; Andrew Purtell
> Cc: Tony Wang; Rohit Nigam; Parmod Mehta; James Ladd
> Subject: RE: scanner deadlock?
>
> 1019 sockets on 50010 in CLOSED_WAIT state.
>
> -geoff
>
> -----Original Message-----
> From: Andrew Purtell [mailto:apurtell@apache.org]
> Sent: Tuesday, September 13, 2011 4:00 PM
> To: user@hbase.apache.org
> Cc: Tony Wang; Rohit Nigam; Parmod Mehta; James Ladd
> Subject: Re: scanner deadlock?
>
>
>
>> My current working theory is that
>> too many sockets are in CLOSE_WAIT state (leading to
>> ClosedChannelException?). We're going to try to adjust some OS
>> parameters.
>
> How many sockets are in that state? netstat -an | grep CLOSE_WAIT | wc -l
>
> CDH3U1 contains HDFS-1836... https://issues.apache.org/jira/browse/HDFS-1836
>
> Best regards,
>
> Â Â Â Â Â Â Â - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
>
>
>>________________________________
>>From: Geoff Hendrey <gh...@decarta.com>
>>To: user@hbase.apache.org
>>Cc: Tony Wang <tw...@decarta.com>; Rohit Nigam <rn...@decarta.com>; Parmod Mehta <pm...@decarta.com>; James Ladd <jl...@decarta.com>
>>Sent: Tuesday, September 13, 2011 9:49 AM
>>Subject: RE: scanner deadlock?
>>
>>Thanks Stack -
>>
>>Answers to all your questions below. My current working theory is that
>>too many sockets are in CLOSE_WAIT state (leading to
>>ClosedChannelException?). We're going to try to adjust some OS
>>parameters.
>>
>>" I'm asking if regionservers are bottlenecking on a single network
>>resource; a particular datanode, dns?"
>>
>>Gotcha. I'm gathering some tools now to collect and analyze netstat
>>output.
>>
>>" the regionserver is going slow getting data out of
>>hdfs. Whats iowait like at the time of slowness? Has it changed from
>>when all was running nicely?"
>>
>>iowait is high (20% above cpu), but not increasing. I'll try to quantify
>>that better.
>>
>>" You talk to hbase in the reducer?   Reducers don't start writing hbase
>>until job is 66% complete IIRC.  Perhaps its slowing as soon as it
>>starts writing hbase? Is that so?"
>>
>>My statement about "running fine" applies to after the reducer has
>>completed sort. We have metrics produced by the reducer that log the
>>results of scans ant Puts. So we know that scans and puts proceed
>>without issue for hours.
>>
>
RE: scanner deadlock?
Posted by Geoff Hendrey <gh...@decarta.com>.
I've upgraded to HotSpot 64 Bit Server VM, with HBase 90.4 and all recommended config changes (100 region server handlers, mslab enabled, etc). No change, if anything it dies faster. Count of sockets in CLOSE_WAIT on 50010 increases linearly. I logged netstat from a random node in the cluster, periodically. Then dumped the output into excel using a pivot table to look at a behavior of TCP. Number of connections from the given node to others on 50010 was relatively uniform (no hotspot). Connections on 50010 from given node to *self* was much way higher than to other nodes, but that's probably a good thing. My guess is it's HBase leveraging locality of files for the region server. Just a guess.
next step will be to test with JD Cryans suggestion:
" In order to completely rule out at least one thing, can you set ipc.server.max.queue.size to 1 and hbase.regionserver.handler.count to a low number (let's say 10)? If payload is putting too much memory pressure, we'll know."
...though I'm not sure what I'm supposed to observe with these settings...but I'll try it and report on the outcome.
-geoff
-----Original Message-----
From: Geoff Hendrey [mailto:ghendrey@decarta.com]
Sent: Tuesday, September 13, 2011 4:50 PM
To: user@hbase.apache.org; Andrew Purtell
Cc: Tony Wang; Rohit Nigam; Parmod Mehta; James Ladd
Subject: RE: scanner deadlock?
1019 sockets on 50010 in CLOSED_WAIT state.
-geoff
-----Original Message-----
From: Andrew Purtell [mailto:apurtell@apache.org]
Sent: Tuesday, September 13, 2011 4:00 PM
To: user@hbase.apache.org
Cc: Tony Wang; Rohit Nigam; Parmod Mehta; James Ladd
Subject: Re: scanner deadlock?
> My current working theory is that
> too many sockets are in CLOSE_WAIT state (leading to
> ClosedChannelException?). We're going to try to adjust some OS
> parameters.
How many sockets are in that state? netstat -an | grep CLOSE_WAIT | wc -l
CDH3U1 contains HDFS-1836... https://issues.apache.org/jira/browse/HDFS-1836
Best regards,
       - Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
>________________________________
>From: Geoff Hendrey <gh...@decarta.com>
>To: user@hbase.apache.org
>Cc: Tony Wang <tw...@decarta.com>; Rohit Nigam <rn...@decarta.com>; Parmod Mehta <pm...@decarta.com>; James Ladd <jl...@decarta.com>
>Sent: Tuesday, September 13, 2011 9:49 AM
>Subject: RE: scanner deadlock?
>
>Thanks Stack -
>
>Answers to all your questions below. My current working theory is that
>too many sockets are in CLOSE_WAIT state (leading to
>ClosedChannelException?). We're going to try to adjust some OS
>parameters.
>
>" I'm asking if regionservers are bottlenecking on a single network
>resource; a particular datanode, dns?"
>
>Gotcha. I'm gathering some tools now to collect and analyze netstat
>output.
>
>" the regionserver is going slow getting data out of
>hdfs. Whats iowait like at the time of slowness? Has it changed from
>when all was running nicely?"
>
>iowait is high (20% above cpu), but not increasing. I'll try to quantify
>that better.
>
>" You talk to hbase in the reducer?   Reducers don't start writing hbase
>until job is 66% complete IIRC.  Perhaps its slowing as soon as it
>starts writing hbase? Is that so?"
>
>My statement about "running fine" applies to after the reducer has
>completed sort. We have metrics produced by the reducer that log the
>results of scans ant Puts. So we know that scans and puts proceed
>without issue for hours.
>Â
RE: scanner deadlock?
Posted by Geoff Hendrey <gh...@decarta.com>.
1019 sockets on 50010 in CLOSED_WAIT state.
-geoff
-----Original Message-----
From: Andrew Purtell [mailto:apurtell@apache.org]
Sent: Tuesday, September 13, 2011 4:00 PM
To: user@hbase.apache.org
Cc: Tony Wang; Rohit Nigam; Parmod Mehta; James Ladd
Subject: Re: scanner deadlock?
> My current working theory is that
> too many sockets are in CLOSE_WAIT state (leading to
> ClosedChannelException?). We're going to try to adjust some OS
> parameters.
How many sockets are in that state? netstat -an | grep CLOSE_WAIT | wc -l
CDH3U1 contains HDFS-1836... https://issues.apache.org/jira/browse/HDFS-1836
Best regards,
       - Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
>________________________________
>From: Geoff Hendrey <gh...@decarta.com>
>To: user@hbase.apache.org
>Cc: Tony Wang <tw...@decarta.com>; Rohit Nigam <rn...@decarta.com>; Parmod Mehta <pm...@decarta.com>; James Ladd <jl...@decarta.com>
>Sent: Tuesday, September 13, 2011 9:49 AM
>Subject: RE: scanner deadlock?
>
>Thanks Stack -
>
>Answers to all your questions below. My current working theory is that
>too many sockets are in CLOSE_WAIT state (leading to
>ClosedChannelException?). We're going to try to adjust some OS
>parameters.
>
>" I'm asking if regionservers are bottlenecking on a single network
>resource; a particular datanode, dns?"
>
>Gotcha. I'm gathering some tools now to collect and analyze netstat
>output.
>
>" the regionserver is going slow getting data out of
>hdfs. Whats iowait like at the time of slowness? Has it changed from
>when all was running nicely?"
>
>iowait is high (20% above cpu), but not increasing. I'll try to quantify
>that better.
>
>" You talk to hbase in the reducer?   Reducers don't start writing hbase
>until job is 66% complete IIRC.  Perhaps its slowing as soon as it
>starts writing hbase? Is that so?"
>
>My statement about "running fine" applies to after the reducer has
>completed sort. We have metrics produced by the reducer that log the
>results of scans ant Puts. So we know that scans and puts proceed
>without issue for hours.
>Â
Re: scanner deadlock?
Posted by Andrew Purtell <ap...@apache.org>.
> My current working theory is that
> too many sockets are in CLOSE_WAIT state (leading to
> ClosedChannelException?). We're going to try to adjust some OS
> parameters.
How many sockets are in that state? netstat -an | grep CLOSE_WAIT | wc -l
CDH3U1 contains HDFS-1836... https://issues.apache.org/jira/browse/HDFS-1836
Best regards,
       - Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
>________________________________
>From: Geoff Hendrey <gh...@decarta.com>
>To: user@hbase.apache.org
>Cc: Tony Wang <tw...@decarta.com>; Rohit Nigam <rn...@decarta.com>; Parmod Mehta <pm...@decarta.com>; James Ladd <jl...@decarta.com>
>Sent: Tuesday, September 13, 2011 9:49 AM
>Subject: RE: scanner deadlock?
>
>Thanks Stack -
>
>Answers to all your questions below. My current working theory is that
>too many sockets are in CLOSE_WAIT state (leading to
>ClosedChannelException?). We're going to try to adjust some OS
>parameters.
>
>" I'm asking if regionservers are bottlenecking on a single network
>resource; a particular datanode, dns?"
>
>Gotcha. I'm gathering some tools now to collect and analyze netstat
>output.
>
>" the regionserver is going slow getting data out of
>hdfs. Whats iowait like at the time of slowness? Has it changed from
>when all was running nicely?"
>
>iowait is high (20% above cpu), but not increasing. I'll try to quantify
>that better.
>
>" You talk to hbase in the reducer?   Reducers don't start writing hbase
>until job is 66% complete IIRC.  Perhaps its slowing as soon as it
>starts writing hbase? Is that so?"
>
>My statement about "running fine" applies to after the reducer has
>completed sort. We have metrics produced by the reducer that log the
>results of scans ant Puts. So we know that scans and puts proceed
>without issue for hours.
>Â
RE: scanner deadlock?
Posted by Geoff Hendrey <gh...@decarta.com>.
Thanks Stack -
Answers to all your questions below. My current working theory is that
too many sockets are in CLOSE_WAIT state (leading to
ClosedChannelException?). We're going to try to adjust some OS
parameters.
" I'm asking if regionservers are bottlenecking on a single network
resource; a particular datanode, dns?"
Gotcha. I'm gathering some tools now to collect and analyze netstat
output.
" the regionserver is going slow getting data out of
hdfs. Whats iowait like at the time of slowness? Has it changed from
when all was running nicely?"
iowait is high (20% above cpu), but not increasing. I'll try to quantify
that better.
" You talk to hbase in the reducer? Reducers don't start writing hbase
until job is 66% complete IIRC. Perhaps its slowing as soon as it
starts writing hbase? Is that so?"
My statement about "running fine" applies to after the reducer has
completed sort. We have metrics produced by the reducer that log the
results of scans ant Puts. So we know that scans and puts proceed
without issue for hours.
-----Original Message-----
From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of
Stack
Sent: Tuesday, September 13, 2011 9:32 AM
To: user@hbase.apache.org
Cc: Tony Wang; Rohit Nigam; Parmod Mehta; James Ladd
Subject: Re: scanner deadlock?
On Tue, Sep 13, 2011 at 8:20 AM, Geoff Hendrey <gh...@decarta.com>
wrote:
> ...but we don't have a slow region server.
I'm asking if regionservers are bottlenecking on a single network
resource; a particular datanode, dns?
Things hum along just fine.
> Suddenly, at roughly the same time, all the region servers begin
giving
> ScannerTimeoutException and ClosedChannel exception.
So, odd that its running fine then cluster slows.
>From the stacktraces you showed me -- and you might want to check
again and do a few stack traces to see that we are stuck trying to get
data from hdfs -- the regionserver is going slow getting data out of
hdfs. Whats iowait like at the time of slowness? Has it changed from
when all was running nicely?
> All the region
> servers are loaded in a pretty identical way by this MR job I am
> running. And they all begin showing the same error, at the same time,
> after performing perfectly for ~40% of the MR job.
>
You talk to hbase in the reducer? Reducers don't start writing hbase
until job is 66% complete IIRC. Perhaps its slowing as soon as it
starts writing hbase? Is that so?
> We have an ops team that monitors all these systems with Nagios.
They've
> reviewed dmsg, and many other low level details which are over my
head.
> In the past they've adjusted MTU's, and unbounded the network cards
(we
> saw some network stack lockups in the past, etc.). I'm going to meet
> with them again, and see if we can put setup some more specific
> monitoring around this job, which we can basically view as a test
> harness.
>
OK. Hopefully these lads can help
> That said, is there any condition that should cause HBase to get a
> ClosedChannelException, and *not* tell the zookeeper that it is
> effectively dead?
Well, sounds like regionserver is not dead. Its just crawling so its
still 'alive'.
St.Ack
Re: scanner deadlock?
Posted by Stack <st...@duboce.net>.
On Tue, Sep 13, 2011 at 8:20 AM, Geoff Hendrey <gh...@decarta.com> wrote:
> ...but we don't have a slow region server.
I'm asking if regionservers are bottlenecking on a single network
resource; a particular datanode, dns?
Things hum along just fine.
> Suddenly, at roughly the same time, all the region servers begin giving
> ScannerTimeoutException and ClosedChannel exception.
So, odd that its running fine then cluster slows.
>From the stacktraces you showed me -- and you might want to check
again and do a few stack traces to see that we are stuck trying to get
data from hdfs -- the regionserver is going slow getting data out of
hdfs. Whats iowait like at the time of slowness? Has it changed from
when all was running nicely?
> All the region
> servers are loaded in a pretty identical way by this MR job I am
> running. And they all begin showing the same error, at the same time,
> after performing perfectly for ~40% of the MR job.
>
You talk to hbase in the reducer? Reducers don't start writing hbase
until job is 66% complete IIRC. Perhaps its slowing as soon as it
starts writing hbase? Is that so?
> We have an ops team that monitors all these systems with Nagios. They've
> reviewed dmsg, and many other low level details which are over my head.
> In the past they've adjusted MTU's, and unbounded the network cards (we
> saw some network stack lockups in the past, etc.). I'm going to meet
> with them again, and see if we can put setup some more specific
> monitoring around this job, which we can basically view as a test
> harness.
>
OK. Hopefully these lads can help
> That said, is there any condition that should cause HBase to get a
> ClosedChannelException, and *not* tell the zookeeper that it is
> effectively dead?
Well, sounds like regionserver is not dead. Its just crawling so its
still 'alive'.
St.Ack
RE: scanner deadlock?
Posted by Geoff Hendrey <gh...@decarta.com>.
" Yeah. On the slow RS, check who its talking too... take a look at a
few of the nodes referenced."
...but we don't have a slow region server. Things hum along just fine.
Suddenly, at roughly the same time, all the region servers begin giving
ScannerTimeoutException and ClosedChannel exception. All the region
servers are loaded in a pretty identical way by this MR job I am
running. And they all begin showing the same error, at the same time,
after performing perfectly for ~40% of the MR job.
We have an ops team that monitors all these systems with Nagios. They've
reviewed dmsg, and many other low level details which are over my head.
In the past they've adjusted MTU's, and unbounded the network cards (we
saw some network stack lockups in the past, etc.). I'm going to meet
with them again, and see if we can put setup some more specific
monitoring around this job, which we can basically view as a test
harness.
That said, is there any condition that should cause HBase to get a
ClosedChannelException, and *not* tell the zookeeper that it is
effectively dead? It seems that regardless of the cause of the
ClosedChannelException, the region servers enter a state in which they
are useless, but the master never shuts them down. Consequently, the
client will start to see "NotServingRegionException" (sorry I forgot the
exact exception name).If I restart HBase while the MR job is in
progress, the problem of goes away...until it recurs in exactly the same
way as before. It seems to me like if there was a bad disk or a network
issue, that this issue would be present from the get go, rather than
suddenly happening on all RS, after some long period of loading. Anyway,
I am speculating. I will try to gather more facts.
Thanks for your ongoing help.
-geoff
-----Original Message-----
From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of
Stack
Sent: Monday, September 12, 2011 11:46 PM
To: user@hbase.apache.org
Cc: Tony Wang; Rohit Nigam; Parmod Mehta; James Ladd
Subject: Re: scanner deadlock?
On Mon, Sep 12, 2011 at 11:27 PM, Geoff Hendrey <gh...@decarta.com>
wrote:
> Do you have any advice on what to look for (or how to sort it) when I
do
> lsof or netstat? A glance at it doesn't show any "standouts" but then
> I'm not entirely sure what to look for. I see lots of connections to
> various nodes in the cluster, from any given node, but I suppose
that's
> quite normal.
Yeah. On the slow RS, check who its talking too... take a look at a
few of the nodes referenced.
Check dmesg across your cluster see if any complaining.
> Ganglia offers no clues either. It's pretty uniform for
> all graphs across all servers.
>
No anomalies around datanodes? Spikes or troughs?
Welcome to the joys of distributed computing. Once you figure whats
going on, you'll be able to enable an alert for the future but
meantime its no fun.
St.Ack
Re: scanner deadlock?
Posted by Stack <st...@duboce.net>.
On Mon, Sep 12, 2011 at 11:27 PM, Geoff Hendrey <gh...@decarta.com> wrote:
> Do you have any advice on what to look for (or how to sort it) when I do
> lsof or netstat? A glance at it doesn't show any "standouts" but then
> I'm not entirely sure what to look for. I see lots of connections to
> various nodes in the cluster, from any given node, but I suppose that's
> quite normal.
Yeah. On the slow RS, check who its talking too... take a look at a
few of the nodes referenced.
Check dmesg across your cluster see if any complaining.
> Ganglia offers no clues either. It's pretty uniform for
> all graphs across all servers.
>
No anomalies around datanodes? Spikes or troughs?
Welcome to the joys of distributed computing. Once you figure whats
going on, you'll be able to enable an alert for the future but
meantime its no fun.
St.Ack
RE: scanner deadlock?
Posted by Geoff Hendrey <gh...@decarta.com>.
Hi Stack -
Do you have any advice on what to look for (or how to sort it) when I do
lsof or netstat? A glance at it doesn't show any "standouts" but then
I'm not entirely sure what to look for. I see lots of connections to
various nodes in the cluster, from any given node, but I suppose that's
quite normal. Ganglia offers no clues either. It's pretty uniform for
all graphs across all servers.
-geoff
-----Original Message-----
From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of
Stack
Sent: Monday, September 12, 2011 9:12 PM
To: user@hbase.apache.org
Cc: Tony Wang; Rohit Nigam; Parmod Mehta; James Ladd
Subject: Re: scanner deadlock?
No slow datanode in your cluster?
When stuff is slow, can you figure who all are trying to talk to?
St.Ack
On Mon, Sep 12, 2011 at 8:37 PM, Geoff Hendrey <gh...@decarta.com>
wrote:
> OK Guys -
>
> We upgraded to 90.4, and made all the suggested config changes. The
only
> thing we have not done yet, but will try soon, is switching from
OpenJDK
> to the HotSpot JVM. Unfortunately, the problem recurs exactly as
before.
> We will test with the HotSpot JVM shortly.
>
> -geoff
>
> -----Original Message-----
> From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of
> Jean-Daniel Cryans
> Sent: Monday, September 12, 2011 11:44 AM
> To: user@hbase.apache.org
> Subject: Re: scanner deadlock?
>
>> I thought that as long as I specified neither -client nor -server,
> that
>> Server Class detection would automatically invoke the "-server"
> option.
>>
>>
>
http://download.oracle.com/javase/6/docs/technotes/guides/vm/server-clas
>> s.html
>>
>> We are running 12-core AMD Opteron which is AMD64, so according to
the
>> guide above, -server is selected automatically. Please let me know if
>> I've misunderstood this. We *definitely* want to be running hotspot!
>
> It's two different JVMs, not a matter of using -client or -server
> (which are just different configurations). What you are running is:
>
> http://openjdk.java.net/
>
> What most people run is:
>
> http://www.oracle.com/us/technologies/java/index.html
>
>>
>> Regarding GC: we are generating GC logs for namenode, datanode,
master
>> and regionserver. We do see long GC from time to time. In fact, I
> played
>> with the mslab option, but didn't find significant improvement. We've
>> seen times on the order of a minute in these logs, and have found no
> way
>> around it (spent countless days and nights experimenting with
> different
>> GC parameters, mslab, different heap sizes, etc).
>
> Sometimes it's just a matter of how much data you have in flight.
> That's why I mentioned scanner pre-caching (set via Scan.setCaching),
> because it can potentially load a lot of rows into the RS's heap. More
> concurrent scanners means also more data loaded into memory.
>
> Are you also inserting at the same time? What's your write buffer
size?
>
> The discussion in this jira could be relevant:
> https://issues.apache.org/jira/browse/HBASE-3813
>
> A temporary fix got committed in 0.90.3 to make
> ipc.server.max.queue.size configurable.
>
> J-D
>
Re: scanner deadlock?
Posted by Stack <st...@duboce.net>.
No slow datanode in your cluster?
When stuff is slow, can you figure who all are trying to talk to?
St.Ack
On Mon, Sep 12, 2011 at 8:37 PM, Geoff Hendrey <gh...@decarta.com> wrote:
> OK Guys -
>
> We upgraded to 90.4, and made all the suggested config changes. The only
> thing we have not done yet, but will try soon, is switching from OpenJDK
> to the HotSpot JVM. Unfortunately, the problem recurs exactly as before.
> We will test with the HotSpot JVM shortly.
>
> -geoff
>
> -----Original Message-----
> From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of
> Jean-Daniel Cryans
> Sent: Monday, September 12, 2011 11:44 AM
> To: user@hbase.apache.org
> Subject: Re: scanner deadlock?
>
>> I thought that as long as I specified neither -client nor -server,
> that
>> Server Class detection would automatically invoke the "-server"
> option.
>>
>>
> http://download.oracle.com/javase/6/docs/technotes/guides/vm/server-clas
>> s.html
>>
>> We are running 12-core AMD Opteron which is AMD64, so according to the
>> guide above, -server is selected automatically. Please let me know if
>> I've misunderstood this. We *definitely* want to be running hotspot!
>
> It's two different JVMs, not a matter of using -client or -server
> (which are just different configurations). What you are running is:
>
> http://openjdk.java.net/
>
> What most people run is:
>
> http://www.oracle.com/us/technologies/java/index.html
>
>>
>> Regarding GC: we are generating GC logs for namenode, datanode, master
>> and regionserver. We do see long GC from time to time. In fact, I
> played
>> with the mslab option, but didn't find significant improvement. We've
>> seen times on the order of a minute in these logs, and have found no
> way
>> around it (spent countless days and nights experimenting with
> different
>> GC parameters, mslab, different heap sizes, etc).
>
> Sometimes it's just a matter of how much data you have in flight.
> That's why I mentioned scanner pre-caching (set via Scan.setCaching),
> because it can potentially load a lot of rows into the RS's heap. More
> concurrent scanners means also more data loaded into memory.
>
> Are you also inserting at the same time? What's your write buffer size?
>
> The discussion in this jira could be relevant:
> https://issues.apache.org/jira/browse/HBASE-3813
>
> A temporary fix got committed in 0.90.3 to make
> ipc.server.max.queue.size configurable.
>
> J-D
>
RE: scanner deadlock?
Posted by Geoff Hendrey <gh...@decarta.com>.
certainly. I'll let you know.
-geoff
-----Original Message-----
From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of
Jean-Daniel Cryans
Sent: Monday, September 12, 2011 8:42 PM
To: user@hbase.apache.org
Subject: Re: scanner deadlock?
In order to completely rule out at least one thing, can you set
ipc.server.max.queue.size to 1 and hbase.regionserver.handler.count to
a low number (let's say 10)? If payload is putting too much memory
pressure, we'll know.
J-D
On Mon, Sep 12, 2011 at 8:37 PM, Geoff Hendrey <gh...@decarta.com>
wrote:
> OK Guys -
>
> We upgraded to 90.4, and made all the suggested config changes. The
only
> thing we have not done yet, but will try soon, is switching from
OpenJDK
> to the HotSpot JVM. Unfortunately, the problem recurs exactly as
before.
> We will test with the HotSpot JVM shortly.
>
> -geoff
>
> -----Original Message-----
> From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of
> Jean-Daniel Cryans
> Sent: Monday, September 12, 2011 11:44 AM
> To: user@hbase.apache.org
> Subject: Re: scanner deadlock?
>
>> I thought that as long as I specified neither -client nor -server,
> that
>> Server Class detection would automatically invoke the "-server"
> option.
>>
>>
>
http://download.oracle.com/javase/6/docs/technotes/guides/vm/server-clas
>> s.html
>>
>> We are running 12-core AMD Opteron which is AMD64, so according to
the
>> guide above, -server is selected automatically. Please let me know if
>> I've misunderstood this. We *definitely* want to be running hotspot!
>
> It's two different JVMs, not a matter of using -client or -server
> (which are just different configurations). What you are running is:
>
> http://openjdk.java.net/
>
> What most people run is:
>
> http://www.oracle.com/us/technologies/java/index.html
>
>>
>> Regarding GC: we are generating GC logs for namenode, datanode,
master
>> and regionserver. We do see long GC from time to time. In fact, I
> played
>> with the mslab option, but didn't find significant improvement. We've
>> seen times on the order of a minute in these logs, and have found no
> way
>> around it (spent countless days and nights experimenting with
> different
>> GC parameters, mslab, different heap sizes, etc).
>
> Sometimes it's just a matter of how much data you have in flight.
> That's why I mentioned scanner pre-caching (set via Scan.setCaching),
> because it can potentially load a lot of rows into the RS's heap. More
> concurrent scanners means also more data loaded into memory.
>
> Are you also inserting at the same time? What's your write buffer
size?
>
> The discussion in this jira could be relevant:
> https://issues.apache.org/jira/browse/HBASE-3813
>
> A temporary fix got committed in 0.90.3 to make
> ipc.server.max.queue.size configurable.
>
> J-D
>
Re: scanner deadlock?
Posted by Jean-Daniel Cryans <jd...@apache.org>.
In order to completely rule out at least one thing, can you set
ipc.server.max.queue.size to 1 and hbase.regionserver.handler.count to
a low number (let's say 10)? If payload is putting too much memory
pressure, we'll know.
J-D
On Mon, Sep 12, 2011 at 8:37 PM, Geoff Hendrey <gh...@decarta.com> wrote:
> OK Guys -
>
> We upgraded to 90.4, and made all the suggested config changes. The only
> thing we have not done yet, but will try soon, is switching from OpenJDK
> to the HotSpot JVM. Unfortunately, the problem recurs exactly as before.
> We will test with the HotSpot JVM shortly.
>
> -geoff
>
> -----Original Message-----
> From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of
> Jean-Daniel Cryans
> Sent: Monday, September 12, 2011 11:44 AM
> To: user@hbase.apache.org
> Subject: Re: scanner deadlock?
>
>> I thought that as long as I specified neither -client nor -server,
> that
>> Server Class detection would automatically invoke the "-server"
> option.
>>
>>
> http://download.oracle.com/javase/6/docs/technotes/guides/vm/server-clas
>> s.html
>>
>> We are running 12-core AMD Opteron which is AMD64, so according to the
>> guide above, -server is selected automatically. Please let me know if
>> I've misunderstood this. We *definitely* want to be running hotspot!
>
> It's two different JVMs, not a matter of using -client or -server
> (which are just different configurations). What you are running is:
>
> http://openjdk.java.net/
>
> What most people run is:
>
> http://www.oracle.com/us/technologies/java/index.html
>
>>
>> Regarding GC: we are generating GC logs for namenode, datanode, master
>> and regionserver. We do see long GC from time to time. In fact, I
> played
>> with the mslab option, but didn't find significant improvement. We've
>> seen times on the order of a minute in these logs, and have found no
> way
>> around it (spent countless days and nights experimenting with
> different
>> GC parameters, mslab, different heap sizes, etc).
>
> Sometimes it's just a matter of how much data you have in flight.
> That's why I mentioned scanner pre-caching (set via Scan.setCaching),
> because it can potentially load a lot of rows into the RS's heap. More
> concurrent scanners means also more data loaded into memory.
>
> Are you also inserting at the same time? What's your write buffer size?
>
> The discussion in this jira could be relevant:
> https://issues.apache.org/jira/browse/HBASE-3813
>
> A temporary fix got committed in 0.90.3 to make
> ipc.server.max.queue.size configurable.
>
> J-D
>
RE: scanner deadlock?
Posted by Geoff Hendrey <gh...@decarta.com>.
OK Guys -
We upgraded to 90.4, and made all the suggested config changes. The only
thing we have not done yet, but will try soon, is switching from OpenJDK
to the HotSpot JVM. Unfortunately, the problem recurs exactly as before.
We will test with the HotSpot JVM shortly.
-geoff
-----Original Message-----
From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of
Jean-Daniel Cryans
Sent: Monday, September 12, 2011 11:44 AM
To: user@hbase.apache.org
Subject: Re: scanner deadlock?
> I thought that as long as I specified neither -client nor -server,
that
> Server Class detection would automatically invoke the "-server"
option.
>
>
http://download.oracle.com/javase/6/docs/technotes/guides/vm/server-clas
> s.html
>
> We are running 12-core AMD Opteron which is AMD64, so according to the
> guide above, -server is selected automatically. Please let me know if
> I've misunderstood this. We *definitely* want to be running hotspot!
It's two different JVMs, not a matter of using -client or -server
(which are just different configurations). What you are running is:
http://openjdk.java.net/
What most people run is:
http://www.oracle.com/us/technologies/java/index.html
>
> Regarding GC: we are generating GC logs for namenode, datanode, master
> and regionserver. We do see long GC from time to time. In fact, I
played
> with the mslab option, but didn't find significant improvement. We've
> seen times on the order of a minute in these logs, and have found no
way
> around it (spent countless days and nights experimenting with
different
> GC parameters, mslab, different heap sizes, etc).
Sometimes it's just a matter of how much data you have in flight.
That's why I mentioned scanner pre-caching (set via Scan.setCaching),
because it can potentially load a lot of rows into the RS's heap. More
concurrent scanners means also more data loaded into memory.
Are you also inserting at the same time? What's your write buffer size?
The discussion in this jira could be relevant:
https://issues.apache.org/jira/browse/HBASE-3813
A temporary fix got committed in 0.90.3 to make
ipc.server.max.queue.size configurable.
J-D
RE: scanner deadlock?
Posted by Sandy Pratt <pr...@adobe.com>.
I see the advice to use Oracle's JDK over OpenJDK periodically on this list, but I've never seen any concrete reasons for this mentioned. The HBase book doesn't mention any from what I can tell (http://hbase.apache.org/book.html#java). My understanding is that OpenJDK is a backport of Oracle JDK7 with a few components that are completely orthogonal to HBase stripped out (fonts, sound mixer, etc.) [http://www.infoq.com/news/2009/01/jdk-openjdk-icedtea]. Is there something introduced with in the JDK7 code base that could cause compatibility issues?
TLDR: OpenJDK ~= Oracle JDK, so why not use it?
Sandy
(FWIW I have a cluster running on OpenJDK currently so it's not entirely academic from my perspective.)
> -----Original Message-----
> From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of Jean-
> Daniel Cryans
> Sent: Monday, September 12, 2011 11:44
> To: user@hbase.apache.org
> Subject: Re: scanner deadlock?
>
> > I thought that as long as I specified neither -client nor -server,
> > that Server Class detection would automatically invoke the "-server"
> option.
> >
> > http://download.oracle.com/javase/6/docs/technotes/guides/vm/server-
> cl
> > as
> > s.html
> >
> > We are running 12-core AMD Opteron which is AMD64, so according to the
> > guide above, -server is selected automatically. Please let me know if
> > I've misunderstood this. We *definitely* want to be running hotspot!
>
> It's two different JVMs, not a matter of using -client or -server (which are just
> different configurations). What you are running is:
>
> http://openjdk.java.net/
>
> What most people run is:
>
> http://www.oracle.com/us/technologies/java/index.html
>
Re: scanner deadlock?
Posted by Jean-Daniel Cryans <jd...@apache.org>.
> I thought that as long as I specified neither -client nor -server, that
> Server Class detection would automatically invoke the "-server" option.
>
> http://download.oracle.com/javase/6/docs/technotes/guides/vm/server-clas
> s.html
>
> We are running 12-core AMD Opteron which is AMD64, so according to the
> guide above, -server is selected automatically. Please let me know if
> I've misunderstood this. We *definitely* want to be running hotspot!
It's two different JVMs, not a matter of using -client or -server
(which are just different configurations). What you are running is:
http://openjdk.java.net/
What most people run is:
http://www.oracle.com/us/technologies/java/index.html
>
> Regarding GC: we are generating GC logs for namenode, datanode, master
> and regionserver. We do see long GC from time to time. In fact, I played
> with the mslab option, but didn't find significant improvement. We've
> seen times on the order of a minute in these logs, and have found no way
> around it (spent countless days and nights experimenting with different
> GC parameters, mslab, different heap sizes, etc).
Sometimes it's just a matter of how much data you have in flight.
That's why I mentioned scanner pre-caching (set via Scan.setCaching),
because it can potentially load a lot of rows into the RS's heap. More
concurrent scanners means also more data loaded into memory.
Are you also inserting at the same time? What's your write buffer size?
The discussion in this jira could be relevant:
https://issues.apache.org/jira/browse/HBASE-3813
A temporary fix got committed in 0.90.3 to make
ipc.server.max.queue.size configurable.
J-D
RE: scanner deadlock?
Posted by Geoff Hendrey <gh...@decarta.com>.
Hi -
I thought that as long as I specified neither -client nor -server, that
Server Class detection would automatically invoke the "-server" option.
http://download.oracle.com/javase/6/docs/technotes/guides/vm/server-clas
s.html
We are running 12-core AMD Opteron which is AMD64, so according to the
guide above, -server is selected automatically. Please let me know if
I've misunderstood this. We *definitely* want to be running hotspot!
Regarding GC: we are generating GC logs for namenode, datanode, master
and regionserver. We do see long GC from time to time. In fact, I played
with the mslab option, but didn't find significant improvement. We've
seen times on the order of a minute in these logs, and have found no way
around it (spent countless days and nights experimenting with different
GC parameters, mslab, different heap sizes, etc).
-geoff
-----Original Message-----
From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of
Jean-Daniel Cryans
Sent: Monday, September 12, 2011 10:55 AM
To: user@hbase.apache.org
Subject: Re: scanner deadlock?
Two other thoughts to add on top of the rest:
- You really should also consider using HotSpot instead of OpenJDK.
- Depending on the size of your RS's heap, the number of concurrent
scanners and if they are doing pre-caching, you may be just GCing like
mad. Did you check that?
J-D
On Mon, Sep 12, 2011 at 10:42 AM, Stack <st...@duboce.net> wrote:
> What Eric says.
>
> Sun java can find deadlocks too, the dumb ones at least.
>
> And then Todd likes this for the same job: http://www.jcarder.org/
>
> St.Ack
Re: scanner deadlock?
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Two other thoughts to add on top of the rest:
- You really should also consider using HotSpot instead of OpenJDK.
- Depending on the size of your RS's heap, the number of concurrent
scanners and if they are doing pre-caching, you may be just GCing like
mad. Did you check that?
J-D
On Mon, Sep 12, 2011 at 10:42 AM, Stack <st...@duboce.net> wrote:
> What Eric says.
>
> Sun java can find deadlocks too, the dumb ones at least.
>
> And then Todd likes this for the same job: http://www.jcarder.org/
>
> St.Ack
Re: scanner deadlock?
Posted by Stack <st...@duboce.net>.
What Eric says.
Sun java can find deadlocks too, the dumb ones at least.
And then Todd likes this for the same job: http://www.jcarder.org/
St.Ack
On Mon, Sep 12, 2011 at 10:34 AM, Geoff Hendrey <gh...@decarta.com> wrote:
> Wow, no I had not been using any tool other than eyeballs. I will
> certainly apply this tool to our jstack.
>
> Thanks!!
>
> -geoff
>
> -----Original Message-----
> From: Eric Charles [mailto:eric.umg.charles@gmail.com]
> Sent: Monday, September 12, 2011 10:28 AM
> To: user@hbase.apache.org
> Subject: Re: scanner deadlock?
>
> Hi,
>
> Probably you've already done something similar, but I happily use 'IBM
> Thread and Monitor Dump Analyzer for Java' to be 100% sure there are no
> dead lock. The tool gives you a nice short report on your threads.
>
> I tried to run it on your dump, but the analyzer does not like
> mail-wrapped lines.
>
> Thx.
>
> [1]
> https://www.ibm.com/developerworks/mydeveloperworks/groups/service/html/
> communityview?communityUuid=2245aa39-fa5c-4475-b891-14c205f7333c
>
> On 12/09/11 09:46, Stack wrote:
>> On Sun, Sep 11, 2011 at 2:40 PM, Geoff Hendrey<gh...@decarta.com>
> wrote:
>>> Hi,
>>>
>>> The cluster is definitely loaded. We are running a MR job with 130
>>> reducers, each using a scanner. Ganglia shows an overall CPU
> utilization
>>> of around 40%, with network pushing 800MB/s. The problem may
> correlate
>>> with a major_compact I initiated couple days ago. Even today, I still
>>> see a lot of compacting (continuous) on region servers.
>>>
>>> Here is the thread dump from jstack. It does not reveal any explicit
>>> deadlock. At the time I dumped the jstack, the region server logs are
>>> reporting "IPC Server handler...ClosedChannelException". As soon as
> this
>>> begins occurring, our network throughput, as seen on Ganglia drops
> from
>>> 800MB/s down to virtually zero. Then of course there is the cascade
> of
>>> failures, eventually leading to the regionservers shutting down.
>>>
>>
>>
>> Looking at the thread dump, as you say, few are 'BLOCKED' (There are a
>> few BLOCKED thread trying to go into a next). Â But most are deep in a
>> next stuck polling looking to read. Â If you thread dump a few times
>> in a row, do any seem to be making progress? Â (They may be making
>> progress but slowly, as you say)
>>
>> So, is it possible you have a slow datanode in your cluster? Â Can you
>> tell who all are connected too... lsof'ing?  That  might give you a
>> clue. Â Perhaps there is a node with sick disks (check dmesg). Â Or
>> perhaps the machine's nework card is sick?
>>
>> St.Ack
>>
>>> I don't see anything unusual in the datanode logs on the same server.
>>> Just HDFS_READ and HDFS_WRITE for the most part.
>>>
>>> jstack here:
>>> ===================================================================
>>> 2011-09-11 14:28:52
>>> Full thread dump OpenJDK 64-Bit Server VM (14.0-b16 mixed mode):
>>>
>>> "IPC Client (47) connection to
>>> doop10.dt.sv4.decarta.com/10.241.8.230:54310 from hroot" daemon
> prio=10
>>> tid=0x0000000001492800 nid=0x75cb in Object.wait()
> [0x0000000047633000]
>>> Â Â java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>> Â Â Â Â at java.lang.Object.wait(Native Method)
>>> Â Â Â Â at
>>> org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:676)
>>> Â Â Â Â - locked<0x00002aaaaf680000> Â (a
>>> org.apache.hadoop.ipc.Client$Connection)
>>> Â Â Â Â at
> org.apache.hadoop.ipc.Client$Connection.run(Client.java:719)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "ResponseProcessor for block blk_2613854253703670198_5032689" daemon
>>> prio=10 tid=0x0000000001988800 nid=0x75bb runnable
> [0x0000000046926000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> Â Â Â Â at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>> Â Â Â Â at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>> Â Â Â Â at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> Â Â Â Â - locked<0x00002aab69337248> Â (a sun.nio.ch.Util$1)
>>> Â Â Â Â - locked<0x00002aab69337230> Â (a
>>> java.util.Collections$UnmodifiableSet)
>>> Â Â Â Â - locked<0x00002aaaf3897930> Â (a
> sun.nio.ch.EPollSelectorImpl)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>> thTimeout.java:332)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>> 157)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>> Â Â Â Â at
> java.io.DataInputStream.readFully(DataInputStream.java:195)
>>> Â Â Â Â at java.io.DataInputStream.readLong(DataInputStream.java:416)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFie
>>> lds(DataTransferProtocol.java:120)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(D
>>> FSClient.java:2638)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "DataStreamer for file
>>>
> /hbase-20.6/NAM_CLUSTERKEYS3/60db9e0113c7578ed913dc37e0856954/.tmp/42977
>>> 03314038441248 block blk_2613854253703670198_5032689" daemon prio=10
>>> tid=0x00002aae40e8e800 nid=0x75b8 in Object.wait()
> [0x0000000046e2b000]
>>> Â Â java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>> Â Â Â Â at java.lang.Object.wait(Native Method)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSCli
>>> ent.java:2483)
>>> Â Â Â Â - locked<0x00002aab6551a2d8> Â (a java.util.LinkedList)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "ResponseProcessor for block blk_8900384789253382699_5032549" daemon
>>> prio=10 tid=0x00000000013cd000 nid=0x71d0 runnable
> [0x0000000047431000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> Â Â Â Â at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>> Â Â Â Â at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>> Â Â Â Â at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> Â Â Â Â - locked<0x00002aaae0569530> Â (a sun.nio.ch.Util$1)
>>> Â Â Â Â - locked<0x00002aaae0569518> Â (a
>>> java.util.Collections$UnmodifiableSet)
>>> Â Â Â Â - locked<0x00002aaadb4c5c68> Â (a
> sun.nio.ch.EPollSelectorImpl)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>> thTimeout.java:332)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>> 157)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>> Â Â Â Â at
> java.io.DataInputStream.readFully(DataInputStream.java:195)
>>> Â Â Â Â at java.io.DataInputStream.readLong(DataInputStream.java:416)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFie
>>> lds(DataTransferProtocol.java:120)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(D
>>> FSClient.java:2638)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "DataStreamer for file
>>>
> /hbase-20.6/.logs/doop5.dt.sv4.decarta.com,60020,1315768059645/doop5.dt.
>>> sv4.decarta.com%3A60020.1315776307423 block
>>> blk_8900384789253382699_5032549" daemon prio=10
> tid=0x00000000016c3800
>>> nid=0x71c8 in Object.wait() [0x000000004712e000]
>>> Â Â java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>> Â Â Â Â at java.lang.Object.wait(Native Method)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSCli
>>> ent.java:2483)
>>> Â Â Â Â - locked<0x00002aab092a7f38> Â (a java.util.LinkedList)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "Attach Listener" daemon prio=10 tid=0x0000000001e32000 nid=0x50f4
>>> waiting on condition [0x0000000000000000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "RS_CLOSE_REGION-doop5.dt.sv4.decarta.com,60020,1315768059645-2"
> daemon
>>> prio=10 tid=0x00000000018cb800 nid=0x505 waiting on condition
>>> [0x0000000047330000]
>>> Â Â java.lang.Thread.State: WAITING (parking)
>>> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>>> Â Â Â Â - parking to wait for<0x00002aaabcefb428> Â (a
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> Â Â Â Â at
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>> Â Â Â Â at
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>> ait(AbstractQueuedSynchronizer.java:2043)
>>> Â Â Â Â at
>>>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>>> 86)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
>>> 1043)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>> a:1103)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>> va:603)
>>> Â Â Â Â at java.lang.Thread.run(Thread.java:636)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "RS_CLOSE_REGION-doop5.dt.sv4.decarta.com,60020,1315768059645-1"
> daemon
>>> prio=10 tid=0x000000000163e000 nid=0x504 waiting on condition
>>> [0x0000000047532000]
>>> Â Â java.lang.Thread.State: WAITING (parking)
>>> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>>> Â Â Â Â - parking to wait for<0x00002aaabcefb428> Â (a
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> Â Â Â Â at
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>> Â Â Â Â at
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>> ait(AbstractQueuedSynchronizer.java:2043)
>>> Â Â Â Â at
>>>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>>> 86)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
>>> 1043)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>> a:1103)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>> va:603)
>>> Â Â Â Â at java.lang.Thread.run(Thread.java:636)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "RS_CLOSE_REGION-doop5.dt.sv4.decarta.com,60020,1315768059645-0"
> daemon
>>> prio=10 tid=0x0000000001bff800 nid=0x503 waiting on condition
>>> [0x0000000046f2c000]
>>> Â Â java.lang.Thread.State: WAITING (parking)
>>> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>>> Â Â Â Â - parking to wait for<0x00002aaabcefb428> Â (a
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> Â Â Â Â at
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>> Â Â Â Â at
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>> ait(AbstractQueuedSynchronizer.java:2043)
>>> Â Â Â Â at
>>>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>>> 86)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
>>> 1043)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>> a:1103)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>> va:603)
>>> Â Â Â Â at java.lang.Thread.run(Thread.java:636)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "sendParams-6" daemon prio=10 tid=0x00002aae401f1800 nid=0x6d3b
> waiting
>>> on condition [0x000000004702d000]
>>> Â Â java.lang.Thread.State: TIMED_WAITING (parking)
>>> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>>> Â Â Â Â - parking to wait for<0x00002aaabceea260> Â (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>> Â Â Â Â at
>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>>> Â Â Â Â at
>>>
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Synchro
>>> nousQueue.java:453)
>>> Â Â Â Â at
>>>
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(Synchronous
>>> Queue.java:352)
>>> Â Â Â Â at
>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:903)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
>>> 1043)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>> a:1103)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>> va:603)
>>> Â Â Â Â at java.lang.Thread.run(Thread.java:636)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "RS_OPEN_REGION-doop5.dt.sv4.decarta.com,60020,1315768059645-2"
> daemon
>>> prio=10 tid=0x0000000001a7f000 nid=0x5b56 waiting on condition
>>> [0x0000000046d2a000]
>>> Â Â java.lang.Thread.State: WAITING (parking)
>>> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>>> Â Â Â Â - parking to wait for<0x00002aaabcefb2e8> Â (a
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> Â Â Â Â at
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>> Â Â Â Â at
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>> ait(AbstractQueuedSynchronizer.java:2043)
>>> Â Â Â Â at
>>>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>>> 86)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
>>> 1043)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>> a:1103)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>> va:603)
>>> Â Â Â Â at java.lang.Thread.run(Thread.java:636)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "RS_OPEN_REGION-doop5.dt.sv4.decarta.com,60020,1315768059645-1"
> daemon
>>> prio=10 tid=0x0000000001af2800 nid=0x5b55 waiting on condition
>>> [0x0000000042deb000]
>>> Â Â java.lang.Thread.State: WAITING (parking)
>>> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>>> Â Â Â Â - parking to wait for<0x00002aaabcefb2e8> Â (a
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> Â Â Â Â at
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>> Â Â Â Â at
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>> ait(AbstractQueuedSynchronizer.java:2043)
>>> Â Â Â Â at
>>>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>>> 86)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
>>> 1043)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>> a:1103)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>> va:603)
>>> Â Â Â Â at java.lang.Thread.run(Thread.java:636)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "RS_OPEN_REGION-doop5.dt.sv4.decarta.com,60020,1315768059645-0"
> daemon
>>> prio=10 tid=0x00000000014b3800 nid=0x5b54 waiting on condition
>>> [0x0000000046c29000]
>>> Â Â java.lang.Thread.State: WAITING (parking)
>>> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>>> Â Â Â Â - parking to wait for<0x00002aaabcefb2e8> Â (a
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> Â Â Â Â at
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>> Â Â Â Â at
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>> ait(AbstractQueuedSynchronizer.java:2043)
>>> Â Â Â Â at
>>>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>>> 86)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
>>> 1043)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>> a:1103)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>> va:603)
>>> Â Â Â Â at java.lang.Thread.run(Thread.java:636)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "RS_OPEN_ROOT-doop5.dt.sv4.decarta.com,60020,1315768059645-0" daemon
>>> prio=10 tid=0x0000000001ad2800 nid=0x556e waiting on condition
>>> [0x0000000046a27000]
>>> Â Â java.lang.Thread.State: WAITING (parking)
>>> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>>> Â Â Â Â - parking to wait for<0x00002aaabcee8870> Â (a
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> Â Â Â Â at
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>> Â Â Â Â at
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>> ait(AbstractQueuedSynchronizer.java:2043)
>>> Â Â Â Â at
>>>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>>> 86)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
>>> 1043)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>> a:1103)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>> va:603)
>>> Â Â Â Â at java.lang.Thread.run(Thread.java:636)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "LRU Statistics #0" prio=10 tid=0x00002aae4048a800 nid=0x5566 waiting
> on
>>> condition [0x0000000046825000]
>>> Â Â java.lang.Thread.State: TIMED_WAITING (parking)
>>> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>>> Â Â Â Â - parking to wait for<0x00002aaabd0695c0> Â (a
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> Â Â Â Â at
>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>>> Â Â Â Â at
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>> aitNanos(AbstractQueuedSynchronizer.java:2081)
>>> Â Â Â Â at java.util.concurrent.DelayQueue.take(DelayQueue.java:193)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(S
>>> cheduledThreadPoolExecutor.java:688)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(S
>>> cheduledThreadPoolExecutor.java:681)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
>>> 1043)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>> a:1103)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>> va:603)
>>> Â Â Â Â at java.lang.Thread.run(Thread.java:636)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "LruBlockCache.EvictionThread" daemon prio=10 tid=0x00002aae4049f800
>>> nid=0x5565 in Object.wait() [0x0000000046724000]
>>> Â Â java.lang.Thread.State: WAITING (on object monitor)
>>> Â Â Â Â at java.lang.Object.wait(Native Method)
>>> Â Â Â Â at java.lang.Object.wait(Object.java:502)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlo
>>> ckCache.java:519)
>>> Â Â Â Â - locked<0x00002aaabce92f50> Â (a
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "PRI IPC Server handler 9 on 60020" daemon prio=10
>>> tid=0x00002aae4057e000 nid=0x5564 waiting on condition
>>> [0x0000000046623000]
>>> Â Â java.lang.Thread.State: WAITING (parking)
>>> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>>> Â Â Â Â - parking to wait for<0x00002aaabbb307e0> Â (a
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> Â Â Â Â at
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>> Â Â Â Â at
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>> ait(AbstractQueuedSynchronizer.java:2043)
>>> Â Â Â Â at
>>>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>>> 86)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
>>> 5)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "PRI IPC Server handler 8 on 60020" daemon prio=10
>>> tid=0x00002aae4057c800 nid=0x5563 waiting on condition
>>> [0x0000000046522000]
>>> Â Â java.lang.Thread.State: WAITING (parking)
>>> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>>> Â Â Â Â - parking to wait for<0x00002aaabbb307e0> Â (a
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> Â Â Â Â at
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>> Â Â Â Â at
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>> ait(AbstractQueuedSynchronizer.java:2043)
>>> Â Â Â Â at
>>>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>>> 86)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
>>> 5)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "PRI IPC Server handler 7 on 60020" daemon prio=10
>>> tid=0x00002aae40559000 nid=0x5562 waiting on condition
>>> [0x0000000046421000]
>>> Â Â java.lang.Thread.State: WAITING (parking)
>>> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>>> Â Â Â Â - parking to wait for<0x00002aaabbb307e0> Â (a
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> Â Â Â Â at
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>> Â Â Â Â at
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>> ait(AbstractQueuedSynchronizer.java:2043)
>>> Â Â Â Â at
>>>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>>> 86)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
>>> 5)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "PRI IPC Server handler 6 on 60020" daemon prio=10
>>> tid=0x00002aae40557800 nid=0x5561 waiting on condition
>>> [0x0000000046320000]
>>> Â Â java.lang.Thread.State: WAITING (parking)
>>> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>>> Â Â Â Â - parking to wait for<0x00002aaabbb307e0> Â (a
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> Â Â Â Â at
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>> Â Â Â Â at
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>> ait(AbstractQueuedSynchronizer.java:2043)
>>> Â Â Â Â at
>>>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>>> 86)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
>>> 5)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "PRI IPC Server handler 5 on 60020" daemon prio=10
>>> tid=0x00002aae40555800 nid=0x5560 waiting on condition
>>> [0x000000004621f000]
>>> Â Â java.lang.Thread.State: WAITING (parking)
>>> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>>> Â Â Â Â - parking to wait for<0x00002aaabbb307e0> Â (a
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> Â Â Â Â at
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>> Â Â Â Â at
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>> ait(AbstractQueuedSynchronizer.java:2043)
>>> Â Â Â Â at
>>>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>>> 86)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
>>> 5)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "PRI IPC Server handler 4 on 60020" daemon prio=10
>>> tid=0x00002aae40553800 nid=0x555f waiting on condition
>>> [0x000000004611e000]
>>> Â Â java.lang.Thread.State: WAITING (parking)
>>> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>>> Â Â Â Â - parking to wait for<0x00002aaabbb307e0> Â (a
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> Â Â Â Â at
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>> Â Â Â Â at
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>> ait(AbstractQueuedSynchronizer.java:2043)
>>> Â Â Â Â at
>>>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>>> 86)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
>>> 5)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "PRI IPC Server handler 3 on 60020" daemon prio=10
>>> tid=0x00002aae4054f000 nid=0x555e waiting on condition
>>> [0x000000004601d000]
>>> Â Â java.lang.Thread.State: WAITING (parking)
>>> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>>> Â Â Â Â - parking to wait for<0x00002aaabbb307e0> Â (a
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> Â Â Â Â at
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>> Â Â Â Â at
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>> ait(AbstractQueuedSynchronizer.java:2043)
>>> Â Â Â Â at
>>>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>>> 86)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
>>> 5)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "PRI IPC Server handler 2 on 60020" daemon prio=10
>>> tid=0x00000000012f8800 nid=0x555d waiting on condition
>>> [0x0000000045f1c000]
>>> Â Â java.lang.Thread.State: WAITING (parking)
>>> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>>> Â Â Â Â - parking to wait for<0x00002aaabbb307e0> Â (a
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> Â Â Â Â at
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>> Â Â Â Â at
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>> ait(AbstractQueuedSynchronizer.java:2043)
>>> Â Â Â Â at
>>>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>>> 86)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
>>> 5)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "PRI IPC Server handler 1 on 60020" daemon prio=10
>>> tid=0x0000000001991000 nid=0x555c waiting on condition
>>> [0x0000000045e1b000]
>>> Â Â java.lang.Thread.State: WAITING (parking)
>>> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>>> Â Â Â Â - parking to wait for<0x00002aaabbb307e0> Â (a
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> Â Â Â Â at
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>> Â Â Â Â at
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>> ait(AbstractQueuedSynchronizer.java:2043)
>>> Â Â Â Â at
>>>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>>> 86)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
>>> 5)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "PRI IPC Server handler 0 on 60020" daemon prio=10
>>> tid=0x000000000198e800 nid=0x555b waiting on condition
>>> [0x0000000045d1a000]
>>> Â Â java.lang.Thread.State: WAITING (parking)
>>> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>>> Â Â Â Â - parking to wait for<0x00002aaabbb307e0> Â (a
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> Â Â Â Â at
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>> Â Â Â Â at
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>> ait(AbstractQueuedSynchronizer.java:2043)
>>> Â Â Â Â at
>>>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>>> 86)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
>>> 5)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "IPC Server handler 29 on 60020" daemon prio=10
> tid=0x000000000198c800
>>> nid=0x555a runnable [0x0000000045c18000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> Â Â Â Â at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>> Â Â Â Â at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>> Â Â Â Â at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> Â Â Â Â - locked<0x00002aac053b8450> Â (a sun.nio.ch.Util$1)
>>> Â Â Â Â - locked<0x00002aac053b8438> Â (a
>>> java.util.Collections$UnmodifiableSet)
>>> Â Â Â Â - locked<0x00002aab1037afc8> Â (a
> sun.nio.ch.EPollSelectorImpl)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>> thTimeout.java:332)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>> 157)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>> Â Â Â Â - locked<0x00002aac92976a30> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>> 50)
>>> Â Â Â Â - locked<0x00002aabc43be0a8> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>> a:237)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>> Â Â Â Â - locked<0x00002aabc43be0a8> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>> Â Â Â Â - locked<0x00002aabc43be0a8> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>> a:1899)
>>> Â Â Â Â - locked<0x00002aac1a4e0ea8> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>> )
>>> Â Â Â Â - locked<0x00002aac1a4e0ea8> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>> dRangeFileInputStream.java:105)
>>> Â Â Â Â - locked<0x00002aac237c37a8> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>> Â Â Â Â - locked<0x00002aad39485080> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>> )
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>> Â Â Â Â - locked<0x00002aab0b5ba8a8> Â (a [B)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>> 76)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>>> ner.java:87)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>> :82)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>> :262)
>>> Â Â Â Â - locked<0x00002aab89f59aa8> Â (a
>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>> :114)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>> HRegion.java:2344)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2301)
>>> Â Â Â Â - locked<0x00002aab016d6988> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2317)
>>> Â Â Â Â - locked<0x00002aab016d6988> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>> va:1832)
>>> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
> Source)
>>> Â Â Â Â at
>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>> Impl.java:43)
>>> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
>>> Â Â Â Â at
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>> 9)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "IPC Server handler 28 on 60020" daemon prio=10
> tid=0x0000000001864000
>>> nid=0x5559 runnable [0x0000000045b17000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> Â Â Â Â at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>> Â Â Â Â at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>> Â Â Â Â at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> Â Â Â Â - locked<0x00002aabaedb0608> Â (a sun.nio.ch.Util$1)
>>> Â Â Â Â - locked<0x00002aabaedb05f0> Â (a
>>> java.util.Collections$UnmodifiableSet)
>>> Â Â Â Â - locked<0x00002aaca096fa50> Â (a
> sun.nio.ch.EPollSelectorImpl)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>> thTimeout.java:332)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>> 157)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>> Â Â Â Â - locked<0x00002aac62a70b28> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>> 50)
>>> Â Â Â Â - locked<0x00002aab856e6220> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>> a:237)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>> Â Â Â Â - locked<0x00002aab856e6220> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>> Â Â Â Â - locked<0x00002aab856e6220> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>> a:1899)
>>> Â Â Â Â - locked<0x00002aab0264afe8> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>> )
>>> Â Â Â Â - locked<0x00002aab0264afe8> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>> dRangeFileInputStream.java:105)
>>> Â Â Â Â - locked<0x00002aaae070be48> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>> Â Â Â Â - locked<0x00002aac18961940> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>> )
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>> Â Â Â Â - locked<0x00002aabad8605d0> Â (a [B)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>> 76)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>>> ner.java:87)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>> :82)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>> :262)
>>> Â Â Â Â - locked<0x00002aab077e5d80> Â (a
>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>> :114)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>> HRegion.java:2344)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2301)
>>> Â Â Â Â - locked<0x00002aad30d57c18> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2317)
>>> Â Â Â Â - locked<0x00002aad30d57c18> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>> va:1832)
>>> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
> Source)
>>> Â Â Â Â at
>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>> Impl.java:43)
>>> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
>>> Â Â Â Â at
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>> 9)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "IPC Server handler 27 on 60020" daemon prio=10
> tid=0x0000000001862000
>>> nid=0x5558 runnable [0x0000000045a16000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> Â Â Â Â at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>> Â Â Â Â at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>> Â Â Â Â at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> Â Â Â Â - locked<0x00002aab2ea61b70> Â (a sun.nio.ch.Util$1)
>>> Â Â Â Â - locked<0x00002aab2ea61b88> Â (a
>>> java.util.Collections$UnmodifiableSet)
>>> Â Â Â Â - locked<0x00002aacbb364578> Â (a
> sun.nio.ch.EPollSelectorImpl)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>> thTimeout.java:332)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>> 157)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>> Â Â Â Â - locked<0x00002aab1ffc0d00> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>> 50)
>>> Â Â Â Â - locked<0x00002aad890f4640> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>> a:237)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>> Â Â Â Â - locked<0x00002aad890f4640> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>> Â Â Â Â - locked<0x00002aad890f4640> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>> a:1899)
>>> Â Â Â Â - locked<0x00002aaba1282b10> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>> )
>>> Â Â Â Â - locked<0x00002aaba1282b10> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>> dRangeFileInputStream.java:105)
>>> Â Â Â Â - locked<0x00002aac528d15d0> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>> Â Â Â Â - locked<0x00002aaaaf68a630> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>> )
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>> Â Â Â Â - locked<0x00002aabc5c948b8> Â (a [B)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>> 76)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
>>> anner.java:115)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
>>> va:255)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
>>> va:394)
>>> Â Â Â Â - locked<0x00002aab95832718> Â (a
>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>> :293)
>>> Â Â Â Â - locked<0x00002aab95832718> Â (a
>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>> :114)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>> HRegion.java:2344)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2301)
>>> Â Â Â Â - locked<0x00002aab28d7de90> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2317)
>>> Â Â Â Â - locked<0x00002aab28d7de90> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>> va:1832)
>>> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
> Source)
>>> Â Â Â Â at
>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>> Impl.java:43)
>>> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
>>> Â Â Â Â at
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>> 9)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "IPC Server handler 26 on 60020" daemon prio=10
> tid=0x0000000001860000
>>> nid=0x5557 runnable [0x0000000045915000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> Â Â Â Â at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>> Â Â Â Â at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>> Â Â Â Â at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> Â Â Â Â - locked<0x00002aabaaa6b808> Â (a sun.nio.ch.Util$1)
>>> Â Â Â Â - locked<0x00002aabaaa6b7f0> Â (a
>>> java.util.Collections$UnmodifiableSet)
>>> Â Â Â Â - locked<0x00002aaaf3899ca8> Â (a
> sun.nio.ch.EPollSelectorImpl)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>> thTimeout.java:332)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>> 157)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>> Â Â Â Â - locked<0x00002aaac1f59118> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>> 50)
>>> Â Â Â Â - locked<0x00002aaac0e99db0> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>> a:237)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>> Â Â Â Â - locked<0x00002aaac0e99db0> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>> Â Â Â Â - locked<0x00002aaac0e99db0> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>> a:1899)
>>> Â Â Â Â - locked<0x00002aabb7221e88> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>> )
>>> Â Â Â Â - locked<0x00002aabb7221e88> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>> dRangeFileInputStream.java:105)
>>> Â Â Â Â - locked<0x00002aac28621da0> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>> Â Â Â Â - locked<0x00002aac18962690> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>> )
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>> Â Â Â Â - locked<0x00002aab6876b850> Â (a [B)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.loadBlock(HFile.ja
>>> va:1447)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.reseekTo(HFile.jav
>>> a:1325)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(St
>>> oreFileScanner.java:152)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
>>> anner.java:110)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
>>> va:255)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
>>> va:394)
>>> Â Â Â Â - locked<0x00002aaabe3468f8> Â (a
>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>> :293)
>>> Â Â Â Â - locked<0x00002aaabe3468f8> Â (a
>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>> :114)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>> HRegion.java:2344)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2301)
>>> Â Â Â Â - locked<0x00002aac8e37dbc8> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2317)
>>> Â Â Â Â - locked<0x00002aac8e37dbc8> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>> va:1832)
>>> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
> Source)
>>> Â Â Â Â at
>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>> Impl.java:43)
>>> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
>>> Â Â Â Â at
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>> 9)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "IPC Server handler 25 on 60020" daemon prio=10
> tid=0x000000000173d800
>>> nid=0x5556 runnable [0x0000000045814000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> Â Â Â Â at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>> Â Â Â Â at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>> Â Â Â Â at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> Â Â Â Â - locked<0x00002aab20178510> Â (a sun.nio.ch.Util$1)
>>> Â Â Â Â - locked<0x00002aab20178528> Â (a
>>> java.util.Collections$UnmodifiableSet)
>>> Â Â Â Â - locked<0x00002aaad6187af0> Â (a
> sun.nio.ch.EPollSelectorImpl)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>> thTimeout.java:332)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>> 157)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>> Â Â Â Â - locked<0x00002aac891a6610> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>> 50)
>>> Â Â Â Â - locked<0x00002aac1de6d9f8> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>> a:237)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>> Â Â Â Â - locked<0x00002aac1de6d9f8> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>> Â Â Â Â - locked<0x00002aac1de6d9f8> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>> a:1899)
>>> Â Â Â Â - locked<0x00002aaac041c6c8> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>> )
>>> Â Â Â Â - locked<0x00002aaac041c6c8> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>> dRangeFileInputStream.java:105)
>>> Â Â Â Â - locked<0x00002aab25e717e0> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>> Â Â Â Â - locked<0x00002aac3e6e4e28> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>> )
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>> Â Â Â Â - locked<0x00002aaad4a425f0> Â (a [B)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>> 76)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>>> ner.java:87)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>> :82)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>> :262)
>>> Â Â Â Â - locked<0x00002aaabd5eb7e0> Â (a
>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>> :114)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>> HRegion.java:2344)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2301)
>>> Â Â Â Â - locked<0x00002aac8e37dc68> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2317)
>>> Â Â Â Â - locked<0x00002aac8e37dc68> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>> va:1832)
>>> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
> Source)
>>> Â Â Â Â at
>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>> Impl.java:43)
>>> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
>>> Â Â Â Â at
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>> 9)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "IPC Server handler 24 on 60020" daemon prio=10
> tid=0x000000000173a000
>>> nid=0x5555 waiting for monitor entry [0x0000000045714000]
>>> Â Â java.lang.Thread.State: BLOCKED (on object monitor)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2317)
>>> Â Â Â Â - waiting to lock<0x00002aab016d6618> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>> va:1832)
>>> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
> Source)
>>> Â Â Â Â at
>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>> Impl.java:43)
>>> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
>>> Â Â Â Â at
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>> 9)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "IPC Server handler 23 on 60020" daemon prio=10
> tid=0x0000000001444000
>>> nid=0x5554 runnable [0x0000000045612000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> Â Â Â Â at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>> Â Â Â Â at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>> Â Â Â Â at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> Â Â Â Â - locked<0x00002aad70044d30> Â (a sun.nio.ch.Util$1)
>>> Â Â Â Â - locked<0x00002aad70044d48> Â (a
>>> java.util.Collections$UnmodifiableSet)
>>> Â Â Â Â - locked<0x00002aab328f1a40> Â (a
> sun.nio.ch.EPollSelectorImpl)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>> thTimeout.java:332)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>> 157)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>> Â Â Â Â - locked<0x00002aaac9d48e68> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>> 50)
>>> Â Â Â Â - locked<0x00002aac1de6dd40> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>> a:237)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>> Â Â Â Â - locked<0x00002aac1de6dd40> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>> Â Â Â Â - locked<0x00002aac1de6dd40> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>> a:1899)
>>> Â Â Â Â - locked<0x00002aab39d88148> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>> )
>>> Â Â Â Â - locked<0x00002aab39d88148> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>> dRangeFileInputStream.java:105)
>>> Â Â Â Â - locked<0x00002aab7addd0d8> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>> Â Â Â Â - locked<0x00002aab2811e8e8> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>> )
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>> Â Â Â Â - locked<0x00002aabe56e8858> Â (a [B)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>> 76)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>>> ner.java:87)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>> :82)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>> :262)
>>> Â Â Â Â - locked<0x00002aad15b7d360> Â (a
>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>> :114)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>> HRegion.java:2344)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2301)
>>> Â Â Â Â - locked<0x00002aabc7a3d258> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2317)
>>> Â Â Â Â - locked<0x00002aabc7a3d258> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>> va:1832)
>>> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
> Source)
>>> Â Â Â Â at
>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>> Impl.java:43)
>>> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
>>> Â Â Â Â at
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>> 9)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "IPC Server handler 22 on 60020" daemon prio=10
> tid=0x0000000001442000
>>> nid=0x5553 runnable [0x0000000045511000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> Â Â Â Â at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>> Â Â Â Â at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>> Â Â Â Â at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> Â Â Â Â - locked<0x00002aabf6e38450> Â (a sun.nio.ch.Util$1)
>>> Â Â Â Â - locked<0x00002aabf6e38468> Â (a
>>> java.util.Collections$UnmodifiableSet)
>>> Â Â Â Â - locked<0x00002aabb9e99468> Â (a
> sun.nio.ch.EPollSelectorImpl)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>> thTimeout.java:332)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>> 157)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>> Â Â Â Â - locked<0x00002aaae13c0340> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>> 50)
>>> Â Â Â Â - locked<0x00002aad427a91c0> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>> a:237)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>> Â Â Â Â - locked<0x00002aad427a91c0> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>> Â Â Â Â - locked<0x00002aad427a91c0> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>> a:1899)
>>> Â Â Â Â - locked<0x00002aab7c7ca0c8> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>> )
>>> Â Â Â Â - locked<0x00002aab7c7ca0c8> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>> dRangeFileInputStream.java:105)
>>> Â Â Â Â - locked<0x00002aab3748f1a8> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>> Â Â Â Â - locked<0x00002aab52d56ed8> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>> )
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>> Â Â Â Â - locked<0x00002aab60a86ca8> Â (a [B)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>> 76)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(St
>>> oreFileScanner.java:158)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
>>> anner.java:110)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
>>> va:255)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
>>> va:394)
>>> Â Â Â Â - locked<0x00002aab706a3e98> Â (a
>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>> :293)
>>> Â Â Â Â - locked<0x00002aab706a3e98> Â (a
>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>> :114)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>> HRegion.java:2344)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2301)
>>> Â Â Â Â - locked<0x00002aaae3489758> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2317)
>>> Â Â Â Â - locked<0x00002aaae3489758> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>> va:1832)
>>> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
> Source)
>>> Â Â Â Â at
>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>> Impl.java:43)
>>> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
>>> Â Â Â Â at
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>> 9)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "IPC Server handler 21 on 60020" daemon prio=10
> tid=0x0000000001792000
>>> nid=0x5552 waiting for monitor entry [0x0000000045411000]
>>> Â Â java.lang.Thread.State: BLOCKED (on object monitor)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2317)
>>> Â Â Â Â - waiting to lock<0x00002aabc7a3d3e8> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>> va:1832)
>>> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
> Source)
>>> Â Â Â Â at
>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>> Impl.java:43)
>>> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
>>> Â Â Â Â at
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>> 9)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "IPC Server handler 20 on 60020" daemon prio=10
> tid=0x000000000178e000
>>> nid=0x5551 runnable [0x000000004530f000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> Â Â Â Â at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>> Â Â Â Â at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>> Â Â Â Â at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> Â Â Â Â - locked<0x00002aab3745e018> Â (a sun.nio.ch.Util$1)
>>> Â Â Â Â - locked<0x00002aab3745e030> Â (a
>>> java.util.Collections$UnmodifiableSet)
>>> Â Â Â Â - locked<0x00002aab7d5cc0c0> Â (a
> sun.nio.ch.EPollSelectorImpl)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>> thTimeout.java:332)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>> 157)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>> Â Â Â Â - locked<0x00002aac035e4990> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>> 50)
>>> Â Â Â Â - locked<0x00002aad890f42f8> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>> a:237)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>> Â Â Â Â - locked<0x00002aad890f42f8> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>> Â Â Â Â - locked<0x00002aad890f42f8> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>> a:1899)
>>> Â Â Â Â - locked<0x00002aab1baffa50> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>> )
>>> Â Â Â Â - locked<0x00002aab1baffa50> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>> dRangeFileInputStream.java:105)
>>> Â Â Â Â - locked<0x00002aaadef99fe8> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>> Â Â Â Â - locked<0x00002aac189625a0> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>> )
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>> Â Â Â Â - locked<0x00002aabb88c7ce0> Â (a [B)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>> 76)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
>>> anner.java:115)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
>>> va:255)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
>>> va:394)
>>> Â Â Â Â - locked<0x00002aaadfa95ec0> Â (a
>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>> :293)
>>> Â Â Â Â - locked<0x00002aaadfa95ec0> Â (a
>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>> :114)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>> HRegion.java:2344)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2301)
>>> Â Â Â Â - locked<0x00002aabf6832658> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2317)
>>> Â Â Â Â - locked<0x00002aabf6832658> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>> va:1832)
>>> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
> Source)
>>> Â Â Â Â at
>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>> Impl.java:43)
>>> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
>>> Â Â Â Â at
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>> 9)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "IPC Server handler 19 on 60020" daemon prio=10
> tid=0x0000000001b19800
>>> nid=0x5550 runnable [0x000000004520e000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> Â Â Â Â at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>> Â Â Â Â at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>> Â Â Â Â at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> Â Â Â Â - locked<0x00002aab94af3848> Â (a sun.nio.ch.Util$1)
>>> Â Â Â Â - locked<0x00002aab94af3830> Â (a
>>> java.util.Collections$UnmodifiableSet)
>>> Â Â Â Â - locked<0x00002aac6ed3e1d8> Â (a
> sun.nio.ch.EPollSelectorImpl)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>> thTimeout.java:332)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>> 157)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>> Â Â Â Â - locked<0x00002aaac1194578> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>> 50)
>>> Â Â Â Â - locked<0x00002aad890f2600> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>> a:237)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>> Â Â Â Â - locked<0x00002aad890f2600> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>> Â Â Â Â - locked<0x00002aad890f2600> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>> a:1899)
>>> Â Â Â Â - locked<0x00002aaae78be750> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>> )
>>> Â Â Â Â - locked<0x00002aaae78be750> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>> dRangeFileInputStream.java:105)
>>> Â Â Â Â - locked<0x00002aabba906cc0> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>> Â Â Â Â - locked<0x00002aac18963ef0> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>> )
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>> Â Â Â Â - locked<0x00002aaacd2974c0> Â (a [B)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>> 76)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
>>> anner.java:115)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
>>> va:255)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
>>> va:394)
>>> Â Â Â Â - locked<0x00002aab328ec218> Â (a
>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>> :293)
>>> Â Â Â Â - locked<0x00002aab328ec218> Â (a
>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>> :114)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>> HRegion.java:2344)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2301)
>>> Â Â Â Â - locked<0x00002aabbaf1c898> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2317)
>>> Â Â Â Â - locked<0x00002aabbaf1c898> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>> va:1832)
>>> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
> Source)
>>> Â Â Â Â at
>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>> Impl.java:43)
>>> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
>>> Â Â Â Â at
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>> 9)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "IPC Server handler 18 on 60020" daemon prio=10
> tid=0x0000000001b17800
>>> nid=0x554f runnable [0x000000004510d000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> Â Â Â Â at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>> Â Â Â Â at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>> Â Â Â Â at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> Â Â Â Â - locked<0x00002aaae98101b0> Â (a sun.nio.ch.Util$1)
>>> Â Â Â Â - locked<0x00002aaae9810198> Â (a
>>> java.util.Collections$UnmodifiableSet)
>>> Â Â Â Â - locked<0x00002aaaeba1b040> Â (a
> sun.nio.ch.EPollSelectorImpl)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>> thTimeout.java:332)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>> 157)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>> Â Â Â Â - locked<0x00002aab1ac7d288> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>> 50)
>>> Â Â Â Â - locked<0x00002aad890f0db8> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>> a:237)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>> Â Â Â Â - locked<0x00002aad890f0db8> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>> Â Â Â Â - locked<0x00002aad890f0db8> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>> a:1899)
>>> Â Â Â Â - locked<0x00002aab8c671d30> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>> )
>>> Â Â Â Â - locked<0x00002aab8c671d30> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>> dRangeFileInputStream.java:105)
>>> Â Â Â Â - locked<0x00002aab8c4f3de0> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>> Â Â Â Â - locked<0x00002aac18963dd0> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>> )
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>> Â Â Â Â - locked<0x00002aab8d9fd7c8> Â (a [B)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>> 76)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
>>> anner.java:115)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
>>> va:255)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
>>> va:394)
>>> Â Â Â Â - locked<0x00002aaba69901b8> Â (a
>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>> :293)
>>> Â Â Â Â - locked<0x00002aaba69901b8> Â (a
>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>> :114)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>> HRegion.java:2344)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2301)
>>> Â Â Â Â - locked<0x00002aabbaf1e468> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2317)
>>> Â Â Â Â - locked<0x00002aabbaf1e468> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>> va:1832)
>>> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
> Source)
>>> Â Â Â Â at
>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>> Impl.java:43)
>>> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
>>> Â Â Â Â at
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>> 9)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "IPC Server handler 17 on 60020" daemon prio=10
> tid=0x0000000001b15800
>>> nid=0x554e runnable [0x000000004500c000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> Â Â Â Â at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>> Â Â Â Â at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>> Â Â Â Â at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> Â Â Â Â - locked<0x00002aaac2d0d348> Â (a sun.nio.ch.Util$1)
>>> Â Â Â Â - locked<0x00002aaac2d0d330> Â (a
>>> java.util.Collections$UnmodifiableSet)
>>> Â Â Â Â - locked<0x00002aacbb364b18> Â (a
> sun.nio.ch.EPollSelectorImpl)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>> thTimeout.java:332)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>> 157)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>> Â Â Â Â - locked<0x00002aaac2750e38> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>> 50)
>>> Â Â Â Â - locked<0x00002aaac1028128> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>> a:237)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>> Â Â Â Â - locked<0x00002aaac1028128> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>> Â Â Â Â - locked<0x00002aaac1028128> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>> a:1899)
>>> Â Â Â Â - locked<0x00002aaae41f13d8> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>> )
>>> Â Â Â Â - locked<0x00002aaae41f13d8> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>> dRangeFileInputStream.java:105)
>>> Â Â Â Â - locked<0x00002aab3c08c348> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>> Â Â Â Â - locked<0x00002aab52d54ef8> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>> )
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>> Â Â Â Â - locked<0x00002aac0cb261b0> Â (a [B)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>> 76)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>>> ner.java:87)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>> :82)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>> :262)
>>> Â Â Â Â - locked<0x00002aaabefb9c10> Â (a
>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>> :114)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>> HRegion.java:2344)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2301)
>>> Â Â Â Â - locked<0x00002aad30d56688> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2317)
>>> Â Â Â Â - locked<0x00002aad30d56688> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>> va:1832)
>>> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
> Source)
>>> Â Â Â Â at
>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>> Impl.java:43)
>>> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
>>> Â Â Â Â at
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>> 9)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "IPC Server handler 16 on 60020" daemon prio=10
> tid=0x00000000013fa000
>>> nid=0x554d runnable [0x0000000044f0b000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> Â Â Â Â at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>> Â Â Â Â at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>> Â Â Â Â at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> Â Â Â Â - locked<0x00002aabec61aa08> Â (a sun.nio.ch.Util$1)
>>> Â Â Â Â - locked<0x00002aabec61a9f0> Â (a
>>> java.util.Collections$UnmodifiableSet)
>>> Â Â Â Â - locked<0x00002aaae7590720> Â (a
> sun.nio.ch.EPollSelectorImpl)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>> thTimeout.java:332)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>> 157)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>> Â Â Â Â - locked<0x00002aacf2246cb0> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>> 50)
>>> Â Â Â Â - locked<0x00002aad06dd2c70> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>> a:237)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>> Â Â Â Â - locked<0x00002aad06dd2c70> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>> Â Â Â Â - locked<0x00002aad06dd2c70> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>> a:1899)
>>> Â Â Â Â - locked<0x00002aabe38abcd8> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>> )
>>> Â Â Â Â - locked<0x00002aabe38abcd8> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>> dRangeFileInputStream.java:105)
>>> Â Â Â Â - locked<0x00002aaad08ca208> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>> Â Â Â Â - locked<0x00002aac189636e0> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>> )
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>> Â Â Â Â - locked<0x00002aab84fe43c8> Â (a [B)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>> 76)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(St
>>> oreFileScanner.java:158)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
>>> anner.java:110)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
>>> va:255)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
>>> va:394)
>>> Â Â Â Â - locked<0x00002aad072dbd30> Â (a
>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>> :293)
>>> Â Â Â Â - locked<0x00002aad072dbd30> Â (a
>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>> :114)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>> HRegion.java:2344)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2301)
>>> Â Â Â Â - locked<0x00002aabdbaa0490> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2317)
>>> Â Â Â Â - locked<0x00002aabdbaa0490> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>> va:1832)
>>> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
> Source)
>>> Â Â Â Â at
>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>> Impl.java:43)
>>> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
>>> Â Â Â Â at
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>> 9)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "IPC Server handler 15 on 60020" daemon prio=10
> tid=0x00000000013f7800
>>> nid=0x554c runnable [0x0000000044e0a000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> Â Â Â Â at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>> Â Â Â Â at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>> Â Â Â Â at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> Â Â Â Â - locked<0x00002aac558cde50> Â (a sun.nio.ch.Util$1)
>>> Â Â Â Â - locked<0x00002aac558cde38> Â (a
>>> java.util.Collections$UnmodifiableSet)
>>> Â Â Â Â - locked<0x00002aac5cd41558> Â (a
> sun.nio.ch.EPollSelectorImpl)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>> thTimeout.java:332)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>> 157)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>> Â Â Â Â - locked<0x00002aadcf530468> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>> 50)
>>> Â Â Â Â - locked<0x00002aad427a97a8> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>> a:237)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>> Â Â Â Â - locked<0x00002aad427a97a8> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>> Â Â Â Â - locked<0x00002aad427a97a8> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>> a:1899)
>>> Â Â Â Â - locked<0x00002aab9410bc00> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>> )
>>> Â Â Â Â - locked<0x00002aab9410bc00> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>> dRangeFileInputStream.java:105)
>>> Â Â Â Â - locked<0x00002aaca7cfcf70> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>> Â Â Â Â - locked<0x00002aacf952a6c8> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>> )
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>> Â Â Â Â - locked<0x00002aaac4472350> Â (a [B)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>> 76)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>>> ner.java:87)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>> :82)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>> :297)
>>> Â Â Â Â - locked<0x00002aaaefe12070> Â (a
>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>> :114)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>> HRegion.java:2344)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2301)
>>> Â Â Â Â - locked<0x00002aabbaf1d1a8> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2317)
>>> Â Â Â Â - locked<0x00002aabbaf1d1a8> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>> va:1832)
>>> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
> Source)
>>> Â Â Â Â at
>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>> Impl.java:43)
>>> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
>>> Â Â Â Â at
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>> 9)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "IPC Server handler 14 on 60020" daemon prio=10
> tid=0x00000000013f5800
>>> nid=0x554b runnable [0x0000000044d09000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> Â Â Â Â at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>> Â Â Â Â at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>> Â Â Â Â at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> Â Â Â Â - locked<0x00002aab28f91848> Â (a sun.nio.ch.Util$1)
>>> Â Â Â Â - locked<0x00002aab28f91860> Â (a
>>> java.util.Collections$UnmodifiableSet)
>>> Â Â Â Â - locked<0x00002aaaf5031970> Â (a
> sun.nio.ch.EPollSelectorImpl)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>> thTimeout.java:332)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>> 157)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>> Â Â Â Â - locked<0x00002aacec8cf590> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>> 50)
>>> Â Â Â Â - locked<0x00002aaaf789b048> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>> a:237)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>> Â Â Â Â - locked<0x00002aaaf789b048> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>> Â Â Â Â - locked<0x00002aaaf789b048> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>> a:1899)
>>> Â Â Â Â - locked<0x00002aab9c1df470> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>> )
>>> Â Â Â Â - locked<0x00002aab9c1df470> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>> dRangeFileInputStream.java:105)
>>> Â Â Â Â - locked<0x00002aab7ec60418> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>> Â Â Â Â - locked<0x00002aab44c84dd8> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>> )
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>> Â Â Â Â - locked<0x00002aab227a2dc8> Â (a [B)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>> 76)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(St
>>> oreFileScanner.java:158)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
>>> anner.java:110)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
>>> va:255)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
>>> va:394)
>>> Â Â Â Â - locked<0x00002aab50d25f28> Â (a
>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>> :293)
>>> Â Â Â Â - locked<0x00002aab50d25f28> Â (a
>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>> :114)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>> HRegion.java:2344)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2301)
>>> Â Â Â Â - locked<0x00002aaae9efd9f0> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2317)
>>> Â Â Â Â - locked<0x00002aaae9efd9f0> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>> va:1832)
>>> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
> Source)
>>> Â Â Â Â at
>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>> Impl.java:43)
>>> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
>>> Â Â Â Â at
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>> 9)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "IPC Server handler 13 on 60020" daemon prio=10
> tid=0x0000000001b61800
>>> nid=0x554a waiting for monitor entry [0x0000000044c09000]
>>> Â Â java.lang.Thread.State: BLOCKED (on object monitor)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2317)
>>> Â Â Â Â - waiting to lock<0x00002aacc684dad0> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>> va:1832)
>>> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
> Source)
>>> Â Â Â Â at
>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>> Impl.java:43)
>>> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
>>> Â Â Â Â at
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>> 9)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "IPC Server handler 12 on 60020" daemon prio=10
> tid=0x0000000001b5f800
>>> nid=0x5549 runnable [0x0000000044b07000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> Â Â Â Â at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>> Â Â Â Â at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>> Â Â Â Â at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> Â Â Â Â - locked<0x00002aab618bb060> Â (a sun.nio.ch.Util$1)
>>> Â Â Â Â - locked<0x00002aabe1d82718> Â (a
>>> java.util.Collections$UnmodifiableSet)
>>> Â Â Â Â - locked<0x00002aab76896238> Â (a
> sun.nio.ch.EPollSelectorImpl)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>> thTimeout.java:332)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>> 157)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>> Â Â Â Â - locked<0x00002aab016dc128> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>> 50)
>>> Â Â Â Â - locked<0x00002aad03182340> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>> a:237)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>> Â Â Â Â - locked<0x00002aad03182340> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>> Â Â Â Â - locked<0x00002aad03182340> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>> a:1899)
>>> Â Â Â Â - locked<0x00002aab8b32e3e0> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>> )
>>> Â Â Â Â - locked<0x00002aab8b32e3e0> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>> dRangeFileInputStream.java:105)
>>> Â Â Â Â - locked<0x00002aab8b436170> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>> Â Â Â Â - locked<0x00002aacd4fb8038> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>> )
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>> Â Â Â Â - locked<0x00002aab8b4b4908> Â (a [B)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>> 76)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>>> ner.java:87)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>> :82)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>> :262)
>>> Â Â Â Â - locked<0x00002aab480f55b8> Â (a
>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>> :114)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>> HRegion.java:2344)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2301)
>>> Â Â Â Â - locked<0x00002aaae9efdb30> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2317)
>>> Â Â Â Â - locked<0x00002aaae9efdb30> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>> va:1832)
>>> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
> Source)
>>> Â Â Â Â at
>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>> Impl.java:43)
>>> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
>>> Â Â Â Â at
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>> 9)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "IPC Server handler 11 on 60020" daemon prio=10
> tid=0x0000000001b5d000
>>> nid=0x5548 waiting for monitor entry [0x0000000044a07000]
>>> Â Â java.lang.Thread.State: BLOCKED (on object monitor)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2317)
>>> Â Â Â Â - waiting to lock<0x00002aacc684dad0> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>> va:1832)
>>> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
> Source)
>>> Â Â Â Â at
>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>> Impl.java:43)
>>> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
>>> Â Â Â Â at
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>> 9)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "IPC Server handler 10 on 60020" daemon prio=10
> tid=0x000000000196a800
>>> nid=0x5547 runnable [0x0000000044905000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> Â Â Â Â at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>> Â Â Â Â at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>> Â Â Â Â at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> Â Â Â Â - locked<0x00002aad7443dcd8> Â (a sun.nio.ch.Util$1)
>>> Â Â Â Â - locked<0x00002aad7a31cb00> Â (a
>>> java.util.Collections$UnmodifiableSet)
>>> Â Â Â Â - locked<0x00002aac9067d2d8> Â (a
> sun.nio.ch.EPollSelectorImpl)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>> thTimeout.java:332)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>> 157)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>> Â Â Â Â - locked<0x00002aad5103b0a0> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>> 50)
>>> Â Â Â Â - locked<0x00002aadc79059c8> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>> a:237)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>> Â Â Â Â - locked<0x00002aadc79059c8> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>> Â Â Â Â - locked<0x00002aadc79059c8> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>> a:1899)
>>> Â Â Â Â - locked<0x00002aabf0f472b0> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>> )
>>> Â Â Â Â - locked<0x00002aabf0f472b0> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>> dRangeFileInputStream.java:105)
>>> Â Â Â Â - locked<0x00002aab8b2b29b0> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>> Â Â Â Â - locked<0x00002aad0477a310> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>> )
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>> Â Â Â Â - locked<0x00002aabb7b16fd0> Â (a [B)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>> 76)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>>> ner.java:87)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>> :82)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>> :262)
>>> Â Â Â Â - locked<0x00002aab59165e10> Â (a
>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>> :114)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>> HRegion.java:2344)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2301)
>>> Â Â Â Â - locked<0x00002aac8e37dd58> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2317)
>>> Â Â Â Â - locked<0x00002aac8e37dd58> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>> va:1832)
>>> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
> Source)
>>> Â Â Â Â at
>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>> Impl.java:43)
>>> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
>>> Â Â Â Â at
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>> 9)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "IPC Server handler 9 on 60020" daemon prio=10 tid=0x0000000001968000
>>> nid=0x5546 waiting for monitor entry [0x0000000044805000]
>>> Â Â java.lang.Thread.State: BLOCKED (on object monitor)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.isFilterDone(
>>> HRegion.java:2324)
>>> Â Â Â Â - waiting to lock<0x00002aacc684dad0> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>> va:1850)
>>> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
> Source)
>>> Â Â Â Â at
>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>> Impl.java:43)
>>> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
>>> Â Â Â Â at
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>> 9)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "IPC Server handler 8 on 60020" daemon prio=10 tid=0x0000000001967000
>>> nid=0x5545 runnable [0x0000000044703000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> Â Â Â Â at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>> Â Â Â Â at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>> Â Â Â Â at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> Â Â Â Â - locked<0x00002aabb0b7b868> Â (a sun.nio.ch.Util$1)
>>> Â Â Â Â - locked<0x00002aabb0b7b880> Â (a
>>> java.util.Collections$UnmodifiableSet)
>>> Â Â Â Â - locked<0x00002aaac79442c8> Â (a
> sun.nio.ch.EPollSelectorImpl)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>> thTimeout.java:332)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>> 157)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>> Â Â Â Â - locked<0x00002aad0106a6b0> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>> 50)
>>> Â Â Â Â - locked<0x00002aac108e6768> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>> a:237)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>> Â Â Â Â - locked<0x00002aac108e6768> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>> Â Â Â Â - locked<0x00002aac108e6768> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>> a:1899)
>>> Â Â Â Â - locked<0x00002aac4d4ecd40> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>> )
>>> Â Â Â Â - locked<0x00002aac4d4ecd40> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>> dRangeFileInputStream.java:105)
>>> Â Â Â Â - locked<0x00002aaad20c3b08> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>> Â Â Â Â - locked<0x00002aac536bda88> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>> )
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>> Â Â Â Â - locked<0x00002aaadfc86320> Â (a [B)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.loadBlock(HFile.ja
>>> va:1447)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.reseekTo(HFile.jav
>>> a:1325)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(St
>>> oreFileScanner.java:152)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
>>> anner.java:110)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
>>> va:255)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
>>> va:394)
>>> Â Â Â Â - locked<0x00002aad072dc1c8> Â (a
>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>> :293)
>>> Â Â Â Â - locked<0x00002aad072dc1c8> Â (a
>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>> :114)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>> HRegion.java:2344)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2301)
>>> Â Â Â Â - locked<0x00002aac8e37d808> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2317)
>>> Â Â Â Â - locked<0x00002aac8e37d808> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>> va:1832)
>>> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
> Source)
>>> Â Â Â Â at
>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>> Impl.java:43)
>>> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
>>> Â Â Â Â at
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>> 9)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "IPC Server handler 7 on 60020" daemon prio=10 tid=0x00000000016de800
>>> nid=0x5544 runnable [0x0000000044602000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> Â Â Â Â at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>> Â Â Â Â at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>> Â Â Â Â at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> Â Â Â Â - locked<0x00002aab9f4dc588> Â (a sun.nio.ch.Util$1)
>>> Â Â Â Â - locked<0x00002aab9f4dc570> Â (a
>>> java.util.Collections$UnmodifiableSet)
>>> Â Â Â Â - locked<0x00002aac6ed3e250> Â (a
> sun.nio.ch.EPollSelectorImpl)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>> thTimeout.java:332)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>> 157)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>> Â Â Â Â - locked<0x00002aad003a0178> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>> 50)
>>> Â Â Â Â - locked<0x00002aabbd698ec8> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>> a:237)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>> Â Â Â Â - locked<0x00002aabbd698ec8> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>> Â Â Â Â - locked<0x00002aabbd698ec8> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>> a:1899)
>>> Â Â Â Â - locked<0x00002aaad91ecfa8> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>> )
>>> Â Â Â Â - locked<0x00002aaad91ecfa8> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>> dRangeFileInputStream.java:105)
>>> Â Â Â Â - locked<0x00002aac237c5090> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>> Â Â Â Â - locked<0x00002aad65d9a7f0> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>> )
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>> Â Â Â Â - locked<0x00002aaaccc5d6c8> Â (a [B)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>> 76)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>>> ner.java:87)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>> :82)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>> :262)
>>> Â Â Â Â - locked<0x00002aad15b7d2b8> Â (a
>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>> :114)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>> HRegion.java:2344)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2301)
>>> Â Â Â Â - locked<0x00002aacc684dad0> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2317)
>>> Â Â Â Â - locked<0x00002aacc684dad0> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>> va:1832)
>>> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
> Source)
>>> Â Â Â Â at
>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>> Impl.java:43)
>>> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
>>> Â Â Â Â at
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>> 9)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "IPC Server handler 6 on 60020" daemon prio=10 tid=0x00000000016dd000
>>> nid=0x5543 runnable [0x0000000044501000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> Â Â Â Â at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>> Â Â Â Â at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>> Â Â Â Â at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> Â Â Â Â - locked<0x00002aaaf7db82f0> Â (a sun.nio.ch.Util$1)
>>> Â Â Â Â - locked<0x00002aaaf7db82d8> Â (a
>>> java.util.Collections$UnmodifiableSet)
>>> Â Â Â Â - locked<0x00002aab6f3339a8> Â (a
> sun.nio.ch.EPollSelectorImpl)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>> thTimeout.java:332)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>> 157)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>> Â Â Â Â - locked<0x00002aab494cdcb8> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>> 50)
>>> Â Â Â Â - locked<0x00002aad890f5018> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>> a:237)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>> Â Â Â Â - locked<0x00002aad890f5018> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>> Â Â Â Â - locked<0x00002aad890f5018> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>> a:1899)
>>> Â Â Â Â - locked<0x00002aaae4d56928> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>> )
>>> Â Â Â Â - locked<0x00002aaae4d56928> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>> dRangeFileInputStream.java:105)
>>> Â Â Â Â - locked<0x00002aaae8d98798> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>> Â Â Â Â - locked<0x00002aaaaf686f40> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>> )
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>> Â Â Â Â - locked<0x00002aaae77f0ad8> Â (a [B)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>> 76)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>>> ner.java:87)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>> :82)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>> :262)
>>> Â Â Â Â - locked<0x00002aaae4cd85a0> Â (a
>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>> :114)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>> HRegion.java:2344)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2301)
>>> Â Â Â Â - locked<0x00002aad30d55d28> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2317)
>>> Â Â Â Â - locked<0x00002aad30d55d28> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>> va:1832)
>>> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
> Source)
>>> Â Â Â Â at
>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>> Impl.java:43)
>>> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
>>> Â Â Â Â at
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>> 9)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "IPC Server handler 5 on 60020" daemon prio=10 tid=0x000000000194c800
>>> nid=0x5542 runnable [0x0000000044400000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> Â Â Â Â at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>> Â Â Â Â at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>> Â Â Â Â at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> Â Â Â Â - locked<0x00002aaac5bf0ff0> Â (a sun.nio.ch.Util$1)
>>> Â Â Â Â - locked<0x00002aaac5bf10c8> Â (a
>>> java.util.Collections$UnmodifiableSet)
>>> Â Â Â Â - locked<0x00002aab0492d6d8> Â (a
> sun.nio.ch.EPollSelectorImpl)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>> thTimeout.java:332)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>> 157)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>> Â Â Â Â - locked<0x00002aac451e5d20> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
>>> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>> 89)
>>> Â Â Â Â - locked<0x00002aabf9312408> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>> a:237)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>> Â Â Â Â - locked<0x00002aabf9312408> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>> Â Â Â Â - locked<0x00002aabf9312408> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>> a:1899)
>>> Â Â Â Â - locked<0x00002aab78dc6068> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>> )
>>> Â Â Â Â - locked<0x00002aab78dc6068> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>> dRangeFileInputStream.java:105)
>>> Â Â Â Â - locked<0x00002aab14390138> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>> Â Â Â Â - locked<0x00002aab7fbb83e8> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>> )
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>> Â Â Â Â - locked<0x00002aac2cbf46c8> Â (a [B)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>> 76)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
>>> anner.java:115)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
>>> va:255)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
>>> va:394)
>>> Â Â Â Â - locked<0x00002aad15b7da98> Â (a
>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>> :293)
>>> Â Â Â Â - locked<0x00002aad15b7da98> Â (a
>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>> :114)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>> HRegion.java:2344)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2301)
>>> Â Â Â Â - locked<0x00002aabc7a3d3e8> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2317)
>>> Â Â Â Â - locked<0x00002aabc7a3d3e8> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>> va:1832)
>>> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
> Source)
>>> Â Â Â Â at
>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>> Impl.java:43)
>>> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
>>> Â Â Â Â at
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>> 9)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "IPC Server handler 4 on 60020" daemon prio=10 tid=0x000000000194a800
>>> nid=0x5541 runnable [0x00000000442ff000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> Â Â Â Â at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>> Â Â Â Â at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>> Â Â Â Â at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> Â Â Â Â - locked<0x00002aabbae85af8> Â (a sun.nio.ch.Util$1)
>>> Â Â Â Â - locked<0x00002aabbae85b10> Â (a
>>> java.util.Collections$UnmodifiableSet)
>>> Â Â Â Â - locked<0x00002aaae379da00> Â (a
> sun.nio.ch.EPollSelectorImpl)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>> thTimeout.java:332)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>> 157)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>> Â Â Â Â - locked<0x00002aac7ecf9bb8> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>> 50)
>>> Â Â Â Â - locked<0x00002aaacf49d528> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>> a:237)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>> Â Â Â Â - locked<0x00002aaacf49d528> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>> Â Â Â Â - locked<0x00002aaacf49d528> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>> a:1899)
>>> Â Â Â Â - locked<0x00002aabdc652620> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>> )
>>> Â Â Â Â - locked<0x00002aabdc652620> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>> dRangeFileInputStream.java:105)
>>> Â Â Â Â - locked<0x00002aac357f4350> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>> Â Â Â Â - locked<0x00002aab9fc22a58> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>> )
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>> Â Â Â Â - locked<0x00002aabdbefbe20> Â (a [B)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>> 76)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>>> ner.java:87)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>> :82)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>> :262)
>>> Â Â Â Â - locked<0x00002aad97b3b7e8> Â (a
>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>> :114)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>> HRegion.java:2344)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2301)
>>> Â Â Â Â - locked<0x00002aab016d6618> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2317)
>>> Â Â Â Â - locked<0x00002aab016d6618> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>> va:1832)
>>> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
> Source)
>>> Â Â Â Â at
>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>> Impl.java:43)
>>> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
>>> Â Â Â Â at
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>> 9)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "IPC Server handler 3 on 60020" daemon prio=10 tid=0x00000000019c4000
>>> nid=0x5540 waiting for monitor entry [0x00000000441ff000]
>>> Â Â java.lang.Thread.State: BLOCKED (on object monitor)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2317)
>>> Â Â Â Â - waiting to lock<0x00002aab28d7de90> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>> va:1832)
>>> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
> Source)
>>> Â Â Â Â at
>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>> Impl.java:43)
>>> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
>>> Â Â Â Â at
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>> 9)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "IPC Server handler 2 on 60020" daemon prio=10 tid=0x00000000019c2000
>>> nid=0x553f runnable [0x00000000440fd000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> Â Â Â Â at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>> Â Â Â Â at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>> Â Â Â Â at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> Â Â Â Â - locked<0x00002aaaeef929f8> Â (a sun.nio.ch.Util$1)
>>> Â Â Â Â - locked<0x00002aaaeef92a10> Â (a
>>> java.util.Collections$UnmodifiableSet)
>>> Â Â Â Â - locked<0x00002aaaffd9ef38> Â (a
> sun.nio.ch.EPollSelectorImpl)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>> thTimeout.java:332)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>> 157)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>> Â Â Â Â - locked<0x00002aab2ac42300> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>> 50)
>>> Â Â Â Â - locked<0x00002aabc43bef18> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>> a:237)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>> Â Â Â Â - locked<0x00002aabc43bef18> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>> Â Â Â Â - locked<0x00002aabc43bef18> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>> a:1899)
>>> Â Â Â Â - locked<0x00002aabd53c1d58> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>> )
>>> Â Â Â Â - locked<0x00002aabd53c1d58> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>> dRangeFileInputStream.java:105)
>>> Â Â Â Â - locked<0x00002aabb602a180> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>> Â Â Â Â - locked<0x00002aab44c85f18> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>> )
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>> Â Â Â Â - locked<0x00002aac0d2be0b0> Â (a [B)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>> 76)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>>> ner.java:87)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>> :82)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>> :262)
>>> Â Â Â Â - locked<0x00002aab4eb7a680> Â (a
>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>> :114)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>> HRegion.java:2344)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2301)
>>> Â Â Â Â - locked<0x00002aabc7a3cc68> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2317)
>>> Â Â Â Â - locked<0x00002aabc7a3cc68> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>> va:1832)
>>> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
> Source)
>>> Â Â Â Â at
>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>> Impl.java:43)
>>> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
>>> Â Â Â Â at
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>> 9)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "IPC Server handler 1 on 60020" daemon prio=10 tid=0x0000000001535800
>>> nid=0x553e runnable [0x0000000043ffc000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> Â Â Â Â at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>> Â Â Â Â at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>> Â Â Â Â at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> Â Â Â Â - locked<0x00002aad28449198> Â (a sun.nio.ch.Util$1)
>>> Â Â Â Â - locked<0x00002aad284491b0> Â (a
>>> java.util.Collections$UnmodifiableSet)
>>> Â Â Â Â - locked<0x00002aab3a682700> Â (a
> sun.nio.ch.EPollSelectorImpl)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>> thTimeout.java:332)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>> 157)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>> Â Â Â Â - locked<0x00002aac5b5cbd28> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>> 50)
>>> Â Â Â Â - locked<0x00002aabc43bd778> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>> a:237)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>> Â Â Â Â - locked<0x00002aabc43bd778> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>> Â Â Â Â - locked<0x00002aabc43bd778> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>> a:1899)
>>> Â Â Â Â - locked<0x00002aac4d4f9b90> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>> )
>>> Â Â Â Â - locked<0x00002aac4d4f9b90> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>> dRangeFileInputStream.java:105)
>>> Â Â Â Â - locked<0x00002aac500ac780> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>> Â Â Â Â - locked<0x00002aabe2211268> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>> )
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>> Â Â Â Â - locked<0x00002aac537126f0> Â (a [B)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>> 76)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>>> ner.java:87)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>> :82)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>> :262)
>>> Â Â Â Â - locked<0x00002aadb912fca8> Â (a
>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>> :114)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>>> HRegion.java:2344)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2301)
>>> Â Â Â Â - locked<0x00002aadcba8fe98> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2317)
>>> Â Â Â Â - locked<0x00002aadcba8fe98> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>> va:1832)
>>> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
> Source)
>>> Â Â Â Â at
>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>> Impl.java:43)
>>> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
>>> Â Â Â Â at
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>> 9)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "IPC Server handler 0 on 60020" daemon prio=10 tid=0x0000000001534000
>>> nid=0x553d waiting for monitor entry [0x0000000043efc000]
>>> Â Â java.lang.Thread.State: BLOCKED (on object monitor)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>>> java:2317)
>>> Â Â Â Â - waiting to lock<0x00002aabc7a3d258> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>>> va:1832)
>>> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
> Source)
>>> Â Â Â Â at
>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>> Impl.java:43)
>>> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
>>> Â Â Â Â at
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>>> 9)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "IPC Server listener on 60020" daemon prio=10 tid=0x00002aae4054e000
>>> nid=0x553c runnable [0x0000000043dfb000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> Â Â Â Â at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>> Â Â Â Â at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>> Â Â Â Â at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> Â Â Â Â - locked<0x00002aaabc18bdf8> Â (a sun.nio.ch.Util$1)
>>> Â Â Â Â - locked<0x00002aaabc18bde0> Â (a
>>> java.util.Collections$UnmodifiableSet)
>>> Â Â Â Â - locked<0x00002aaabbb1ced8> Â (a
> sun.nio.ch.EPollSelectorImpl)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:41
>>> 4)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "IPC Server Responder" daemon prio=10 tid=0x00002aae4053d800
> nid=0x553b
>>> runnable [0x0000000043cfa000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> Â Â Â Â at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>> Â Â Â Â at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>> Â Â Â Â at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> Â Â Â Â - locked<0x00002aaabc2ceed8> Â (a sun.nio.ch.Util$1)
>>> Â Â Â Â - locked<0x00002aaabc2ceef0> Â (a
>>> java.util.Collections$UnmodifiableSet)
>>> Â Â Â Â - locked<0x00002aaabbb33fa8> Â (a
> sun.nio.ch.EPollSelectorImpl)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:5
>>> 88)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "Timer-0" daemon prio=10 tid=0x00002aae4053c800 nid=0x553a in
>>> Object.wait() [0x0000000043bf9000]
>>> Â Â java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>> Â Â Â Â at java.lang.Object.wait(Native Method)
>>> Â Â Â Â at java.util.TimerThread.mainLoop(Timer.java:531)
>>> Â Â Â Â - locked<0x00002aaabd062850> Â (a java.util.TaskQueue)
>>> Â Â Â Â at java.util.TimerThread.run(Timer.java:484)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "983053578@qtp-1891789590-1 - Acceptor0
>>> SelectChannelConnector@0.0.0.0:60030" prio=10 tid=0x00002aae40381000
>>> nid=0x5539 runnable [0x0000000043af8000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> Â Â Â Â at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>> Â Â Â Â at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>> Â Â Â Â at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> Â Â Â Â - locked<0x00002aaabd07c460> Â (a sun.nio.ch.Util$1)
>>> Â Â Â Â - locked<0x00002aaabd07c478> Â (a
>>> java.util.Collections$UnmodifiableSet)
>>> Â Â Â Â - locked<0x00002aaabbb26b78> Â (a
> sun.nio.ch.EPollSelectorImpl)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> Â Â Â Â at
>>>
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.ja
>>> va:498)
>>> Â Â Â Â at
>>> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
>>> Â Â Â Â at
>>>
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnect
>>> or.java:124)
>>> Â Â Â Â at
>>>
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:
>>> 708)
>>> Â Â Â Â at
>>>
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java
>>> :582)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "1148428095@qtp-1891789590-0" prio=10 tid=0x00002aae403d2000
> nid=0x5538
>>> in Object.wait() [0x00000000439f7000]
>>> Â Â java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>> Â Â Â Â at java.lang.Object.wait(Native Method)
>>> Â Â Â Â at
>>>
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java
>>> :626)
>>> Â Â Â Â - locked<0x00002aaabbc4c8f0> Â (a
>>> org.mortbay.thread.QueuedThreadPool$PoolThread)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "regionserver60020.leaseChecker" daemon prio=10
> tid=0x00002aae402ee800
>>> nid=0x5537 waiting for monitor entry [0x00000000438f6000]
>>> Â Â java.lang.Thread.State: BLOCKED (on object monitor)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.close(HRegion
>>> .java:2409)
>>> Â Â Â Â - waiting to lock<0x00002aadcba8fe98> Â (a
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegionServer$ScannerListener.lease
>>> Expired(HRegionServer.java:1892)
>>> Â Â Â Â at
>>> org.apache.hadoop.hbase.regionserver.Leases.run(Leases.java:99)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "regionserver60020.majorCompactionChecker" daemon prio=10
>>> tid=0x00002aae402ec800 nid=0x5536 in Object.wait()
> [0x00000000437f5000]
>>> Â Â java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>> Â Â Â Â at java.lang.Object.wait(Native Method)
>>> Â Â Â Â - waiting on<0x00002aaabc1f7bf0> Â (a java.lang.Object)
>>> Â Â Â Â at
> org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
>>> Â Â Â Â - locked<0x00002aaabc1f7bf0> Â (a java.lang.Object)
>>> Â Â Â Â at org.apache.hadoop.hbase.Chore.run(Chore.java:74)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "regionserver60020.compactor" daemon prio=10 tid=0x00002aae402eb000
>>> nid=0x5535 runnable [0x00000000436f4000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> Â Â Â Â at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>> Â Â Â Â at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>> Â Â Â Â at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> Â Â Â Â - locked<0x00002aab26f462e0> Â (a sun.nio.ch.Util$1)
>>> Â Â Â Â - locked<0x00002aab26f462c8> Â (a
>>> java.util.Collections$UnmodifiableSet)
>>> Â Â Â Â - locked<0x00002aaaeba1ac80> Â (a
> sun.nio.ch.EPollSelectorImpl)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>>> thTimeout.java:332)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>>> 157)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>> Â Â Â Â - locked<0x00002aab44c85258> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>>> 50)
>>> Â Â Â Â - locked<0x00002aaac6caf140> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>>> a:237)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>>> Â Â Â Â at
>>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>>> Â Â Â Â - locked<0x00002aaac6caf140> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>>> Â Â Â Â - locked<0x00002aaac6caf140> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>>> a:1899)
>>> Â Â Â Â - locked<0x00002aaabd63fbb0> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>>> )
>>> Â Â Â Â - locked<0x00002aaabd63fbb0> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>>> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>>> dRangeFileInputStream.java:105)
>>> Â Â Â Â - locked<0x00002aab21b4e148> Â (a
>>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>> Â Â Â Â at
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>> Â Â Â Â - locked<0x00002aaaaedc39d0> Â (a java.io.BufferedInputStream)
>>> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>>> )
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>>> Â Â Â Â - locked<0x00002aab458cc280> Â (a [B)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>>> 76)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>>> ner.java:87)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>>> :82)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>> :297)
>>> Â Â Â Â - locked<0x00002aaace390ec0> Â (a
>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>>> :326)
>>> Â Â Â Â - locked<0x00002aaace390ec0> Â (a
>>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>> Â Â Â Â at
>>> org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:927)
>>> Â Â Â Â at
>>> org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:733)
>>> Â Â Â Â - locked<0x00002aaacd20e840> Â (a java.lang.Object)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:
>>> 769)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:
>>> 714)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplit
>>> Thread.java:81)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â -<0x00002aaabce779f8> Â (a
>>> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>>>
>>> "regionserver60020.cacheFlusher" daemon prio=10
> tid=0x00002aae402d1800
>>> nid=0x5534 waiting on condition [0x00000000435f3000]
>>> Â Â java.lang.Thread.State: TIMED_WAITING (parking)
>>> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>>> Â Â Â Â - parking to wait for<0x00002aaabc241a10> Â (a
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> Â Â Â Â at
>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>>> Â Â Â Â at
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>> aitNanos(AbstractQueuedSynchronizer.java:2081)
>>> Â Â Â Â at java.util.concurrent.DelayQueue.poll(DelayQueue.java:230)
>>> Â Â Â Â at java.util.concurrent.DelayQueue.poll(DelayQueue.java:68)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher
>>> .java:216)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "regionserver60020.logRoller" daemon prio=10 tid=0x00002aae402d1000
>>> nid=0x5533 in Object.wait() [0x00000000434f2000]
>>> Â Â java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>> Â Â Â Â at java.lang.Object.wait(Native Method)
>>> Â Â Â Â at
>>> org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:78)
>>> Â Â Â Â - locked<0x00002aaabcec18e0> Â (a
>>> java.util.concurrent.atomic.AtomicBoolean)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "Timer thread for monitoring jvm" daemon prio=10
> tid=0x00002aae402b4000
>>> nid=0x5532 in Object.wait() [0x00000000433f1000]
>>> Â Â java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>> Â Â Â Â at java.lang.Object.wait(Native Method)
>>> Â Â Â Â at java.util.TimerThread.mainLoop(Timer.java:531)
>>> Â Â Â Â - locked<0x00002aaabcedce48> Â (a java.util.TaskQueue)
>>> Â Â Â Â at java.util.TimerThread.run(Timer.java:484)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "Timer thread for monitoring hbase" daemon prio=10
>>> tid=0x00002aae402b2800 nid=0x5531 in Object.wait()
> [0x00000000432f0000]
>>> Â Â java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>> Â Â Â Â at java.lang.Object.wait(Native Method)
>>> Â Â Â Â at java.util.TimerThread.mainLoop(Timer.java:531)
>>> Â Â Â Â - locked<0x00002aaabceca148> Â (a java.util.TaskQueue)
>>> Â Â Â Â at java.util.TimerThread.run(Timer.java:484)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "regionserver60020.logSyncer" daemon prio=10 tid=0x00002aae402b1000
>>> nid=0x5530 waiting on condition [0x00000000431ef000]
>>> Â Â java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> Â Â Â Â at java.lang.Thread.sleep(Native Method)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:96
>>> 3)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "LeaseChecker" daemon prio=10 tid=0x00002aae4019a800 nid=0x552f
>>> sleeping[0x00000000430ee000]
>>> Â Â java.lang.Thread.State: TIMED_WAITING (sleeping)
>>> Â Â Â Â at java.lang.Thread.sleep(Native Method)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1167)
>>> Â Â Â Â at java.lang.Thread.run(Thread.java:636)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "IPC Client (47) connection to
>>> doop10.dt.sv4.decarta.com/10.241.8.230:60000 from an unknown user"
>>> daemon prio=10 tid=0x00000000015c9800 nid=0x5528 in Object.wait()
>>> [0x0000000042cea000]
>>> Â Â java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>> Â Â Â Â at java.lang.Object.wait(Native Method)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.waitForWork(HBaseClie
>>> nt.java:431)
>>> Â Â Â Â - locked<0x00002aaabc24dfc0> Â (a
>>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:
>>> 476)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "DestroyJavaVM" prio=10 tid=0x00002aae4015c800 nid=0x54fa waiting on
>>> condition [0x0000000000000000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "regionserver60020-EventThread" daemon prio=10 tid=0x00000000015c5000
>>> nid=0x5526 waiting on condition [0x0000000042be9000]
>>> Â Â java.lang.Thread.State: WAITING (parking)
>>> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>>> Â Â Â Â - parking to wait for<0x00002aaabc241a60> Â (a
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> Â Â Â Â at
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>> Â Â Â Â at
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>> ait(AbstractQueuedSynchronizer.java:2043)
>>> Â Â Â Â at
>>>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>>> 86)
>>> Â Â Â Â at
>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "regionserver60020-SendThread(doop3.dt.sv4.decarta.com:2181)" daemon
>>> prio=10 tid=0x0000000001486800 nid=0x5525 runnable
> [0x0000000042ae8000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> Â Â Â Â at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>> Â Â Â Â at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>> Â Â Â Â at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> Â Â Â Â - locked<0x00002aaabbb3e538> Â (a sun.nio.ch.Util$1)
>>> Â Â Â Â - locked<0x00002aaabbb3e550> Â (a
>>> java.util.Collections$UnmodifiableSet)
>>> Â Â Â Â - locked<0x00002aaabbb1db08> Â (a
> sun.nio.ch.EPollSelectorImpl)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> Â Â Â Â at
>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "regionserver60020" prio=10 tid=0x00002aae40187000 nid=0x5523 waiting
> on
>>> condition [0x000000004028d000]
>>> Â Â java.lang.Thread.State: TIMED_WAITING (parking)
>>> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>>> Â Â Â Â - parking to wait for<0x00002aaabc2faea0> Â (a
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> Â Â Â Â at
>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>>> Â Â Â Â at
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>> aitNanos(AbstractQueuedSynchronizer.java:2081)
>>> Â Â Â Â at
>>>
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:4
>>> 23)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.jav
>>> a:621)
>>> Â Â Â Â at java.lang.Thread.run(Thread.java:636)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "Timer thread for monitoring rpc" daemon prio=10
> tid=0x00002aae40135800
>>> nid=0x5522 in Object.wait() [0x00000000429e7000]
>>> Â Â java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>> Â Â Â Â at java.lang.Object.wait(Native Method)
>>> Â Â Â Â at java.util.TimerThread.mainLoop(Timer.java:531)
>>> Â Â Â Â - locked<0x00002aaabbb3cb08> Â (a java.util.TaskQueue)
>>> Â Â Â Â at java.util.TimerThread.run(Timer.java:484)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "IPC Reader 9 on port 60020" prio=10 tid=0x00002aae40133000
> nid=0x5521
>>> runnable [0x00000000428e6000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> Â Â Â Â at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>> Â Â Â Â at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>> Â Â Â Â at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> Â Â Â Â - locked<0x00002aaabbb010a8> Â (a sun.nio.ch.Util$1)
>>> Â Â Â Â - locked<0x00002aaabbb010c0> Â (a
>>> java.util.Collections$UnmodifiableSet)
>>> Â Â Â Â - locked<0x00002aaabbb1e198> Â (a
> sun.nio.ch.EPollSelectorImpl)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
>>> java:305)
>>> Â Â Â Â - locked<0x00002aaabbb250e8> Â (a
>>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>> a:1110)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>> va:603)
>>> Â Â Â Â at java.lang.Thread.run(Thread.java:636)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â -<0x00002aaabbb4b008> Â (a
>>> java.util.concurrent.ThreadPoolExecutor$Worker)
>>>
>>> "IPC Reader 8 on port 60020" prio=10 tid=0x00002aae40118000
> nid=0x5520
>>> runnable [0x00000000427e5000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> Â Â Â Â at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>> Â Â Â Â at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>> Â Â Â Â at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> Â Â Â Â - locked<0x00002aaabbb16118> Â (a sun.nio.ch.Util$1)
>>> Â Â Â Â - locked<0x00002aaabbb16130> Â (a
>>> java.util.Collections$UnmodifiableSet)
>>> Â Â Â Â - locked<0x00002aaabbb144f8> Â (a
> sun.nio.ch.EPollSelectorImpl)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
>>> java:305)
>>> Â Â Â Â - locked<0x00002aaabbb30308> Â (a
>>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>> a:1110)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>> va:603)
>>> Â Â Â Â at java.lang.Thread.run(Thread.java:636)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â -<0x00002aaabbb32888> Â (a
>>> java.util.concurrent.ThreadPoolExecutor$Worker)
>>>
>>> "IPC Reader 7 on port 60020" prio=10 tid=0x00002aae400fd000
> nid=0x551f
>>> runnable [0x00000000426e4000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> Â Â Â Â at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>> Â Â Â Â at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>> Â Â Â Â at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> Â Â Â Â - locked<0x00002aaabbb2a228> Â (a sun.nio.ch.Util$1)
>>> Â Â Â Â - locked<0x00002aaabbb2a240> Â (a
>>> java.util.Collections$UnmodifiableSet)
>>> Â Â Â Â - locked<0x00002aaabbb28ab8> Â (a
> sun.nio.ch.EPollSelectorImpl)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
>>> java:305)
>>> Â Â Â Â - locked<0x00002aaabbb2a6d8> Â (a
>>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>> a:1110)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>> va:603)
>>> Â Â Â Â at java.lang.Thread.run(Thread.java:636)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â -<0x00002aaabbb4a068> Â (a
>>> java.util.concurrent.ThreadPoolExecutor$Worker)
>>>
>>> "IPC Reader 6 on port 60020" prio=10 tid=0x00002aae400e2800
> nid=0x551e
>>> runnable [0x00000000425e3000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> Â Â Â Â at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>> Â Â Â Â at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>> Â Â Â Â at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> Â Â Â Â - locked<0x00002aaabbb27028> Â (a sun.nio.ch.Util$1)
>>> Â Â Â Â - locked<0x00002aaabbb27040> Â (a
>>> java.util.Collections$UnmodifiableSet)
>>> Â Â Â Â - locked<0x00002aaabbb258b8> Â (a
> sun.nio.ch.EPollSelectorImpl)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
>>> java:305)
>>> Â Â Â Â - locked<0x00002aaabbb282e8> Â (a
>>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>> a:1110)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>> va:603)
>>> Â Â Â Â at java.lang.Thread.run(Thread.java:636)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â -<0x00002aaabbb4fc98> Â (a
>>> java.util.concurrent.ThreadPoolExecutor$Worker)
>>>
>>> "IPC Reader 5 on port 60020" prio=10 tid=0x00002aae400c7800
> nid=0x551d
>>> runnable [0x00000000413e5000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> Â Â Â Â at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>> Â Â Â Â at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>> Â Â Â Â at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> Â Â Â Â - locked<0x00002aaabbb2ea08> Â (a sun.nio.ch.Util$1)
>>> Â Â Â Â - locked<0x00002aaabbb2ea20> Â (a
>>> java.util.Collections$UnmodifiableSet)
>>> Â Â Â Â - locked<0x00002aaabbb2d298> Â (a
> sun.nio.ch.EPollSelectorImpl)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
>>> java:305)
>>> Â Â Â Â - locked<0x00002aaabbb50c38> Â (a
>>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>> a:1110)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>> va:603)
>>> Â Â Â Â at java.lang.Thread.run(Thread.java:636)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â -<0x00002aaabbb523a8> Â (a
>>> java.util.concurrent.ThreadPoolExecutor$Worker)
>>>
>>> "IPC Reader 4 on port 60020" prio=10 tid=0x00002aae400ac800
> nid=0x551c
>>> runnable [0x00000000412e4000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> Â Â Â Â at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>> Â Â Â Â at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>> Â Â Â Â at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> Â Â Â Â - locked<0x00002aaabbb2aea8> Â (a sun.nio.ch.Util$1)
>>> Â Â Â Â - locked<0x00002aaabbb2aec0> Â (a
>>> java.util.Collections$UnmodifiableSet)
>>> Â Â Â Â - locked<0x00002aaabbb1ca28> Â (a
> sun.nio.ch.EPollSelectorImpl)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
>>> java:305)
>>> Â Â Â Â - locked<0x00002aaabbb2c168> Â (a
>>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>> a:1110)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>> va:603)
>>> Â Â Â Â at java.lang.Thread.run(Thread.java:636)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â -<0x00002aaabbb48128> Â (a
>>> java.util.concurrent.ThreadPoolExecutor$Worker)
>>>
>>> "IPC Reader 3 on port 60020" prio=10 tid=0x00002aae40091800
> nid=0x551b
>>> runnable [0x00000000411e3000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> Â Â Â Â at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>> Â Â Â Â at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>> Â Â Â Â at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> Â Â Â Â - locked<0x00002aaabbb15c68> Â (a sun.nio.ch.Util$1)
>>> Â Â Â Â - locked<0x00002aaabbb15c80> Â (a
>>> java.util.Collections$UnmodifiableSet)
>>> Â Â Â Â - locked<0x00002aaabbb12d88> Â (a
> sun.nio.ch.EPollSelectorImpl)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
>>> java:305)
>>> Â Â Â Â - locked<0x00002aaabbb165c8> Â (a
>>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>> a:1110)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>> va:603)
>>> Â Â Â Â at java.lang.Thread.run(Thread.java:636)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â -<0x00002aaabbb30ad8> Â (a
>>> java.util.concurrent.ThreadPoolExecutor$Worker)
>>>
>>> "IPC Reader 2 on port 60020" prio=10 tid=0x00002aae40077000
> nid=0x551a
>>> runnable [0x00000000410e2000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> Â Â Â Â at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>> Â Â Â Â at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>> Â Â Â Â at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> Â Â Â Â - locked<0x00002aaabbb3d148> Â (a sun.nio.ch.Util$1)
>>> Â Â Â Â - locked<0x00002aaabbb3d160> Â (a
>>> java.util.Collections$UnmodifiableSet)
>>> Â Â Â Â - locked<0x00002aaabbb33828> Â (a
> sun.nio.ch.EPollSelectorImpl)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
>>> java:305)
>>> Â Â Â Â - locked<0x00002aaabbb3ebd8> Â (a
>>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>> a:1110)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>> va:603)
>>> Â Â Â Â at java.lang.Thread.run(Thread.java:636)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â -<0x00002aaabbb40028> Â (a
>>> java.util.concurrent.ThreadPoolExecutor$Worker)
>>>
>>> "IPC Reader 1 on port 60020" prio=10 tid=0x00002aae4005d000
> nid=0x5519
>>> runnable [0x0000000040fe1000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> Â Â Â Â at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>> Â Â Â Â at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>> Â Â Â Â at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> Â Â Â Â - locked<0x00002aaabbb15c98> Â (a sun.nio.ch.Util$1)
>>> Â Â Â Â - locked<0x00002aaabbb15cb0> Â (a
>>> java.util.Collections$UnmodifiableSet)
>>> Â Â Â Â - locked<0x00002aaabbb12e00> Â (a
> sun.nio.ch.EPollSelectorImpl)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
>>> java:305)
>>> Â Â Â Â - locked<0x00002aaabbb165f0> Â (a
>>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>> a:1110)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>> va:603)
>>> Â Â Â Â at java.lang.Thread.run(Thread.java:636)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â -<0x00002aaabbb30b28> Â (a
>>> java.util.concurrent.ThreadPoolExecutor$Worker)
>>>
>>> "IPC Reader 0 on port 60020" prio=10 tid=0x00002aae40003000
> nid=0x5518
>>> runnable [0x0000000040ee0000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> Â Â Â Â at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>> Â Â Â Â at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>> Â Â Â Â at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> Â Â Â Â - locked<0x00002aaabbb16148> Â (a sun.nio.ch.Util$1)
>>> Â Â Â Â - locked<0x00002aaabbb16160> Â (a
>>> java.util.Collections$UnmodifiableSet)
>>> Â Â Â Â - locked<0x00002aaabbb14570> Â (a
> sun.nio.ch.EPollSelectorImpl)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>>> Â Â Â Â at
>>>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
>>> java:305)
>>> Â Â Â Â - locked<0x00002aaabbb30330> Â (a
>>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>> a:1110)
>>> Â Â Â Â at
>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>> va:603)
>>> Â Â Â Â at java.lang.Thread.run(Thread.java:636)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â -<0x00002aaabbb328d8> Â (a
>>> java.util.concurrent.ThreadPoolExecutor$Worker)
>>>
>>> "main-EventThread" daemon prio=10 tid=0x00002aae3c1b6000 nid=0x5513
>>> waiting on condition [0x00000000424e2000]
>>> Â Â java.lang.Thread.State: WAITING (parking)
>>> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>>> Â Â Â Â - parking to wait for<0x00002aaabbb16618> Â (a
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> Â Â Â Â at
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>> Â Â Â Â at
>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>>> ait(AbstractQueuedSynchronizer.java:2043)
>>> Â Â Â Â at
>>>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>>> 86)
>>> Â Â Â Â at
>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "main-SendThread(doop7.dt.sv4.decarta.com:2181)" daemon prio=10
>>> tid=0x00002aae3c11b000 nid=0x5512 runnable [0x00000000423e1000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> Â Â Â Â at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>>> Â Â Â Â at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>>> Â Â Â Â at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> Â Â Â Â - locked<0x00002aaabbb16178> Â (a sun.nio.ch.Util$1)
>>> Â Â Â Â - locked<0x00002aaabbb16190> Â (a
>>> java.util.Collections$UnmodifiableSet)
>>> Â Â Â Â - locked<0x00002aaabbb145e8> Â (a
> sun.nio.ch.EPollSelectorImpl)
>>> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> Â Â Â Â at
>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "Low Memory Detector" daemon prio=10 tid=0x000000000126c800
> nid=0x5510
>>> runnable [0x0000000000000000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "CompilerThread1" daemon prio=10 tid=0x000000000126a800 nid=0x550f
>>> waiting on condition [0x0000000000000000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "CompilerThread0" daemon prio=10 tid=0x0000000001266000 nid=0x550e
>>> waiting on condition [0x0000000000000000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "Signal Dispatcher" daemon prio=10 tid=0x0000000001264000 nid=0x550d
>>> runnable [0x0000000000000000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "Surrogate Locker Thread (CMS)" daemon prio=10 tid=0x0000000001262000
>>> nid=0x550c waiting on condition [0x0000000000000000]
>>> Â Â java.lang.Thread.State: RUNNABLE
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "Finalizer" daemon prio=10 tid=0x000000000123b800 nid=0x550b in
>>> Object.wait() [0x00000000422e0000]
>>> Â Â java.lang.Thread.State: WAITING (on object monitor)
>>> Â Â Â Â at java.lang.Object.wait(Native Method)
>>> Â Â Â Â at
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:133)
>>> Â Â Â Â - locked<0x00002aaabbb15cf8> Â (a
>>> java.lang.ref.ReferenceQueue$Lock)
>>> Â Â Â Â at
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:149)
>>> Â Â Â Â at
>>> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "Reference Handler" daemon prio=10 tid=0x0000000001239800 nid=0x550a
> in
>>> Object.wait() [0x00000000421df000]
>>> Â Â java.lang.Thread.State: WAITING (on object monitor)
>>> Â Â Â Â at java.lang.Object.wait(Native Method)
>>> Â Â Â Â at java.lang.Object.wait(Object.java:502)
>>> Â Â Â Â at
>>> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
>>> Â Â Â Â - locked<0x00002aaabbb16d98> Â (a
> java.lang.ref.Reference$Lock)
>>>
>>> Â Â Locked ownable synchronizers:
>>> Â Â Â Â - None
>>>
>>> "VM Thread" prio=10 tid=0x0000000001234800 nid=0x5509 runnable
>>>
>>> "Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x0000000000fe9000
>>> nid=0x54fb runnable
>>>
>>> "Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x0000000000feb000
>>> nid=0x54fc runnable
>>>
>>> "Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x0000000000fec800
>>> nid=0x54fd runnable
>>>
>>> "Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x0000000000fee800
>>> nid=0x54fe runnable
>>>
>>> "Gang worker#4 (Parallel GC Threads)" prio=10 tid=0x0000000000ff0800
>>> nid=0x54ff runnable
>>>
>>> "Gang worker#5 (Parallel GC Threads)" prio=10 tid=0x0000000000ff2000
>>> nid=0x5500 runnable
>>>
>>> "Gang worker#6 (Parallel GC Threads)" prio=10 tid=0x0000000000ff4000
>>> nid=0x5501 runnable
>>>
>>> "Gang worker#7 (Parallel GC Threads)" prio=10 tid=0x0000000000ff6000
>>> nid=0x5502 runnable
>>>
>>> "Gang worker#8 (Parallel GC Threads)" prio=10 tid=0x0000000000ff7800
>>> nid=0x5503 runnable
>>>
>>> "Gang worker#9 (Parallel GC Threads)" prio=10 tid=0x0000000000ff9800
>>> nid=0x5504 runnable
>>>
>>> "Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00000000010f4000
>>> nid=0x5508 runnable
>>> "Gang worker#0 (Parallel CMS Threads)" prio=10 tid=0x00000000010ee000
>>> nid=0x5505 runnable
>>>
>>> "Gang worker#1 (Parallel CMS Threads)" prio=10 tid=0x00000000010f0000
>>> nid=0x5506 runnable
>>>
>>> "Gang worker#2 (Parallel CMS Threads)" prio=10 tid=0x00000000010f2000
>>> nid=0x5507 runnable
>>>
>>> "VM Periodic Task Thread" prio=10 tid=0x000000000126f800 nid=0x5511
>>> waiting on condition
>>>
>>> JNI global references: 906
>>>
>>>
>>> -----Original Message-----
>>> From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of
>>> Stack
>>> Sent: Sunday, September 11, 2011 2:27 PM
>>> To: user@hbase.apache.org
>>> Subject: Re: scanner deadlock?
>>>
>>> 0.90.4 fixes two deadlocks (HBASE-4101 and HBASE-4077). Â Since then,
>>> there is HBASE-4367 (Which has a posted patch).
>>>
>>> Below sounds like slowness. Â Can you thread dump the particular
>>> regionserver and see what its up to? Â Is there other loading on the
>>> system at the time? Â For example, loading on hdfs? Â Anything in the
>>> hdfs logs for the datanode running beside the slow regionserver?
>>>
>>> St.Ack
>>>
>>> On Sat, Sep 10, 2011 at 5:50 PM, Geoff Hendrey<gh...@decarta.com>
>>> wrote:
>>>> Hi all -
>>>>
>>>>
>>>>
>>>> I'm still dealing with the saga of ScannerTimeoutException,
>>>> UnknownScannerException, etc. I rewrote my code, in the hope that
>>> simply
>>>> a different approach and some different code paths might yield
> better
>>>> results. No change. I tried many variations (caching 1 row vs
> caching
>>>> many rows, changing the regionserver's lease, increasing the number
> of
>>>> allowed zookeeper connections, etc). I created a fresh table on the
>>>> thought that maybe there was some problem with the table...no
> change.
>>>>
>>>>
>>>>
>>>> I am dealing with what appears to be some sort of scanner deadlock.
> I
>>>> have a total order partitioned mapreduce job. In the reducer, as
> long
>>> as
>>>> I use just one reducer, the task finishes quickly. But as soon as
> more
>>>> than one reducer opens a scanner, the tasks procede in what I would
>>> call
>>>> a "jittery" lock step. They both are able to do ResultScanner.next()
> a
>>>> few times, but then the call to next() freezes for a long period,
> and
>>>> ScannerTimeoutException. I catch the exception, and get a new
>>>> ResultScanner, and the pattern repeats. Jittery lockstep consisting
> of
>>>> being able to get a few successful next() calls, then a lock up.
>>>>
>>>>
>>>>
>>>> I have not yet tried upgrading from 90.1 to higher. Nor have I tried
>>>> tsuna's async client. Can anyone think of anything else I can try to
>>>> resolve this? I've sunk quite a few late nights into this, and would
>>> be
>>>> very excited to find a solution.
>>>>
>>>>
>>>>
>>>> -geoff
>>>>
>>>>
>>>
>
> --
> Eric
> http://about.echarles.net
>
RE: scanner deadlock?
Posted by Geoff Hendrey <gh...@decarta.com>.
Wow, no I had not been using any tool other than eyeballs. I will
certainly apply this tool to our jstack.
Thanks!!
-geoff
-----Original Message-----
From: Eric Charles [mailto:eric.umg.charles@gmail.com]
Sent: Monday, September 12, 2011 10:28 AM
To: user@hbase.apache.org
Subject: Re: scanner deadlock?
Hi,
Probably you've already done something similar, but I happily use 'IBM
Thread and Monitor Dump Analyzer for Java' to be 100% sure there are no
dead lock. The tool gives you a nice short report on your threads.
I tried to run it on your dump, but the analyzer does not like
mail-wrapped lines.
Thx.
[1]
https://www.ibm.com/developerworks/mydeveloperworks/groups/service/html/
communityview?communityUuid=2245aa39-fa5c-4475-b891-14c205f7333c
On 12/09/11 09:46, Stack wrote:
> On Sun, Sep 11, 2011 at 2:40 PM, Geoff Hendrey<gh...@decarta.com>
wrote:
>> Hi,
>>
>> The cluster is definitely loaded. We are running a MR job with 130
>> reducers, each using a scanner. Ganglia shows an overall CPU
utilization
>> of around 40%, with network pushing 800MB/s. The problem may
correlate
>> with a major_compact I initiated couple days ago. Even today, I still
>> see a lot of compacting (continuous) on region servers.
>>
>> Here is the thread dump from jstack. It does not reveal any explicit
>> deadlock. At the time I dumped the jstack, the region server logs are
>> reporting "IPC Server handler...ClosedChannelException". As soon as
this
>> begins occurring, our network throughput, as seen on Ganglia drops
from
>> 800MB/s down to virtually zero. Then of course there is the cascade
of
>> failures, eventually leading to the regionservers shutting down.
>>
>
>
> Looking at the thread dump, as you say, few are 'BLOCKED' (There are a
> few BLOCKED thread trying to go into a next). But most are deep in a
> next stuck polling looking to read. If you thread dump a few times
> in a row, do any seem to be making progress? (They may be making
> progress but slowly, as you say)
>
> So, is it possible you have a slow datanode in your cluster? Can you
> tell who all are connected too... lsof'ing? That might give you a
> clue. Perhaps there is a node with sick disks (check dmesg). Or
> perhaps the machine's nework card is sick?
>
> St.Ack
>
>> I don't see anything unusual in the datanode logs on the same server.
>> Just HDFS_READ and HDFS_WRITE for the most part.
>>
>> jstack here:
>> ===================================================================
>> 2011-09-11 14:28:52
>> Full thread dump OpenJDK 64-Bit Server VM (14.0-b16 mixed mode):
>>
>> "IPC Client (47) connection to
>> doop10.dt.sv4.decarta.com/10.241.8.230:54310 from hroot" daemon
prio=10
>> tid=0x0000000001492800 nid=0x75cb in Object.wait()
[0x0000000047633000]
>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at
>> org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:676)
>> - locked<0x00002aaaaf680000> (a
>> org.apache.hadoop.ipc.Client$Connection)
>> at
org.apache.hadoop.ipc.Client$Connection.run(Client.java:719)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "ResponseProcessor for block blk_2613854253703670198_5032689" daemon
>> prio=10 tid=0x0000000001988800 nid=0x75bb runnable
[0x0000000046926000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aab69337248> (a sun.nio.ch.Util$1)
>> - locked<0x00002aab69337230> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaaf3897930> (a
sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
java.io.DataInputStream.readFully(DataInputStream.java:195)
>> at java.io.DataInputStream.readLong(DataInputStream.java:416)
>> at
>>
org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFie
>> lds(DataTransferProtocol.java:120)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(D
>> FSClient.java:2638)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "DataStreamer for file
>>
/hbase-20.6/NAM_CLUSTERKEYS3/60db9e0113c7578ed913dc37e0856954/.tmp/42977
>> 03314038441248 block blk_2613854253703670198_5032689" daemon prio=10
>> tid=0x00002aae40e8e800 nid=0x75b8 in Object.wait()
[0x0000000046e2b000]
>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSCli
>> ent.java:2483)
>> - locked<0x00002aab6551a2d8> (a java.util.LinkedList)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "ResponseProcessor for block blk_8900384789253382699_5032549" daemon
>> prio=10 tid=0x00000000013cd000 nid=0x71d0 runnable
[0x0000000047431000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aaae0569530> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaae0569518> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaadb4c5c68> (a
sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
java.io.DataInputStream.readFully(DataInputStream.java:195)
>> at java.io.DataInputStream.readLong(DataInputStream.java:416)
>> at
>>
org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFie
>> lds(DataTransferProtocol.java:120)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(D
>> FSClient.java:2638)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "DataStreamer for file
>>
/hbase-20.6/.logs/doop5.dt.sv4.decarta.com,60020,1315768059645/doop5.dt.
>> sv4.decarta.com%3A60020.1315776307423 block
>> blk_8900384789253382699_5032549" daemon prio=10
tid=0x00000000016c3800
>> nid=0x71c8 in Object.wait() [0x000000004712e000]
>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSCli
>> ent.java:2483)
>> - locked<0x00002aab092a7f38> (a java.util.LinkedList)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "Attach Listener" daemon prio=10 tid=0x0000000001e32000 nid=0x50f4
>> waiting on condition [0x0000000000000000]
>> java.lang.Thread.State: RUNNABLE
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "RS_CLOSE_REGION-doop5.dt.sv4.decarta.com,60020,1315768059645-2"
daemon
>> prio=10 tid=0x00000000018cb800 nid=0x505 waiting on condition
>> [0x0000000047330000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabcefb428> (a
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> at
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> ait(AbstractQueuedSynchronizer.java:2043)
>> at
>>
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>> 86)
>> at
>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
>> 1043)
>> at
>>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>> a:1103)
>> at
>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>> va:603)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "RS_CLOSE_REGION-doop5.dt.sv4.decarta.com,60020,1315768059645-1"
daemon
>> prio=10 tid=0x000000000163e000 nid=0x504 waiting on condition
>> [0x0000000047532000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabcefb428> (a
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> at
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> ait(AbstractQueuedSynchronizer.java:2043)
>> at
>>
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>> 86)
>> at
>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
>> 1043)
>> at
>>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>> a:1103)
>> at
>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>> va:603)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "RS_CLOSE_REGION-doop5.dt.sv4.decarta.com,60020,1315768059645-0"
daemon
>> prio=10 tid=0x0000000001bff800 nid=0x503 waiting on condition
>> [0x0000000046f2c000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabcefb428> (a
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> at
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> ait(AbstractQueuedSynchronizer.java:2043)
>> at
>>
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>> 86)
>> at
>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
>> 1043)
>> at
>>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>> a:1103)
>> at
>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>> va:603)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "sendParams-6" daemon prio=10 tid=0x00002aae401f1800 nid=0x6d3b
waiting
>> on condition [0x000000004702d000]
>> java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabceea260> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>> at
>>
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>> at
>>
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Synchro
>> nousQueue.java:453)
>> at
>>
java.util.concurrent.SynchronousQueue$TransferStack.transfer(Synchronous
>> Queue.java:352)
>> at
>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:903)
>> at
>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
>> 1043)
>> at
>>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>> a:1103)
>> at
>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>> va:603)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "RS_OPEN_REGION-doop5.dt.sv4.decarta.com,60020,1315768059645-2"
daemon
>> prio=10 tid=0x0000000001a7f000 nid=0x5b56 waiting on condition
>> [0x0000000046d2a000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabcefb2e8> (a
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> at
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> ait(AbstractQueuedSynchronizer.java:2043)
>> at
>>
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>> 86)
>> at
>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
>> 1043)
>> at
>>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>> a:1103)
>> at
>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>> va:603)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "RS_OPEN_REGION-doop5.dt.sv4.decarta.com,60020,1315768059645-1"
daemon
>> prio=10 tid=0x0000000001af2800 nid=0x5b55 waiting on condition
>> [0x0000000042deb000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabcefb2e8> (a
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> at
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> ait(AbstractQueuedSynchronizer.java:2043)
>> at
>>
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>> 86)
>> at
>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
>> 1043)
>> at
>>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>> a:1103)
>> at
>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>> va:603)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "RS_OPEN_REGION-doop5.dt.sv4.decarta.com,60020,1315768059645-0"
daemon
>> prio=10 tid=0x00000000014b3800 nid=0x5b54 waiting on condition
>> [0x0000000046c29000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabcefb2e8> (a
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> at
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> ait(AbstractQueuedSynchronizer.java:2043)
>> at
>>
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>> 86)
>> at
>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
>> 1043)
>> at
>>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>> a:1103)
>> at
>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>> va:603)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "RS_OPEN_ROOT-doop5.dt.sv4.decarta.com,60020,1315768059645-0" daemon
>> prio=10 tid=0x0000000001ad2800 nid=0x556e waiting on condition
>> [0x0000000046a27000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabcee8870> (a
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> at
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> ait(AbstractQueuedSynchronizer.java:2043)
>> at
>>
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>> 86)
>> at
>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
>> 1043)
>> at
>>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>> a:1103)
>> at
>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>> va:603)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "LRU Statistics #0" prio=10 tid=0x00002aae4048a800 nid=0x5566 waiting
on
>> condition [0x0000000046825000]
>> java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabd0695c0> (a
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>>
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>> at
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> aitNanos(AbstractQueuedSynchronizer.java:2081)
>> at java.util.concurrent.DelayQueue.take(DelayQueue.java:193)
>> at
>>
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(S
>> cheduledThreadPoolExecutor.java:688)
>> at
>>
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(S
>> cheduledThreadPoolExecutor.java:681)
>> at
>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
>> 1043)
>> at
>>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>> a:1103)
>> at
>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>> va:603)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "LruBlockCache.EvictionThread" daemon prio=10 tid=0x00002aae4049f800
>> nid=0x5565 in Object.wait() [0x0000000046724000]
>> java.lang.Thread.State: WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at java.lang.Object.wait(Object.java:502)
>> at
>>
org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlo
>> ckCache.java:519)
>> - locked<0x00002aaabce92f50> (a
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "PRI IPC Server handler 9 on 60020" daemon prio=10
>> tid=0x00002aae4057e000 nid=0x5564 waiting on condition
>> [0x0000000046623000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabbb307e0> (a
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> at
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> ait(AbstractQueuedSynchronizer.java:2043)
>> at
>>
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>> 86)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
>> 5)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "PRI IPC Server handler 8 on 60020" daemon prio=10
>> tid=0x00002aae4057c800 nid=0x5563 waiting on condition
>> [0x0000000046522000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabbb307e0> (a
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> at
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> ait(AbstractQueuedSynchronizer.java:2043)
>> at
>>
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>> 86)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
>> 5)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "PRI IPC Server handler 7 on 60020" daemon prio=10
>> tid=0x00002aae40559000 nid=0x5562 waiting on condition
>> [0x0000000046421000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabbb307e0> (a
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> at
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> ait(AbstractQueuedSynchronizer.java:2043)
>> at
>>
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>> 86)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
>> 5)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "PRI IPC Server handler 6 on 60020" daemon prio=10
>> tid=0x00002aae40557800 nid=0x5561 waiting on condition
>> [0x0000000046320000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabbb307e0> (a
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> at
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> ait(AbstractQueuedSynchronizer.java:2043)
>> at
>>
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>> 86)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
>> 5)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "PRI IPC Server handler 5 on 60020" daemon prio=10
>> tid=0x00002aae40555800 nid=0x5560 waiting on condition
>> [0x000000004621f000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabbb307e0> (a
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> at
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> ait(AbstractQueuedSynchronizer.java:2043)
>> at
>>
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>> 86)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
>> 5)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "PRI IPC Server handler 4 on 60020" daemon prio=10
>> tid=0x00002aae40553800 nid=0x555f waiting on condition
>> [0x000000004611e000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabbb307e0> (a
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> at
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> ait(AbstractQueuedSynchronizer.java:2043)
>> at
>>
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>> 86)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
>> 5)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "PRI IPC Server handler 3 on 60020" daemon prio=10
>> tid=0x00002aae4054f000 nid=0x555e waiting on condition
>> [0x000000004601d000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabbb307e0> (a
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> at
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> ait(AbstractQueuedSynchronizer.java:2043)
>> at
>>
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>> 86)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
>> 5)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "PRI IPC Server handler 2 on 60020" daemon prio=10
>> tid=0x00000000012f8800 nid=0x555d waiting on condition
>> [0x0000000045f1c000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabbb307e0> (a
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> at
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> ait(AbstractQueuedSynchronizer.java:2043)
>> at
>>
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>> 86)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
>> 5)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "PRI IPC Server handler 1 on 60020" daemon prio=10
>> tid=0x0000000001991000 nid=0x555c waiting on condition
>> [0x0000000045e1b000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabbb307e0> (a
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> at
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> ait(AbstractQueuedSynchronizer.java:2043)
>> at
>>
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>> 86)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
>> 5)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "PRI IPC Server handler 0 on 60020" daemon prio=10
>> tid=0x000000000198e800 nid=0x555b waiting on condition
>> [0x0000000045d1a000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabbb307e0> (a
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> at
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> ait(AbstractQueuedSynchronizer.java:2043)
>> at
>>
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>> 86)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
>> 5)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 29 on 60020" daemon prio=10
tid=0x000000000198c800
>> nid=0x555a runnable [0x0000000045c18000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aac053b8450> (a sun.nio.ch.Util$1)
>> - locked<0x00002aac053b8438> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aab1037afc8> (a
sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aac92976a30> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aabc43be0a8> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aabc43be0a8> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aabc43be0a8> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aac1a4e0ea8> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aac1a4e0ea8> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>>
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aac237c37a8> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aad39485080> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aab0b5ba8a8> (a [B)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>> ner.java:87)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :82)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :262)
>> - locked<0x00002aab89f59aa8> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aab016d6988> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aab016d6988> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
Source)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 28 on 60020" daemon prio=10
tid=0x0000000001864000
>> nid=0x5559 runnable [0x0000000045b17000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aabaedb0608> (a sun.nio.ch.Util$1)
>> - locked<0x00002aabaedb05f0> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaca096fa50> (a
sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aac62a70b28> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aab856e6220> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aab856e6220> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aab856e6220> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aab0264afe8> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aab0264afe8> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>>
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aaae070be48> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aac18961940> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aabad8605d0> (a [B)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>> ner.java:87)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :82)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :262)
>> - locked<0x00002aab077e5d80> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aad30d57c18> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aad30d57c18> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
Source)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 27 on 60020" daemon prio=10
tid=0x0000000001862000
>> nid=0x5558 runnable [0x0000000045a16000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aab2ea61b70> (a sun.nio.ch.Util$1)
>> - locked<0x00002aab2ea61b88> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aacbb364578> (a
sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aab1ffc0d00> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aad890f4640> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aad890f4640> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aad890f4640> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aaba1282b10> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aaba1282b10> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>>
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aac528d15d0> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aaaaf68a630> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aabc5c948b8> (a [B)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
>> anner.java:115)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
>> va:255)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
>> va:394)
>> - locked<0x00002aab95832718> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :293)
>> - locked<0x00002aab95832718> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aab28d7de90> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aab28d7de90> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
Source)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 26 on 60020" daemon prio=10
tid=0x0000000001860000
>> nid=0x5557 runnable [0x0000000045915000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aabaaa6b808> (a sun.nio.ch.Util$1)
>> - locked<0x00002aabaaa6b7f0> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaaf3899ca8> (a
sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aaac1f59118> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aaac0e99db0> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aaac0e99db0> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aaac0e99db0> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aabb7221e88> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aabb7221e88> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>>
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aac28621da0> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aac18962690> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aab6876b850> (a [B)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.loadBlock(HFile.ja
>> va:1447)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.reseekTo(HFile.jav
>> a:1325)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(St
>> oreFileScanner.java:152)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
>> anner.java:110)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
>> va:255)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
>> va:394)
>> - locked<0x00002aaabe3468f8> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :293)
>> - locked<0x00002aaabe3468f8> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aac8e37dbc8> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aac8e37dbc8> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
Source)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 25 on 60020" daemon prio=10
tid=0x000000000173d800
>> nid=0x5556 runnable [0x0000000045814000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aab20178510> (a sun.nio.ch.Util$1)
>> - locked<0x00002aab20178528> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaad6187af0> (a
sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aac891a6610> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aac1de6d9f8> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aac1de6d9f8> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aac1de6d9f8> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aaac041c6c8> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aaac041c6c8> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>>
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aab25e717e0> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aac3e6e4e28> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aaad4a425f0> (a [B)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>> ner.java:87)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :82)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :262)
>> - locked<0x00002aaabd5eb7e0> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aac8e37dc68> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aac8e37dc68> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
Source)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 24 on 60020" daemon prio=10
tid=0x000000000173a000
>> nid=0x5555 waiting for monitor entry [0x0000000045714000]
>> java.lang.Thread.State: BLOCKED (on object monitor)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - waiting to lock<0x00002aab016d6618> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
Source)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 23 on 60020" daemon prio=10
tid=0x0000000001444000
>> nid=0x5554 runnable [0x0000000045612000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aad70044d30> (a sun.nio.ch.Util$1)
>> - locked<0x00002aad70044d48> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aab328f1a40> (a
sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aaac9d48e68> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aac1de6dd40> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aac1de6dd40> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aac1de6dd40> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aab39d88148> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aab39d88148> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>>
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aab7addd0d8> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aab2811e8e8> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aabe56e8858> (a [B)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>> ner.java:87)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :82)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :262)
>> - locked<0x00002aad15b7d360> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aabc7a3d258> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aabc7a3d258> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
Source)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 22 on 60020" daemon prio=10
tid=0x0000000001442000
>> nid=0x5553 runnable [0x0000000045511000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aabf6e38450> (a sun.nio.ch.Util$1)
>> - locked<0x00002aabf6e38468> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aabb9e99468> (a
sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aaae13c0340> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aad427a91c0> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aad427a91c0> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aad427a91c0> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aab7c7ca0c8> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aab7c7ca0c8> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>>
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aab3748f1a8> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aab52d56ed8> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aab60a86ca8> (a [B)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(St
>> oreFileScanner.java:158)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
>> anner.java:110)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
>> va:255)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
>> va:394)
>> - locked<0x00002aab706a3e98> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :293)
>> - locked<0x00002aab706a3e98> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aaae3489758> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aaae3489758> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
Source)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 21 on 60020" daemon prio=10
tid=0x0000000001792000
>> nid=0x5552 waiting for monitor entry [0x0000000045411000]
>> java.lang.Thread.State: BLOCKED (on object monitor)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - waiting to lock<0x00002aabc7a3d3e8> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
Source)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 20 on 60020" daemon prio=10
tid=0x000000000178e000
>> nid=0x5551 runnable [0x000000004530f000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aab3745e018> (a sun.nio.ch.Util$1)
>> - locked<0x00002aab3745e030> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aab7d5cc0c0> (a
sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aac035e4990> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aad890f42f8> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aad890f42f8> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aad890f42f8> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aab1baffa50> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aab1baffa50> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>>
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aaadef99fe8> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aac189625a0> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aabb88c7ce0> (a [B)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
>> anner.java:115)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
>> va:255)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
>> va:394)
>> - locked<0x00002aaadfa95ec0> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :293)
>> - locked<0x00002aaadfa95ec0> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aabf6832658> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aabf6832658> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
Source)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 19 on 60020" daemon prio=10
tid=0x0000000001b19800
>> nid=0x5550 runnable [0x000000004520e000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aab94af3848> (a sun.nio.ch.Util$1)
>> - locked<0x00002aab94af3830> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aac6ed3e1d8> (a
sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aaac1194578> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aad890f2600> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aad890f2600> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aad890f2600> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aaae78be750> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aaae78be750> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>>
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aabba906cc0> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aac18963ef0> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aaacd2974c0> (a [B)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
>> anner.java:115)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
>> va:255)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
>> va:394)
>> - locked<0x00002aab328ec218> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :293)
>> - locked<0x00002aab328ec218> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aabbaf1c898> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aabbaf1c898> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
Source)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 18 on 60020" daemon prio=10
tid=0x0000000001b17800
>> nid=0x554f runnable [0x000000004510d000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aaae98101b0> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaae9810198> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaaeba1b040> (a
sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aab1ac7d288> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aad890f0db8> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aad890f0db8> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aad890f0db8> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aab8c671d30> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aab8c671d30> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>>
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aab8c4f3de0> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aac18963dd0> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aab8d9fd7c8> (a [B)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
>> anner.java:115)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
>> va:255)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
>> va:394)
>> - locked<0x00002aaba69901b8> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :293)
>> - locked<0x00002aaba69901b8> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aabbaf1e468> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aabbaf1e468> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
Source)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 17 on 60020" daemon prio=10
tid=0x0000000001b15800
>> nid=0x554e runnable [0x000000004500c000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aaac2d0d348> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaac2d0d330> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aacbb364b18> (a
sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aaac2750e38> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aaac1028128> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aaac1028128> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aaac1028128> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aaae41f13d8> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aaae41f13d8> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>>
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aab3c08c348> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aab52d54ef8> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aac0cb261b0> (a [B)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>> ner.java:87)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :82)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :262)
>> - locked<0x00002aaabefb9c10> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aad30d56688> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aad30d56688> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
Source)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 16 on 60020" daemon prio=10
tid=0x00000000013fa000
>> nid=0x554d runnable [0x0000000044f0b000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aabec61aa08> (a sun.nio.ch.Util$1)
>> - locked<0x00002aabec61a9f0> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaae7590720> (a
sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aacf2246cb0> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aad06dd2c70> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aad06dd2c70> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aad06dd2c70> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aabe38abcd8> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aabe38abcd8> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>>
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aaad08ca208> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aac189636e0> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aab84fe43c8> (a [B)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(St
>> oreFileScanner.java:158)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
>> anner.java:110)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
>> va:255)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
>> va:394)
>> - locked<0x00002aad072dbd30> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :293)
>> - locked<0x00002aad072dbd30> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aabdbaa0490> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aabdbaa0490> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
Source)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 15 on 60020" daemon prio=10
tid=0x00000000013f7800
>> nid=0x554c runnable [0x0000000044e0a000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aac558cde50> (a sun.nio.ch.Util$1)
>> - locked<0x00002aac558cde38> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aac5cd41558> (a
sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aadcf530468> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aad427a97a8> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aad427a97a8> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aad427a97a8> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aab9410bc00> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aab9410bc00> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>>
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aaca7cfcf70> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aacf952a6c8> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aaac4472350> (a [B)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>> ner.java:87)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :82)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :297)
>> - locked<0x00002aaaefe12070> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aabbaf1d1a8> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aabbaf1d1a8> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
Source)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 14 on 60020" daemon prio=10
tid=0x00000000013f5800
>> nid=0x554b runnable [0x0000000044d09000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aab28f91848> (a sun.nio.ch.Util$1)
>> - locked<0x00002aab28f91860> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaaf5031970> (a
sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aacec8cf590> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aaaf789b048> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aaaf789b048> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aaaf789b048> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aab9c1df470> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aab9c1df470> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>>
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aab7ec60418> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aab44c84dd8> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aab227a2dc8> (a [B)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(St
>> oreFileScanner.java:158)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
>> anner.java:110)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
>> va:255)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
>> va:394)
>> - locked<0x00002aab50d25f28> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :293)
>> - locked<0x00002aab50d25f28> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aaae9efd9f0> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aaae9efd9f0> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
Source)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 13 on 60020" daemon prio=10
tid=0x0000000001b61800
>> nid=0x554a waiting for monitor entry [0x0000000044c09000]
>> java.lang.Thread.State: BLOCKED (on object monitor)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - waiting to lock<0x00002aacc684dad0> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
Source)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 12 on 60020" daemon prio=10
tid=0x0000000001b5f800
>> nid=0x5549 runnable [0x0000000044b07000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aab618bb060> (a sun.nio.ch.Util$1)
>> - locked<0x00002aabe1d82718> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aab76896238> (a
sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aab016dc128> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aad03182340> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aad03182340> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aad03182340> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aab8b32e3e0> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aab8b32e3e0> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>>
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aab8b436170> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aacd4fb8038> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aab8b4b4908> (a [B)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>> ner.java:87)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :82)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :262)
>> - locked<0x00002aab480f55b8> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aaae9efdb30> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aaae9efdb30> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
Source)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 11 on 60020" daemon prio=10
tid=0x0000000001b5d000
>> nid=0x5548 waiting for monitor entry [0x0000000044a07000]
>> java.lang.Thread.State: BLOCKED (on object monitor)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - waiting to lock<0x00002aacc684dad0> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
Source)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 10 on 60020" daemon prio=10
tid=0x000000000196a800
>> nid=0x5547 runnable [0x0000000044905000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aad7443dcd8> (a sun.nio.ch.Util$1)
>> - locked<0x00002aad7a31cb00> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aac9067d2d8> (a
sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aad5103b0a0> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aadc79059c8> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aadc79059c8> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aadc79059c8> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aabf0f472b0> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aabf0f472b0> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>>
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aab8b2b29b0> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aad0477a310> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aabb7b16fd0> (a [B)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>> ner.java:87)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :82)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :262)
>> - locked<0x00002aab59165e10> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aac8e37dd58> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aac8e37dd58> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
Source)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 9 on 60020" daemon prio=10 tid=0x0000000001968000
>> nid=0x5546 waiting for monitor entry [0x0000000044805000]
>> java.lang.Thread.State: BLOCKED (on object monitor)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.isFilterDone(
>> HRegion.java:2324)
>> - waiting to lock<0x00002aacc684dad0> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1850)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
Source)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 8 on 60020" daemon prio=10 tid=0x0000000001967000
>> nid=0x5545 runnable [0x0000000044703000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aabb0b7b868> (a sun.nio.ch.Util$1)
>> - locked<0x00002aabb0b7b880> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaac79442c8> (a
sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aad0106a6b0> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aac108e6768> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aac108e6768> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aac108e6768> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aac4d4ecd40> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aac4d4ecd40> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>>
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aaad20c3b08> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aac536bda88> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aaadfc86320> (a [B)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.loadBlock(HFile.ja
>> va:1447)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.reseekTo(HFile.jav
>> a:1325)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(St
>> oreFileScanner.java:152)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
>> anner.java:110)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
>> va:255)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
>> va:394)
>> - locked<0x00002aad072dc1c8> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :293)
>> - locked<0x00002aad072dc1c8> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aac8e37d808> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aac8e37d808> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
Source)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 7 on 60020" daemon prio=10 tid=0x00000000016de800
>> nid=0x5544 runnable [0x0000000044602000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aab9f4dc588> (a sun.nio.ch.Util$1)
>> - locked<0x00002aab9f4dc570> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aac6ed3e250> (a
sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aad003a0178> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aabbd698ec8> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aabbd698ec8> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aabbd698ec8> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aaad91ecfa8> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aaad91ecfa8> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>>
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aac237c5090> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aad65d9a7f0> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aaaccc5d6c8> (a [B)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>> ner.java:87)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :82)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :262)
>> - locked<0x00002aad15b7d2b8> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aacc684dad0> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aacc684dad0> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
Source)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 6 on 60020" daemon prio=10 tid=0x00000000016dd000
>> nid=0x5543 runnable [0x0000000044501000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aaaf7db82f0> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaaf7db82d8> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aab6f3339a8> (a
sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aab494cdcb8> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aad890f5018> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aad890f5018> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aad890f5018> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aaae4d56928> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aaae4d56928> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>>
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aaae8d98798> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aaaaf686f40> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aaae77f0ad8> (a [B)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>> ner.java:87)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :82)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :262)
>> - locked<0x00002aaae4cd85a0> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aad30d55d28> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aad30d55d28> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
Source)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 5 on 60020" daemon prio=10 tid=0x000000000194c800
>> nid=0x5542 runnable [0x0000000044400000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aaac5bf0ff0> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaac5bf10c8> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aab0492d6d8> (a
sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aac451e5d20> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 89)
>> - locked<0x00002aabf9312408> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aabf9312408> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aabf9312408> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aab78dc6068> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aab78dc6068> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>>
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aab14390138> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aab7fbb83e8> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aac2cbf46c8> (a [B)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
>> anner.java:115)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
>> va:255)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
>> va:394)
>> - locked<0x00002aad15b7da98> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :293)
>> - locked<0x00002aad15b7da98> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aabc7a3d3e8> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aabc7a3d3e8> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
Source)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 4 on 60020" daemon prio=10 tid=0x000000000194a800
>> nid=0x5541 runnable [0x00000000442ff000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aabbae85af8> (a sun.nio.ch.Util$1)
>> - locked<0x00002aabbae85b10> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaae379da00> (a
sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aac7ecf9bb8> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aaacf49d528> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aaacf49d528> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aaacf49d528> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aabdc652620> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aabdc652620> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>>
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aac357f4350> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aab9fc22a58> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aabdbefbe20> (a [B)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>> ner.java:87)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :82)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :262)
>> - locked<0x00002aad97b3b7e8> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aab016d6618> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aab016d6618> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
Source)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 3 on 60020" daemon prio=10 tid=0x00000000019c4000
>> nid=0x5540 waiting for monitor entry [0x00000000441ff000]
>> java.lang.Thread.State: BLOCKED (on object monitor)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - waiting to lock<0x00002aab28d7de90> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
Source)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 2 on 60020" daemon prio=10 tid=0x00000000019c2000
>> nid=0x553f runnable [0x00000000440fd000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aaaeef929f8> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaaeef92a10> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaaffd9ef38> (a
sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aab2ac42300> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aabc43bef18> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aabc43bef18> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aabc43bef18> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aabd53c1d58> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aabd53c1d58> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>>
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aabb602a180> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aab44c85f18> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aac0d2be0b0> (a [B)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>> ner.java:87)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :82)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :262)
>> - locked<0x00002aab4eb7a680> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aabc7a3cc68> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aabc7a3cc68> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
Source)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 1 on 60020" daemon prio=10 tid=0x0000000001535800
>> nid=0x553e runnable [0x0000000043ffc000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aad28449198> (a sun.nio.ch.Util$1)
>> - locked<0x00002aad284491b0> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aab3a682700> (a
sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aac5b5cbd28> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aabc43bd778> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aabc43bd778> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aabc43bd778> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aac4d4f9b90> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aac4d4f9b90> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>>
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aac500ac780> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aabe2211268> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aac537126f0> (a [B)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>> ner.java:87)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :82)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :262)
>> - locked<0x00002aadb912fca8> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aadcba8fe98> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aadcba8fe98> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
Source)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 0 on 60020" daemon prio=10 tid=0x0000000001534000
>> nid=0x553d waiting for monitor entry [0x0000000043efc000]
>> java.lang.Thread.State: BLOCKED (on object monitor)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - waiting to lock<0x00002aabc7a3d258> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
Source)
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server listener on 60020" daemon prio=10 tid=0x00002aae4054e000
>> nid=0x553c runnable [0x0000000043dfb000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aaabc18bdf8> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaabc18bde0> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaabbb1ced8> (a
sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:41
>> 4)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server Responder" daemon prio=10 tid=0x00002aae4053d800
nid=0x553b
>> runnable [0x0000000043cfa000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aaabc2ceed8> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaabc2ceef0> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaabbb33fa8> (a
sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:5
>> 88)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "Timer-0" daemon prio=10 tid=0x00002aae4053c800 nid=0x553a in
>> Object.wait() [0x0000000043bf9000]
>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at java.util.TimerThread.mainLoop(Timer.java:531)
>> - locked<0x00002aaabd062850> (a java.util.TaskQueue)
>> at java.util.TimerThread.run(Timer.java:484)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "983053578@qtp-1891789590-1 - Acceptor0
>> SelectChannelConnector@0.0.0.0:60030" prio=10 tid=0x00002aae40381000
>> nid=0x5539 runnable [0x0000000043af8000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aaabd07c460> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaabd07c478> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaabbb26b78> (a
sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>>
org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.ja
>> va:498)
>> at
>> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
>> at
>>
org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnect
>> or.java:124)
>> at
>>
org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:
>> 708)
>> at
>>
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java
>> :582)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "1148428095@qtp-1891789590-0" prio=10 tid=0x00002aae403d2000
nid=0x5538
>> in Object.wait() [0x00000000439f7000]
>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at
>>
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java
>> :626)
>> - locked<0x00002aaabbc4c8f0> (a
>> org.mortbay.thread.QueuedThreadPool$PoolThread)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "regionserver60020.leaseChecker" daemon prio=10
tid=0x00002aae402ee800
>> nid=0x5537 waiting for monitor entry [0x00000000438f6000]
>> java.lang.Thread.State: BLOCKED (on object monitor)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.close(HRegion
>> .java:2409)
>> - waiting to lock<0x00002aadcba8fe98> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegionServer$ScannerListener.lease
>> Expired(HRegionServer.java:1892)
>> at
>> org.apache.hadoop.hbase.regionserver.Leases.run(Leases.java:99)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "regionserver60020.majorCompactionChecker" daemon prio=10
>> tid=0x00002aae402ec800 nid=0x5536 in Object.wait()
[0x00000000437f5000]
>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> - waiting on<0x00002aaabc1f7bf0> (a java.lang.Object)
>> at
org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
>> - locked<0x00002aaabc1f7bf0> (a java.lang.Object)
>> at org.apache.hadoop.hbase.Chore.run(Chore.java:74)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "regionserver60020.compactor" daemon prio=10 tid=0x00002aae402eb000
>> nid=0x5535 runnable [0x00000000436f4000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aab26f462e0> (a sun.nio.ch.Util$1)
>> - locked<0x00002aab26f462c8> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaaeba1ac80> (a
sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>>
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>>
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aab44c85258> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aaac6caf140> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aaac6caf140> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aaac6caf140> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aaabd63fbb0> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aaabd63fbb0> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>>
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aab21b4e148> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aaaaedc39d0> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aab458cc280> (a [B)
>> at
>>
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>> ner.java:87)
>> at
>>
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :82)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :297)
>> - locked<0x00002aaace390ec0> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>>
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :326)
>> - locked<0x00002aaace390ec0> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:927)
>> at
>> org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:733)
>> - locked<0x00002aaacd20e840> (a java.lang.Object)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:
>> 769)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:
>> 714)
>> at
>>
org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplit
>> Thread.java:81)
>>
>> Locked ownable synchronizers:
>> -<0x00002aaabce779f8> (a
>> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>>
>> "regionserver60020.cacheFlusher" daemon prio=10
tid=0x00002aae402d1800
>> nid=0x5534 waiting on condition [0x00000000435f3000]
>> java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabc241a10> (a
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>>
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>> at
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> aitNanos(AbstractQueuedSynchronizer.java:2081)
>> at java.util.concurrent.DelayQueue.poll(DelayQueue.java:230)
>> at java.util.concurrent.DelayQueue.poll(DelayQueue.java:68)
>> at
>>
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher
>> .java:216)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "regionserver60020.logRoller" daemon prio=10 tid=0x00002aae402d1000
>> nid=0x5533 in Object.wait() [0x00000000434f2000]
>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at
>> org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:78)
>> - locked<0x00002aaabcec18e0> (a
>> java.util.concurrent.atomic.AtomicBoolean)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "Timer thread for monitoring jvm" daemon prio=10
tid=0x00002aae402b4000
>> nid=0x5532 in Object.wait() [0x00000000433f1000]
>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at java.util.TimerThread.mainLoop(Timer.java:531)
>> - locked<0x00002aaabcedce48> (a java.util.TaskQueue)
>> at java.util.TimerThread.run(Timer.java:484)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "Timer thread for monitoring hbase" daemon prio=10
>> tid=0x00002aae402b2800 nid=0x5531 in Object.wait()
[0x00000000432f0000]
>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at java.util.TimerThread.mainLoop(Timer.java:531)
>> - locked<0x00002aaabceca148> (a java.util.TaskQueue)
>> at java.util.TimerThread.run(Timer.java:484)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "regionserver60020.logSyncer" daemon prio=10 tid=0x00002aae402b1000
>> nid=0x5530 waiting on condition [0x00000000431ef000]
>> java.lang.Thread.State: TIMED_WAITING (sleeping)
>> at java.lang.Thread.sleep(Native Method)
>> at
>>
org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:96
>> 3)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "LeaseChecker" daemon prio=10 tid=0x00002aae4019a800 nid=0x552f
>> sleeping[0x00000000430ee000]
>> java.lang.Thread.State: TIMED_WAITING (sleeping)
>> at java.lang.Thread.sleep(Native Method)
>> at
>>
org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1167)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Client (47) connection to
>> doop10.dt.sv4.decarta.com/10.241.8.230:60000 from an unknown user"
>> daemon prio=10 tid=0x00000000015c9800 nid=0x5528 in Object.wait()
>> [0x0000000042cea000]
>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.waitForWork(HBaseClie
>> nt.java:431)
>> - locked<0x00002aaabc24dfc0> (a
>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:
>> 476)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "DestroyJavaVM" prio=10 tid=0x00002aae4015c800 nid=0x54fa waiting on
>> condition [0x0000000000000000]
>> java.lang.Thread.State: RUNNABLE
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "regionserver60020-EventThread" daemon prio=10 tid=0x00000000015c5000
>> nid=0x5526 waiting on condition [0x0000000042be9000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabc241a60> (a
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> at
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> ait(AbstractQueuedSynchronizer.java:2043)
>> at
>>
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>> 86)
>> at
>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "regionserver60020-SendThread(doop3.dt.sv4.decarta.com:2181)" daemon
>> prio=10 tid=0x0000000001486800 nid=0x5525 runnable
[0x0000000042ae8000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aaabbb3e538> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaabbb3e550> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaabbb1db08> (a
sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "regionserver60020" prio=10 tid=0x00002aae40187000 nid=0x5523 waiting
on
>> condition [0x000000004028d000]
>> java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabc2faea0> (a
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>>
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>> at
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> aitNanos(AbstractQueuedSynchronizer.java:2081)
>> at
>>
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:4
>> 23)
>> at
>>
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.jav
>> a:621)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "Timer thread for monitoring rpc" daemon prio=10
tid=0x00002aae40135800
>> nid=0x5522 in Object.wait() [0x00000000429e7000]
>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at java.util.TimerThread.mainLoop(Timer.java:531)
>> - locked<0x00002aaabbb3cb08> (a java.util.TaskQueue)
>> at java.util.TimerThread.run(Timer.java:484)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Reader 9 on port 60020" prio=10 tid=0x00002aae40133000
nid=0x5521
>> runnable [0x00000000428e6000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aaabbb010a8> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaabbb010c0> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaabbb1e198> (a
sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
>> java:305)
>> - locked<0x00002aaabbb250e8> (a
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>> at
>>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>> a:1110)
>> at
>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>> va:603)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Locked ownable synchronizers:
>> -<0x00002aaabbb4b008> (a
>> java.util.concurrent.ThreadPoolExecutor$Worker)
>>
>> "IPC Reader 8 on port 60020" prio=10 tid=0x00002aae40118000
nid=0x5520
>> runnable [0x00000000427e5000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aaabbb16118> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaabbb16130> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaabbb144f8> (a
sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
>> java:305)
>> - locked<0x00002aaabbb30308> (a
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>> at
>>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>> a:1110)
>> at
>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>> va:603)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Locked ownable synchronizers:
>> -<0x00002aaabbb32888> (a
>> java.util.concurrent.ThreadPoolExecutor$Worker)
>>
>> "IPC Reader 7 on port 60020" prio=10 tid=0x00002aae400fd000
nid=0x551f
>> runnable [0x00000000426e4000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aaabbb2a228> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaabbb2a240> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaabbb28ab8> (a
sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
>> java:305)
>> - locked<0x00002aaabbb2a6d8> (a
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>> at
>>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>> a:1110)
>> at
>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>> va:603)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Locked ownable synchronizers:
>> -<0x00002aaabbb4a068> (a
>> java.util.concurrent.ThreadPoolExecutor$Worker)
>>
>> "IPC Reader 6 on port 60020" prio=10 tid=0x00002aae400e2800
nid=0x551e
>> runnable [0x00000000425e3000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aaabbb27028> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaabbb27040> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaabbb258b8> (a
sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
>> java:305)
>> - locked<0x00002aaabbb282e8> (a
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>> at
>>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>> a:1110)
>> at
>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>> va:603)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Locked ownable synchronizers:
>> -<0x00002aaabbb4fc98> (a
>> java.util.concurrent.ThreadPoolExecutor$Worker)
>>
>> "IPC Reader 5 on port 60020" prio=10 tid=0x00002aae400c7800
nid=0x551d
>> runnable [0x00000000413e5000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aaabbb2ea08> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaabbb2ea20> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaabbb2d298> (a
sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
>> java:305)
>> - locked<0x00002aaabbb50c38> (a
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>> at
>>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>> a:1110)
>> at
>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>> va:603)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Locked ownable synchronizers:
>> -<0x00002aaabbb523a8> (a
>> java.util.concurrent.ThreadPoolExecutor$Worker)
>>
>> "IPC Reader 4 on port 60020" prio=10 tid=0x00002aae400ac800
nid=0x551c
>> runnable [0x00000000412e4000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aaabbb2aea8> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaabbb2aec0> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaabbb1ca28> (a
sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
>> java:305)
>> - locked<0x00002aaabbb2c168> (a
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>> at
>>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>> a:1110)
>> at
>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>> va:603)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Locked ownable synchronizers:
>> -<0x00002aaabbb48128> (a
>> java.util.concurrent.ThreadPoolExecutor$Worker)
>>
>> "IPC Reader 3 on port 60020" prio=10 tid=0x00002aae40091800
nid=0x551b
>> runnable [0x00000000411e3000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aaabbb15c68> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaabbb15c80> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaabbb12d88> (a
sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
>> java:305)
>> - locked<0x00002aaabbb165c8> (a
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>> at
>>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>> a:1110)
>> at
>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>> va:603)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Locked ownable synchronizers:
>> -<0x00002aaabbb30ad8> (a
>> java.util.concurrent.ThreadPoolExecutor$Worker)
>>
>> "IPC Reader 2 on port 60020" prio=10 tid=0x00002aae40077000
nid=0x551a
>> runnable [0x00000000410e2000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aaabbb3d148> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaabbb3d160> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaabbb33828> (a
sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
>> java:305)
>> - locked<0x00002aaabbb3ebd8> (a
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>> at
>>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>> a:1110)
>> at
>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>> va:603)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Locked ownable synchronizers:
>> -<0x00002aaabbb40028> (a
>> java.util.concurrent.ThreadPoolExecutor$Worker)
>>
>> "IPC Reader 1 on port 60020" prio=10 tid=0x00002aae4005d000
nid=0x5519
>> runnable [0x0000000040fe1000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aaabbb15c98> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaabbb15cb0> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaabbb12e00> (a
sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
>> java:305)
>> - locked<0x00002aaabbb165f0> (a
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>> at
>>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>> a:1110)
>> at
>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>> va:603)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Locked ownable synchronizers:
>> -<0x00002aaabbb30b28> (a
>> java.util.concurrent.ThreadPoolExecutor$Worker)
>>
>> "IPC Reader 0 on port 60020" prio=10 tid=0x00002aae40003000
nid=0x5518
>> runnable [0x0000000040ee0000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aaabbb16148> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaabbb16160> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaabbb14570> (a
sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> at
>>
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
>> java:305)
>> - locked<0x00002aaabbb30330> (a
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>> at
>>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>> a:1110)
>> at
>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>> va:603)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Locked ownable synchronizers:
>> -<0x00002aaabbb328d8> (a
>> java.util.concurrent.ThreadPoolExecutor$Worker)
>>
>> "main-EventThread" daemon prio=10 tid=0x00002aae3c1b6000 nid=0x5513
>> waiting on condition [0x00000000424e2000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabbb16618> (a
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> at
>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> ait(AbstractQueuedSynchronizer.java:2043)
>> at
>>
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>> 86)
>> at
>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "main-SendThread(doop7.dt.sv4.decarta.com:2181)" daemon prio=10
>> tid=0x00002aae3c11b000 nid=0x5512 runnable [0x00000000423e1000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aaabbb16178> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaabbb16190> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaabbb145e8> (a
sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "Low Memory Detector" daemon prio=10 tid=0x000000000126c800
nid=0x5510
>> runnable [0x0000000000000000]
>> java.lang.Thread.State: RUNNABLE
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "CompilerThread1" daemon prio=10 tid=0x000000000126a800 nid=0x550f
>> waiting on condition [0x0000000000000000]
>> java.lang.Thread.State: RUNNABLE
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "CompilerThread0" daemon prio=10 tid=0x0000000001266000 nid=0x550e
>> waiting on condition [0x0000000000000000]
>> java.lang.Thread.State: RUNNABLE
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "Signal Dispatcher" daemon prio=10 tid=0x0000000001264000 nid=0x550d
>> runnable [0x0000000000000000]
>> java.lang.Thread.State: RUNNABLE
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "Surrogate Locker Thread (CMS)" daemon prio=10 tid=0x0000000001262000
>> nid=0x550c waiting on condition [0x0000000000000000]
>> java.lang.Thread.State: RUNNABLE
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "Finalizer" daemon prio=10 tid=0x000000000123b800 nid=0x550b in
>> Object.wait() [0x00000000422e0000]
>> java.lang.Thread.State: WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:133)
>> - locked<0x00002aaabbb15cf8> (a
>> java.lang.ref.ReferenceQueue$Lock)
>> at
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:149)
>> at
>> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "Reference Handler" daemon prio=10 tid=0x0000000001239800 nid=0x550a
in
>> Object.wait() [0x00000000421df000]
>> java.lang.Thread.State: WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at java.lang.Object.wait(Object.java:502)
>> at
>> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
>> - locked<0x00002aaabbb16d98> (a
java.lang.ref.Reference$Lock)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "VM Thread" prio=10 tid=0x0000000001234800 nid=0x5509 runnable
>>
>> "Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x0000000000fe9000
>> nid=0x54fb runnable
>>
>> "Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x0000000000feb000
>> nid=0x54fc runnable
>>
>> "Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x0000000000fec800
>> nid=0x54fd runnable
>>
>> "Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x0000000000fee800
>> nid=0x54fe runnable
>>
>> "Gang worker#4 (Parallel GC Threads)" prio=10 tid=0x0000000000ff0800
>> nid=0x54ff runnable
>>
>> "Gang worker#5 (Parallel GC Threads)" prio=10 tid=0x0000000000ff2000
>> nid=0x5500 runnable
>>
>> "Gang worker#6 (Parallel GC Threads)" prio=10 tid=0x0000000000ff4000
>> nid=0x5501 runnable
>>
>> "Gang worker#7 (Parallel GC Threads)" prio=10 tid=0x0000000000ff6000
>> nid=0x5502 runnable
>>
>> "Gang worker#8 (Parallel GC Threads)" prio=10 tid=0x0000000000ff7800
>> nid=0x5503 runnable
>>
>> "Gang worker#9 (Parallel GC Threads)" prio=10 tid=0x0000000000ff9800
>> nid=0x5504 runnable
>>
>> "Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00000000010f4000
>> nid=0x5508 runnable
>> "Gang worker#0 (Parallel CMS Threads)" prio=10 tid=0x00000000010ee000
>> nid=0x5505 runnable
>>
>> "Gang worker#1 (Parallel CMS Threads)" prio=10 tid=0x00000000010f0000
>> nid=0x5506 runnable
>>
>> "Gang worker#2 (Parallel CMS Threads)" prio=10 tid=0x00000000010f2000
>> nid=0x5507 runnable
>>
>> "VM Periodic Task Thread" prio=10 tid=0x000000000126f800 nid=0x5511
>> waiting on condition
>>
>> JNI global references: 906
>>
>>
>> -----Original Message-----
>> From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of
>> Stack
>> Sent: Sunday, September 11, 2011 2:27 PM
>> To: user@hbase.apache.org
>> Subject: Re: scanner deadlock?
>>
>> 0.90.4 fixes two deadlocks (HBASE-4101 and HBASE-4077). Since then,
>> there is HBASE-4367 (Which has a posted patch).
>>
>> Below sounds like slowness. Can you thread dump the particular
>> regionserver and see what its up to? Is there other loading on the
>> system at the time? For example, loading on hdfs? Anything in the
>> hdfs logs for the datanode running beside the slow regionserver?
>>
>> St.Ack
>>
>> On Sat, Sep 10, 2011 at 5:50 PM, Geoff Hendrey<gh...@decarta.com>
>> wrote:
>>> Hi all -
>>>
>>>
>>>
>>> I'm still dealing with the saga of ScannerTimeoutException,
>>> UnknownScannerException, etc. I rewrote my code, in the hope that
>> simply
>>> a different approach and some different code paths might yield
better
>>> results. No change. I tried many variations (caching 1 row vs
caching
>>> many rows, changing the regionserver's lease, increasing the number
of
>>> allowed zookeeper connections, etc). I created a fresh table on the
>>> thought that maybe there was some problem with the table...no
change.
>>>
>>>
>>>
>>> I am dealing with what appears to be some sort of scanner deadlock.
I
>>> have a total order partitioned mapreduce job. In the reducer, as
long
>> as
>>> I use just one reducer, the task finishes quickly. But as soon as
more
>>> than one reducer opens a scanner, the tasks procede in what I would
>> call
>>> a "jittery" lock step. They both are able to do ResultScanner.next()
a
>>> few times, but then the call to next() freezes for a long period,
and
>>> ScannerTimeoutException. I catch the exception, and get a new
>>> ResultScanner, and the pattern repeats. Jittery lockstep consisting
of
>>> being able to get a few successful next() calls, then a lock up.
>>>
>>>
>>>
>>> I have not yet tried upgrading from 90.1 to higher. Nor have I tried
>>> tsuna's async client. Can anyone think of anything else I can try to
>>> resolve this? I've sunk quite a few late nights into this, and would
>> be
>>> very excited to find a solution.
>>>
>>>
>>>
>>> -geoff
>>>
>>>
>>
--
Eric
http://about.echarles.net
Re: scanner deadlock?
Posted by Eric Charles <er...@gmail.com>.
Hi,
Probably you've already done something similar, but I happily use 'IBM
Thread and Monitor Dump Analyzer for Java' to be 100% sure there are no
dead lock. The tool gives you a nice short report on your threads.
I tried to run it on your dump, but the analyzer does not like
mail-wrapped lines.
Thx.
[1]
https://www.ibm.com/developerworks/mydeveloperworks/groups/service/html/communityview?communityUuid=2245aa39-fa5c-4475-b891-14c205f7333c
On 12/09/11 09:46, Stack wrote:
> On Sun, Sep 11, 2011 at 2:40 PM, Geoff Hendrey<gh...@decarta.com> wrote:
>> Hi,
>>
>> The cluster is definitely loaded. We are running a MR job with 130
>> reducers, each using a scanner. Ganglia shows an overall CPU utilization
>> of around 40%, with network pushing 800MB/s. The problem may correlate
>> with a major_compact I initiated couple days ago. Even today, I still
>> see a lot of compacting (continuous) on region servers.
>>
>> Here is the thread dump from jstack. It does not reveal any explicit
>> deadlock. At the time I dumped the jstack, the region server logs are
>> reporting "IPC Server handler...ClosedChannelException". As soon as this
>> begins occurring, our network throughput, as seen on Ganglia drops from
>> 800MB/s down to virtually zero. Then of course there is the cascade of
>> failures, eventually leading to the regionservers shutting down.
>>
>
>
> Looking at the thread dump, as you say, few are 'BLOCKED' (There are a
> few BLOCKED thread trying to go into a next). But most are deep in a
> next stuck polling looking to read. If you thread dump a few times
> in a row, do any seem to be making progress? (They may be making
> progress but slowly, as you say)
>
> So, is it possible you have a slow datanode in your cluster? Can you
> tell who all are connected too... lsof'ing? That might give you a
> clue. Perhaps there is a node with sick disks (check dmesg). Or
> perhaps the machine's nework card is sick?
>
> St.Ack
>
>> I don't see anything unusual in the datanode logs on the same server.
>> Just HDFS_READ and HDFS_WRITE for the most part.
>>
>> jstack here:
>> ===================================================================
>> 2011-09-11 14:28:52
>> Full thread dump OpenJDK 64-Bit Server VM (14.0-b16 mixed mode):
>>
>> "IPC Client (47) connection to
>> doop10.dt.sv4.decarta.com/10.241.8.230:54310 from hroot" daemon prio=10
>> tid=0x0000000001492800 nid=0x75cb in Object.wait() [0x0000000047633000]
>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at
>> org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:676)
>> - locked<0x00002aaaaf680000> (a
>> org.apache.hadoop.ipc.Client$Connection)
>> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:719)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "ResponseProcessor for block blk_2613854253703670198_5032689" daemon
>> prio=10 tid=0x0000000001988800 nid=0x75bb runnable [0x0000000046926000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aab69337248> (a sun.nio.ch.Util$1)
>> - locked<0x00002aab69337230> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaaf3897930> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at java.io.DataInputStream.readFully(DataInputStream.java:195)
>> at java.io.DataInputStream.readLong(DataInputStream.java:416)
>> at
>> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFie
>> lds(DataTransferProtocol.java:120)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(D
>> FSClient.java:2638)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "DataStreamer for file
>> /hbase-20.6/NAM_CLUSTERKEYS3/60db9e0113c7578ed913dc37e0856954/.tmp/42977
>> 03314038441248 block blk_2613854253703670198_5032689" daemon prio=10
>> tid=0x00002aae40e8e800 nid=0x75b8 in Object.wait() [0x0000000046e2b000]
>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSCli
>> ent.java:2483)
>> - locked<0x00002aab6551a2d8> (a java.util.LinkedList)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "ResponseProcessor for block blk_8900384789253382699_5032549" daemon
>> prio=10 tid=0x00000000013cd000 nid=0x71d0 runnable [0x0000000047431000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aaae0569530> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaae0569518> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaadb4c5c68> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at java.io.DataInputStream.readFully(DataInputStream.java:195)
>> at java.io.DataInputStream.readLong(DataInputStream.java:416)
>> at
>> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFie
>> lds(DataTransferProtocol.java:120)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(D
>> FSClient.java:2638)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "DataStreamer for file
>> /hbase-20.6/.logs/doop5.dt.sv4.decarta.com,60020,1315768059645/doop5.dt.
>> sv4.decarta.com%3A60020.1315776307423 block
>> blk_8900384789253382699_5032549" daemon prio=10 tid=0x00000000016c3800
>> nid=0x71c8 in Object.wait() [0x000000004712e000]
>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSCli
>> ent.java:2483)
>> - locked<0x00002aab092a7f38> (a java.util.LinkedList)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "Attach Listener" daemon prio=10 tid=0x0000000001e32000 nid=0x50f4
>> waiting on condition [0x0000000000000000]
>> java.lang.Thread.State: RUNNABLE
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "RS_CLOSE_REGION-doop5.dt.sv4.decarta.com,60020,1315768059645-2" daemon
>> prio=10 tid=0x00000000018cb800 nid=0x505 waiting on condition
>> [0x0000000047330000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabcefb428> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> ait(AbstractQueuedSynchronizer.java:2043)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>> 86)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
>> 1043)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>> a:1103)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>> va:603)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "RS_CLOSE_REGION-doop5.dt.sv4.decarta.com,60020,1315768059645-1" daemon
>> prio=10 tid=0x000000000163e000 nid=0x504 waiting on condition
>> [0x0000000047532000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabcefb428> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> ait(AbstractQueuedSynchronizer.java:2043)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>> 86)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
>> 1043)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>> a:1103)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>> va:603)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "RS_CLOSE_REGION-doop5.dt.sv4.decarta.com,60020,1315768059645-0" daemon
>> prio=10 tid=0x0000000001bff800 nid=0x503 waiting on condition
>> [0x0000000046f2c000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabcefb428> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> ait(AbstractQueuedSynchronizer.java:2043)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>> 86)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
>> 1043)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>> a:1103)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>> va:603)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "sendParams-6" daemon prio=10 tid=0x00002aae401f1800 nid=0x6d3b waiting
>> on condition [0x000000004702d000]
>> java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabceea260> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>> at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Synchro
>> nousQueue.java:453)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(Synchronous
>> Queue.java:352)
>> at
>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:903)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
>> 1043)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>> a:1103)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>> va:603)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "RS_OPEN_REGION-doop5.dt.sv4.decarta.com,60020,1315768059645-2" daemon
>> prio=10 tid=0x0000000001a7f000 nid=0x5b56 waiting on condition
>> [0x0000000046d2a000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabcefb2e8> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> ait(AbstractQueuedSynchronizer.java:2043)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>> 86)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
>> 1043)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>> a:1103)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>> va:603)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "RS_OPEN_REGION-doop5.dt.sv4.decarta.com,60020,1315768059645-1" daemon
>> prio=10 tid=0x0000000001af2800 nid=0x5b55 waiting on condition
>> [0x0000000042deb000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabcefb2e8> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> ait(AbstractQueuedSynchronizer.java:2043)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>> 86)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
>> 1043)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>> a:1103)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>> va:603)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "RS_OPEN_REGION-doop5.dt.sv4.decarta.com,60020,1315768059645-0" daemon
>> prio=10 tid=0x00000000014b3800 nid=0x5b54 waiting on condition
>> [0x0000000046c29000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabcefb2e8> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> ait(AbstractQueuedSynchronizer.java:2043)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>> 86)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
>> 1043)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>> a:1103)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>> va:603)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "RS_OPEN_ROOT-doop5.dt.sv4.decarta.com,60020,1315768059645-0" daemon
>> prio=10 tid=0x0000000001ad2800 nid=0x556e waiting on condition
>> [0x0000000046a27000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabcee8870> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> ait(AbstractQueuedSynchronizer.java:2043)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>> 86)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
>> 1043)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>> a:1103)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>> va:603)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "LRU Statistics #0" prio=10 tid=0x00002aae4048a800 nid=0x5566 waiting on
>> condition [0x0000000046825000]
>> java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabd0695c0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> aitNanos(AbstractQueuedSynchronizer.java:2081)
>> at java.util.concurrent.DelayQueue.take(DelayQueue.java:193)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(S
>> cheduledThreadPoolExecutor.java:688)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(S
>> cheduledThreadPoolExecutor.java:681)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
>> 1043)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>> a:1103)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>> va:603)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "LruBlockCache.EvictionThread" daemon prio=10 tid=0x00002aae4049f800
>> nid=0x5565 in Object.wait() [0x0000000046724000]
>> java.lang.Thread.State: WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at java.lang.Object.wait(Object.java:502)
>> at
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlo
>> ckCache.java:519)
>> - locked<0x00002aaabce92f50> (a
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "PRI IPC Server handler 9 on 60020" daemon prio=10
>> tid=0x00002aae4057e000 nid=0x5564 waiting on condition
>> [0x0000000046623000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabbb307e0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> ait(AbstractQueuedSynchronizer.java:2043)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>> 86)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
>> 5)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "PRI IPC Server handler 8 on 60020" daemon prio=10
>> tid=0x00002aae4057c800 nid=0x5563 waiting on condition
>> [0x0000000046522000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabbb307e0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> ait(AbstractQueuedSynchronizer.java:2043)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>> 86)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
>> 5)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "PRI IPC Server handler 7 on 60020" daemon prio=10
>> tid=0x00002aae40559000 nid=0x5562 waiting on condition
>> [0x0000000046421000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabbb307e0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> ait(AbstractQueuedSynchronizer.java:2043)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>> 86)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
>> 5)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "PRI IPC Server handler 6 on 60020" daemon prio=10
>> tid=0x00002aae40557800 nid=0x5561 waiting on condition
>> [0x0000000046320000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabbb307e0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> ait(AbstractQueuedSynchronizer.java:2043)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>> 86)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
>> 5)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "PRI IPC Server handler 5 on 60020" daemon prio=10
>> tid=0x00002aae40555800 nid=0x5560 waiting on condition
>> [0x000000004621f000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabbb307e0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> ait(AbstractQueuedSynchronizer.java:2043)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>> 86)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
>> 5)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "PRI IPC Server handler 4 on 60020" daemon prio=10
>> tid=0x00002aae40553800 nid=0x555f waiting on condition
>> [0x000000004611e000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabbb307e0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> ait(AbstractQueuedSynchronizer.java:2043)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>> 86)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
>> 5)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "PRI IPC Server handler 3 on 60020" daemon prio=10
>> tid=0x00002aae4054f000 nid=0x555e waiting on condition
>> [0x000000004601d000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabbb307e0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> ait(AbstractQueuedSynchronizer.java:2043)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>> 86)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
>> 5)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "PRI IPC Server handler 2 on 60020" daemon prio=10
>> tid=0x00000000012f8800 nid=0x555d waiting on condition
>> [0x0000000045f1c000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabbb307e0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> ait(AbstractQueuedSynchronizer.java:2043)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>> 86)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
>> 5)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "PRI IPC Server handler 1 on 60020" daemon prio=10
>> tid=0x0000000001991000 nid=0x555c waiting on condition
>> [0x0000000045e1b000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabbb307e0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> ait(AbstractQueuedSynchronizer.java:2043)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>> 86)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
>> 5)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "PRI IPC Server handler 0 on 60020" daemon prio=10
>> tid=0x000000000198e800 nid=0x555b waiting on condition
>> [0x0000000045d1a000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabbb307e0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> ait(AbstractQueuedSynchronizer.java:2043)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>> 86)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
>> 5)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 29 on 60020" daemon prio=10 tid=0x000000000198c800
>> nid=0x555a runnable [0x0000000045c18000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aac053b8450> (a sun.nio.ch.Util$1)
>> - locked<0x00002aac053b8438> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aab1037afc8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aac92976a30> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aabc43be0a8> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aabc43be0a8> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aabc43be0a8> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aac1a4e0ea8> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aac1a4e0ea8> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aac237c37a8> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aad39485080> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aab0b5ba8a8> (a [B)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>> ner.java:87)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :82)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :262)
>> - locked<0x00002aab89f59aa8> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aab016d6988> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aab016d6988> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 28 on 60020" daemon prio=10 tid=0x0000000001864000
>> nid=0x5559 runnable [0x0000000045b17000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aabaedb0608> (a sun.nio.ch.Util$1)
>> - locked<0x00002aabaedb05f0> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaca096fa50> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aac62a70b28> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aab856e6220> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aab856e6220> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aab856e6220> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aab0264afe8> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aab0264afe8> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aaae070be48> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aac18961940> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aabad8605d0> (a [B)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>> ner.java:87)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :82)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :262)
>> - locked<0x00002aab077e5d80> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aad30d57c18> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aad30d57c18> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 27 on 60020" daemon prio=10 tid=0x0000000001862000
>> nid=0x5558 runnable [0x0000000045a16000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aab2ea61b70> (a sun.nio.ch.Util$1)
>> - locked<0x00002aab2ea61b88> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aacbb364578> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aab1ffc0d00> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aad890f4640> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aad890f4640> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aad890f4640> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aaba1282b10> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aaba1282b10> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aac528d15d0> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aaaaf68a630> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aabc5c948b8> (a [B)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
>> anner.java:115)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
>> va:255)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
>> va:394)
>> - locked<0x00002aab95832718> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :293)
>> - locked<0x00002aab95832718> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aab28d7de90> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aab28d7de90> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 26 on 60020" daemon prio=10 tid=0x0000000001860000
>> nid=0x5557 runnable [0x0000000045915000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aabaaa6b808> (a sun.nio.ch.Util$1)
>> - locked<0x00002aabaaa6b7f0> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaaf3899ca8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aaac1f59118> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aaac0e99db0> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aaac0e99db0> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aaac0e99db0> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aabb7221e88> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aabb7221e88> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aac28621da0> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aac18962690> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aab6876b850> (a [B)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.loadBlock(HFile.ja
>> va:1447)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.reseekTo(HFile.jav
>> a:1325)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(St
>> oreFileScanner.java:152)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
>> anner.java:110)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
>> va:255)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
>> va:394)
>> - locked<0x00002aaabe3468f8> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :293)
>> - locked<0x00002aaabe3468f8> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aac8e37dbc8> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aac8e37dbc8> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 25 on 60020" daemon prio=10 tid=0x000000000173d800
>> nid=0x5556 runnable [0x0000000045814000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aab20178510> (a sun.nio.ch.Util$1)
>> - locked<0x00002aab20178528> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaad6187af0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aac891a6610> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aac1de6d9f8> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aac1de6d9f8> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aac1de6d9f8> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aaac041c6c8> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aaac041c6c8> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aab25e717e0> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aac3e6e4e28> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aaad4a425f0> (a [B)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>> ner.java:87)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :82)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :262)
>> - locked<0x00002aaabd5eb7e0> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aac8e37dc68> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aac8e37dc68> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 24 on 60020" daemon prio=10 tid=0x000000000173a000
>> nid=0x5555 waiting for monitor entry [0x0000000045714000]
>> java.lang.Thread.State: BLOCKED (on object monitor)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - waiting to lock<0x00002aab016d6618> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 23 on 60020" daemon prio=10 tid=0x0000000001444000
>> nid=0x5554 runnable [0x0000000045612000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aad70044d30> (a sun.nio.ch.Util$1)
>> - locked<0x00002aad70044d48> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aab328f1a40> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aaac9d48e68> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aac1de6dd40> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aac1de6dd40> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aac1de6dd40> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aab39d88148> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aab39d88148> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aab7addd0d8> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aab2811e8e8> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aabe56e8858> (a [B)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>> ner.java:87)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :82)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :262)
>> - locked<0x00002aad15b7d360> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aabc7a3d258> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aabc7a3d258> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 22 on 60020" daemon prio=10 tid=0x0000000001442000
>> nid=0x5553 runnable [0x0000000045511000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aabf6e38450> (a sun.nio.ch.Util$1)
>> - locked<0x00002aabf6e38468> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aabb9e99468> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aaae13c0340> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aad427a91c0> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aad427a91c0> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aad427a91c0> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aab7c7ca0c8> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aab7c7ca0c8> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aab3748f1a8> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aab52d56ed8> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aab60a86ca8> (a [B)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(St
>> oreFileScanner.java:158)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
>> anner.java:110)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
>> va:255)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
>> va:394)
>> - locked<0x00002aab706a3e98> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :293)
>> - locked<0x00002aab706a3e98> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aaae3489758> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aaae3489758> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 21 on 60020" daemon prio=10 tid=0x0000000001792000
>> nid=0x5552 waiting for monitor entry [0x0000000045411000]
>> java.lang.Thread.State: BLOCKED (on object monitor)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - waiting to lock<0x00002aabc7a3d3e8> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 20 on 60020" daemon prio=10 tid=0x000000000178e000
>> nid=0x5551 runnable [0x000000004530f000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aab3745e018> (a sun.nio.ch.Util$1)
>> - locked<0x00002aab3745e030> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aab7d5cc0c0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aac035e4990> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aad890f42f8> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aad890f42f8> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aad890f42f8> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aab1baffa50> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aab1baffa50> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aaadef99fe8> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aac189625a0> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aabb88c7ce0> (a [B)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
>> anner.java:115)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
>> va:255)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
>> va:394)
>> - locked<0x00002aaadfa95ec0> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :293)
>> - locked<0x00002aaadfa95ec0> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aabf6832658> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aabf6832658> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 19 on 60020" daemon prio=10 tid=0x0000000001b19800
>> nid=0x5550 runnable [0x000000004520e000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aab94af3848> (a sun.nio.ch.Util$1)
>> - locked<0x00002aab94af3830> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aac6ed3e1d8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aaac1194578> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aad890f2600> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aad890f2600> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aad890f2600> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aaae78be750> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aaae78be750> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aabba906cc0> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aac18963ef0> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aaacd2974c0> (a [B)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
>> anner.java:115)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
>> va:255)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
>> va:394)
>> - locked<0x00002aab328ec218> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :293)
>> - locked<0x00002aab328ec218> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aabbaf1c898> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aabbaf1c898> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 18 on 60020" daemon prio=10 tid=0x0000000001b17800
>> nid=0x554f runnable [0x000000004510d000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aaae98101b0> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaae9810198> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaaeba1b040> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aab1ac7d288> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aad890f0db8> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aad890f0db8> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aad890f0db8> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aab8c671d30> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aab8c671d30> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aab8c4f3de0> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aac18963dd0> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aab8d9fd7c8> (a [B)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
>> anner.java:115)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
>> va:255)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
>> va:394)
>> - locked<0x00002aaba69901b8> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :293)
>> - locked<0x00002aaba69901b8> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aabbaf1e468> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aabbaf1e468> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 17 on 60020" daemon prio=10 tid=0x0000000001b15800
>> nid=0x554e runnable [0x000000004500c000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aaac2d0d348> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaac2d0d330> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aacbb364b18> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aaac2750e38> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aaac1028128> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aaac1028128> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aaac1028128> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aaae41f13d8> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aaae41f13d8> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aab3c08c348> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aab52d54ef8> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aac0cb261b0> (a [B)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>> ner.java:87)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :82)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :262)
>> - locked<0x00002aaabefb9c10> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aad30d56688> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aad30d56688> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 16 on 60020" daemon prio=10 tid=0x00000000013fa000
>> nid=0x554d runnable [0x0000000044f0b000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aabec61aa08> (a sun.nio.ch.Util$1)
>> - locked<0x00002aabec61a9f0> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaae7590720> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aacf2246cb0> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aad06dd2c70> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aad06dd2c70> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aad06dd2c70> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aabe38abcd8> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aabe38abcd8> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aaad08ca208> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aac189636e0> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aab84fe43c8> (a [B)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(St
>> oreFileScanner.java:158)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
>> anner.java:110)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
>> va:255)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
>> va:394)
>> - locked<0x00002aad072dbd30> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :293)
>> - locked<0x00002aad072dbd30> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aabdbaa0490> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aabdbaa0490> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 15 on 60020" daemon prio=10 tid=0x00000000013f7800
>> nid=0x554c runnable [0x0000000044e0a000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aac558cde50> (a sun.nio.ch.Util$1)
>> - locked<0x00002aac558cde38> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aac5cd41558> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aadcf530468> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aad427a97a8> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aad427a97a8> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aad427a97a8> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aab9410bc00> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aab9410bc00> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aaca7cfcf70> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aacf952a6c8> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aaac4472350> (a [B)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>> ner.java:87)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :82)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :297)
>> - locked<0x00002aaaefe12070> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aabbaf1d1a8> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aabbaf1d1a8> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 14 on 60020" daemon prio=10 tid=0x00000000013f5800
>> nid=0x554b runnable [0x0000000044d09000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aab28f91848> (a sun.nio.ch.Util$1)
>> - locked<0x00002aab28f91860> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaaf5031970> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aacec8cf590> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aaaf789b048> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aaaf789b048> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aaaf789b048> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aab9c1df470> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aab9c1df470> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aab7ec60418> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aab44c84dd8> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aab227a2dc8> (a [B)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(St
>> oreFileScanner.java:158)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
>> anner.java:110)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
>> va:255)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
>> va:394)
>> - locked<0x00002aab50d25f28> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :293)
>> - locked<0x00002aab50d25f28> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aaae9efd9f0> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aaae9efd9f0> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 13 on 60020" daemon prio=10 tid=0x0000000001b61800
>> nid=0x554a waiting for monitor entry [0x0000000044c09000]
>> java.lang.Thread.State: BLOCKED (on object monitor)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - waiting to lock<0x00002aacc684dad0> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 12 on 60020" daemon prio=10 tid=0x0000000001b5f800
>> nid=0x5549 runnable [0x0000000044b07000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aab618bb060> (a sun.nio.ch.Util$1)
>> - locked<0x00002aabe1d82718> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aab76896238> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aab016dc128> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aad03182340> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aad03182340> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aad03182340> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aab8b32e3e0> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aab8b32e3e0> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aab8b436170> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aacd4fb8038> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aab8b4b4908> (a [B)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>> ner.java:87)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :82)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :262)
>> - locked<0x00002aab480f55b8> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aaae9efdb30> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aaae9efdb30> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 11 on 60020" daemon prio=10 tid=0x0000000001b5d000
>> nid=0x5548 waiting for monitor entry [0x0000000044a07000]
>> java.lang.Thread.State: BLOCKED (on object monitor)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - waiting to lock<0x00002aacc684dad0> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 10 on 60020" daemon prio=10 tid=0x000000000196a800
>> nid=0x5547 runnable [0x0000000044905000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aad7443dcd8> (a sun.nio.ch.Util$1)
>> - locked<0x00002aad7a31cb00> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aac9067d2d8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aad5103b0a0> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aadc79059c8> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aadc79059c8> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aadc79059c8> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aabf0f472b0> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aabf0f472b0> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aab8b2b29b0> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aad0477a310> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aabb7b16fd0> (a [B)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>> ner.java:87)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :82)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :262)
>> - locked<0x00002aab59165e10> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aac8e37dd58> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aac8e37dd58> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 9 on 60020" daemon prio=10 tid=0x0000000001968000
>> nid=0x5546 waiting for monitor entry [0x0000000044805000]
>> java.lang.Thread.State: BLOCKED (on object monitor)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.isFilterDone(
>> HRegion.java:2324)
>> - waiting to lock<0x00002aacc684dad0> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1850)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 8 on 60020" daemon prio=10 tid=0x0000000001967000
>> nid=0x5545 runnable [0x0000000044703000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aabb0b7b868> (a sun.nio.ch.Util$1)
>> - locked<0x00002aabb0b7b880> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaac79442c8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aad0106a6b0> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aac108e6768> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aac108e6768> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aac108e6768> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aac4d4ecd40> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aac4d4ecd40> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aaad20c3b08> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aac536bda88> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aaadfc86320> (a [B)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.loadBlock(HFile.ja
>> va:1447)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.reseekTo(HFile.jav
>> a:1325)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(St
>> oreFileScanner.java:152)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
>> anner.java:110)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
>> va:255)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
>> va:394)
>> - locked<0x00002aad072dc1c8> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :293)
>> - locked<0x00002aad072dc1c8> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aac8e37d808> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aac8e37d808> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 7 on 60020" daemon prio=10 tid=0x00000000016de800
>> nid=0x5544 runnable [0x0000000044602000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aab9f4dc588> (a sun.nio.ch.Util$1)
>> - locked<0x00002aab9f4dc570> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aac6ed3e250> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aad003a0178> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aabbd698ec8> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aabbd698ec8> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aabbd698ec8> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aaad91ecfa8> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aaad91ecfa8> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aac237c5090> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aad65d9a7f0> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aaaccc5d6c8> (a [B)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>> ner.java:87)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :82)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :262)
>> - locked<0x00002aad15b7d2b8> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aacc684dad0> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aacc684dad0> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 6 on 60020" daemon prio=10 tid=0x00000000016dd000
>> nid=0x5543 runnable [0x0000000044501000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aaaf7db82f0> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaaf7db82d8> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aab6f3339a8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aab494cdcb8> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aad890f5018> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aad890f5018> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aad890f5018> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aaae4d56928> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aaae4d56928> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aaae8d98798> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aaaaf686f40> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aaae77f0ad8> (a [B)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>> ner.java:87)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :82)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :262)
>> - locked<0x00002aaae4cd85a0> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aad30d55d28> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aad30d55d28> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 5 on 60020" daemon prio=10 tid=0x000000000194c800
>> nid=0x5542 runnable [0x0000000044400000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aaac5bf0ff0> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaac5bf10c8> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aab0492d6d8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aac451e5d20> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 89)
>> - locked<0x00002aabf9312408> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aabf9312408> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aabf9312408> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aab78dc6068> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aab78dc6068> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aab14390138> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aab7fbb83e8> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aac2cbf46c8> (a [B)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
>> anner.java:115)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
>> va:255)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
>> va:394)
>> - locked<0x00002aad15b7da98> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :293)
>> - locked<0x00002aad15b7da98> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aabc7a3d3e8> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aabc7a3d3e8> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 4 on 60020" daemon prio=10 tid=0x000000000194a800
>> nid=0x5541 runnable [0x00000000442ff000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aabbae85af8> (a sun.nio.ch.Util$1)
>> - locked<0x00002aabbae85b10> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaae379da00> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aac7ecf9bb8> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aaacf49d528> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aaacf49d528> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aaacf49d528> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aabdc652620> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aabdc652620> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aac357f4350> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aab9fc22a58> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aabdbefbe20> (a [B)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>> ner.java:87)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :82)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :262)
>> - locked<0x00002aad97b3b7e8> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aab016d6618> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aab016d6618> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 3 on 60020" daemon prio=10 tid=0x00000000019c4000
>> nid=0x5540 waiting for monitor entry [0x00000000441ff000]
>> java.lang.Thread.State: BLOCKED (on object monitor)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - waiting to lock<0x00002aab28d7de90> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 2 on 60020" daemon prio=10 tid=0x00000000019c2000
>> nid=0x553f runnable [0x00000000440fd000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aaaeef929f8> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaaeef92a10> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaaffd9ef38> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aab2ac42300> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aabc43bef18> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aabc43bef18> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aabc43bef18> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aabd53c1d58> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aabd53c1d58> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aabb602a180> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aab44c85f18> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aac0d2be0b0> (a [B)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>> ner.java:87)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :82)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :262)
>> - locked<0x00002aab4eb7a680> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aabc7a3cc68> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aabc7a3cc68> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 1 on 60020" daemon prio=10 tid=0x0000000001535800
>> nid=0x553e runnable [0x0000000043ffc000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aad28449198> (a sun.nio.ch.Util$1)
>> - locked<0x00002aad284491b0> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aab3a682700> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aac5b5cbd28> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aabc43bd778> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aabc43bd778> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aabc43bd778> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aac4d4f9b90> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aac4d4f9b90> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aac500ac780> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aabe2211268> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aac537126f0> (a [B)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>> ner.java:87)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :82)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :262)
>> - locked<0x00002aadb912fca8> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :114)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
>> HRegion.java:2344)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2301)
>> - locked<0x00002aadcba8fe98> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - locked<0x00002aadcba8fe98> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server handler 0 on 60020" daemon prio=10 tid=0x0000000001534000
>> nid=0x553d waiting for monitor entry [0x0000000043efc000]
>> java.lang.Thread.State: BLOCKED (on object monitor)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
>> java:2317)
>> - waiting to lock<0x00002aabc7a3d258> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
>> va:1832)
>> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>> Impl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
>> 9)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server listener on 60020" daemon prio=10 tid=0x00002aae4054e000
>> nid=0x553c runnable [0x0000000043dfb000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aaabc18bdf8> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaabc18bde0> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaabbb1ced8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:41
>> 4)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Server Responder" daemon prio=10 tid=0x00002aae4053d800 nid=0x553b
>> runnable [0x0000000043cfa000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aaabc2ceed8> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaabc2ceef0> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaabbb33fa8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:5
>> 88)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "Timer-0" daemon prio=10 tid=0x00002aae4053c800 nid=0x553a in
>> Object.wait() [0x0000000043bf9000]
>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at java.util.TimerThread.mainLoop(Timer.java:531)
>> - locked<0x00002aaabd062850> (a java.util.TaskQueue)
>> at java.util.TimerThread.run(Timer.java:484)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "983053578@qtp-1891789590-1 - Acceptor0
>> SelectChannelConnector@0.0.0.0:60030" prio=10 tid=0x00002aae40381000
>> nid=0x5539 runnable [0x0000000043af8000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aaabd07c460> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaabd07c478> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaabbb26b78> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.ja
>> va:498)
>> at
>> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
>> at
>> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnect
>> or.java:124)
>> at
>> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:
>> 708)
>> at
>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java
>> :582)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "1148428095@qtp-1891789590-0" prio=10 tid=0x00002aae403d2000 nid=0x5538
>> in Object.wait() [0x00000000439f7000]
>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at
>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java
>> :626)
>> - locked<0x00002aaabbc4c8f0> (a
>> org.mortbay.thread.QueuedThreadPool$PoolThread)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "regionserver60020.leaseChecker" daemon prio=10 tid=0x00002aae402ee800
>> nid=0x5537 waiting for monitor entry [0x00000000438f6000]
>> java.lang.Thread.State: BLOCKED (on object monitor)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.close(HRegion
>> .java:2409)
>> - waiting to lock<0x00002aadcba8fe98> (a
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer$ScannerListener.lease
>> Expired(HRegionServer.java:1892)
>> at
>> org.apache.hadoop.hbase.regionserver.Leases.run(Leases.java:99)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "regionserver60020.majorCompactionChecker" daemon prio=10
>> tid=0x00002aae402ec800 nid=0x5536 in Object.wait() [0x00000000437f5000]
>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> - waiting on<0x00002aaabc1f7bf0> (a java.lang.Object)
>> at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
>> - locked<0x00002aaabc1f7bf0> (a java.lang.Object)
>> at org.apache.hadoop.hbase.Chore.run(Chore.java:74)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "regionserver60020.compactor" daemon prio=10 tid=0x00002aae402eb000
>> nid=0x5535 runnable [0x00000000436f4000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aab26f462e0> (a sun.nio.ch.Util$1)
>> - locked<0x00002aab26f462c8> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaaeba1ac80> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
>> thTimeout.java:332)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
>> 157)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>> at
>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>> at
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>> - locked<0x00002aab44c85258> (a java.io.BufferedInputStream)
>> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
>> 50)
>> - locked<0x00002aaac6caf140> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
>> a:237)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>> at
>> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>> - locked<0x00002aaac6caf140> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
>> - locked<0x00002aaac6caf140> (a
>> org.apache.hadoop.hdfs.DFSClient$BlockReader)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
>> a:1899)
>> - locked<0x00002aaabd63fbb0> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
>> )
>> - locked<0x00002aaabd63fbb0> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at
>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
>> dRangeFileInputStream.java:105)
>> - locked<0x00002aab21b4e148> (a
>> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> - locked<0x00002aaaaedc39d0> (a java.io.BufferedInputStream)
>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
>> )
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
>> - locked<0x00002aab458cc280> (a [B)
>> at
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
>> 76)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
>> ner.java:87)
>> at
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
>> :82)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :297)
>> - locked<0x00002aaace390ec0> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
>> :326)
>> - locked<0x00002aaace390ec0> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>> at
>> org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:927)
>> at
>> org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:733)
>> - locked<0x00002aaacd20e840> (a java.lang.Object)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:
>> 769)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:
>> 714)
>> at
>> org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplit
>> Thread.java:81)
>>
>> Locked ownable synchronizers:
>> -<0x00002aaabce779f8> (a
>> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>>
>> "regionserver60020.cacheFlusher" daemon prio=10 tid=0x00002aae402d1800
>> nid=0x5534 waiting on condition [0x00000000435f3000]
>> java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabc241a10> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> aitNanos(AbstractQueuedSynchronizer.java:2081)
>> at java.util.concurrent.DelayQueue.poll(DelayQueue.java:230)
>> at java.util.concurrent.DelayQueue.poll(DelayQueue.java:68)
>> at
>> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher
>> .java:216)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "regionserver60020.logRoller" daemon prio=10 tid=0x00002aae402d1000
>> nid=0x5533 in Object.wait() [0x00000000434f2000]
>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at
>> org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:78)
>> - locked<0x00002aaabcec18e0> (a
>> java.util.concurrent.atomic.AtomicBoolean)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "Timer thread for monitoring jvm" daemon prio=10 tid=0x00002aae402b4000
>> nid=0x5532 in Object.wait() [0x00000000433f1000]
>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at java.util.TimerThread.mainLoop(Timer.java:531)
>> - locked<0x00002aaabcedce48> (a java.util.TaskQueue)
>> at java.util.TimerThread.run(Timer.java:484)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "Timer thread for monitoring hbase" daemon prio=10
>> tid=0x00002aae402b2800 nid=0x5531 in Object.wait() [0x00000000432f0000]
>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at java.util.TimerThread.mainLoop(Timer.java:531)
>> - locked<0x00002aaabceca148> (a java.util.TaskQueue)
>> at java.util.TimerThread.run(Timer.java:484)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "regionserver60020.logSyncer" daemon prio=10 tid=0x00002aae402b1000
>> nid=0x5530 waiting on condition [0x00000000431ef000]
>> java.lang.Thread.State: TIMED_WAITING (sleeping)
>> at java.lang.Thread.sleep(Native Method)
>> at
>> org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:96
>> 3)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "LeaseChecker" daemon prio=10 tid=0x00002aae4019a800 nid=0x552f
>> sleeping[0x00000000430ee000]
>> java.lang.Thread.State: TIMED_WAITING (sleeping)
>> at java.lang.Thread.sleep(Native Method)
>> at
>> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1167)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Client (47) connection to
>> doop10.dt.sv4.decarta.com/10.241.8.230:60000 from an unknown user"
>> daemon prio=10 tid=0x00000000015c9800 nid=0x5528 in Object.wait()
>> [0x0000000042cea000]
>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.waitForWork(HBaseClie
>> nt.java:431)
>> - locked<0x00002aaabc24dfc0> (a
>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:
>> 476)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "DestroyJavaVM" prio=10 tid=0x00002aae4015c800 nid=0x54fa waiting on
>> condition [0x0000000000000000]
>> java.lang.Thread.State: RUNNABLE
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "regionserver60020-EventThread" daemon prio=10 tid=0x00000000015c5000
>> nid=0x5526 waiting on condition [0x0000000042be9000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabc241a60> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> ait(AbstractQueuedSynchronizer.java:2043)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>> 86)
>> at
>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "regionserver60020-SendThread(doop3.dt.sv4.decarta.com:2181)" daemon
>> prio=10 tid=0x0000000001486800 nid=0x5525 runnable [0x0000000042ae8000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aaabbb3e538> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaabbb3e550> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaabbb1db08> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "regionserver60020" prio=10 tid=0x00002aae40187000 nid=0x5523 waiting on
>> condition [0x000000004028d000]
>> java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabc2faea0> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> aitNanos(AbstractQueuedSynchronizer.java:2081)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:4
>> 23)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.jav
>> a:621)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "Timer thread for monitoring rpc" daemon prio=10 tid=0x00002aae40135800
>> nid=0x5522 in Object.wait() [0x00000000429e7000]
>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at java.util.TimerThread.mainLoop(Timer.java:531)
>> - locked<0x00002aaabbb3cb08> (a java.util.TaskQueue)
>> at java.util.TimerThread.run(Timer.java:484)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "IPC Reader 9 on port 60020" prio=10 tid=0x00002aae40133000 nid=0x5521
>> runnable [0x00000000428e6000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aaabbb010a8> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaabbb010c0> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaabbb1e198> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
>> java:305)
>> - locked<0x00002aaabbb250e8> (a
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>> a:1110)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>> va:603)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Locked ownable synchronizers:
>> -<0x00002aaabbb4b008> (a
>> java.util.concurrent.ThreadPoolExecutor$Worker)
>>
>> "IPC Reader 8 on port 60020" prio=10 tid=0x00002aae40118000 nid=0x5520
>> runnable [0x00000000427e5000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aaabbb16118> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaabbb16130> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaabbb144f8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
>> java:305)
>> - locked<0x00002aaabbb30308> (a
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>> a:1110)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>> va:603)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Locked ownable synchronizers:
>> -<0x00002aaabbb32888> (a
>> java.util.concurrent.ThreadPoolExecutor$Worker)
>>
>> "IPC Reader 7 on port 60020" prio=10 tid=0x00002aae400fd000 nid=0x551f
>> runnable [0x00000000426e4000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aaabbb2a228> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaabbb2a240> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaabbb28ab8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
>> java:305)
>> - locked<0x00002aaabbb2a6d8> (a
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>> a:1110)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>> va:603)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Locked ownable synchronizers:
>> -<0x00002aaabbb4a068> (a
>> java.util.concurrent.ThreadPoolExecutor$Worker)
>>
>> "IPC Reader 6 on port 60020" prio=10 tid=0x00002aae400e2800 nid=0x551e
>> runnable [0x00000000425e3000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aaabbb27028> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaabbb27040> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaabbb258b8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
>> java:305)
>> - locked<0x00002aaabbb282e8> (a
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>> a:1110)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>> va:603)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Locked ownable synchronizers:
>> -<0x00002aaabbb4fc98> (a
>> java.util.concurrent.ThreadPoolExecutor$Worker)
>>
>> "IPC Reader 5 on port 60020" prio=10 tid=0x00002aae400c7800 nid=0x551d
>> runnable [0x00000000413e5000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aaabbb2ea08> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaabbb2ea20> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaabbb2d298> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
>> java:305)
>> - locked<0x00002aaabbb50c38> (a
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>> a:1110)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>> va:603)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Locked ownable synchronizers:
>> -<0x00002aaabbb523a8> (a
>> java.util.concurrent.ThreadPoolExecutor$Worker)
>>
>> "IPC Reader 4 on port 60020" prio=10 tid=0x00002aae400ac800 nid=0x551c
>> runnable [0x00000000412e4000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aaabbb2aea8> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaabbb2aec0> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaabbb1ca28> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
>> java:305)
>> - locked<0x00002aaabbb2c168> (a
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>> a:1110)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>> va:603)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Locked ownable synchronizers:
>> -<0x00002aaabbb48128> (a
>> java.util.concurrent.ThreadPoolExecutor$Worker)
>>
>> "IPC Reader 3 on port 60020" prio=10 tid=0x00002aae40091800 nid=0x551b
>> runnable [0x00000000411e3000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aaabbb15c68> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaabbb15c80> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaabbb12d88> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
>> java:305)
>> - locked<0x00002aaabbb165c8> (a
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>> a:1110)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>> va:603)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Locked ownable synchronizers:
>> -<0x00002aaabbb30ad8> (a
>> java.util.concurrent.ThreadPoolExecutor$Worker)
>>
>> "IPC Reader 2 on port 60020" prio=10 tid=0x00002aae40077000 nid=0x551a
>> runnable [0x00000000410e2000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aaabbb3d148> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaabbb3d160> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaabbb33828> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
>> java:305)
>> - locked<0x00002aaabbb3ebd8> (a
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>> a:1110)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>> va:603)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Locked ownable synchronizers:
>> -<0x00002aaabbb40028> (a
>> java.util.concurrent.ThreadPoolExecutor$Worker)
>>
>> "IPC Reader 1 on port 60020" prio=10 tid=0x00002aae4005d000 nid=0x5519
>> runnable [0x0000000040fe1000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aaabbb15c98> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaabbb15cb0> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaabbb12e00> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
>> java:305)
>> - locked<0x00002aaabbb165f0> (a
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>> a:1110)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>> va:603)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Locked ownable synchronizers:
>> -<0x00002aaabbb30b28> (a
>> java.util.concurrent.ThreadPoolExecutor$Worker)
>>
>> "IPC Reader 0 on port 60020" prio=10 tid=0x00002aae40003000 nid=0x5518
>> runnable [0x0000000040ee0000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aaabbb16148> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaabbb16160> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaabbb14570> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
>> java:305)
>> - locked<0x00002aaabbb30330> (a
>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>> a:1110)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>> va:603)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Locked ownable synchronizers:
>> -<0x00002aaabbb328d8> (a
>> java.util.concurrent.ThreadPoolExecutor$Worker)
>>
>> "main-EventThread" daemon prio=10 tid=0x00002aae3c1b6000 nid=0x5513
>> waiting on condition [0x00000000424e2000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for<0x00002aaabbb16618> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
>> ait(AbstractQueuedSynchronizer.java:2043)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
>> 86)
>> at
>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "main-SendThread(doop7.dt.sv4.decarta.com:2181)" daemon prio=10
>> tid=0x00002aae3c11b000 nid=0x5512 runnable [0x00000000423e1000]
>> java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> - locked<0x00002aaabbb16178> (a sun.nio.ch.Util$1)
>> - locked<0x00002aaabbb16190> (a
>> java.util.Collections$UnmodifiableSet)
>> - locked<0x00002aaabbb145e8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "Low Memory Detector" daemon prio=10 tid=0x000000000126c800 nid=0x5510
>> runnable [0x0000000000000000]
>> java.lang.Thread.State: RUNNABLE
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "CompilerThread1" daemon prio=10 tid=0x000000000126a800 nid=0x550f
>> waiting on condition [0x0000000000000000]
>> java.lang.Thread.State: RUNNABLE
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "CompilerThread0" daemon prio=10 tid=0x0000000001266000 nid=0x550e
>> waiting on condition [0x0000000000000000]
>> java.lang.Thread.State: RUNNABLE
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "Signal Dispatcher" daemon prio=10 tid=0x0000000001264000 nid=0x550d
>> runnable [0x0000000000000000]
>> java.lang.Thread.State: RUNNABLE
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "Surrogate Locker Thread (CMS)" daemon prio=10 tid=0x0000000001262000
>> nid=0x550c waiting on condition [0x0000000000000000]
>> java.lang.Thread.State: RUNNABLE
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "Finalizer" daemon prio=10 tid=0x000000000123b800 nid=0x550b in
>> Object.wait() [0x00000000422e0000]
>> java.lang.Thread.State: WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:133)
>> - locked<0x00002aaabbb15cf8> (a
>> java.lang.ref.ReferenceQueue$Lock)
>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:149)
>> at
>> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "Reference Handler" daemon prio=10 tid=0x0000000001239800 nid=0x550a in
>> Object.wait() [0x00000000421df000]
>> java.lang.Thread.State: WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at java.lang.Object.wait(Object.java:502)
>> at
>> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
>> - locked<0x00002aaabbb16d98> (a java.lang.ref.Reference$Lock)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "VM Thread" prio=10 tid=0x0000000001234800 nid=0x5509 runnable
>>
>> "Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x0000000000fe9000
>> nid=0x54fb runnable
>>
>> "Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x0000000000feb000
>> nid=0x54fc runnable
>>
>> "Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x0000000000fec800
>> nid=0x54fd runnable
>>
>> "Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x0000000000fee800
>> nid=0x54fe runnable
>>
>> "Gang worker#4 (Parallel GC Threads)" prio=10 tid=0x0000000000ff0800
>> nid=0x54ff runnable
>>
>> "Gang worker#5 (Parallel GC Threads)" prio=10 tid=0x0000000000ff2000
>> nid=0x5500 runnable
>>
>> "Gang worker#6 (Parallel GC Threads)" prio=10 tid=0x0000000000ff4000
>> nid=0x5501 runnable
>>
>> "Gang worker#7 (Parallel GC Threads)" prio=10 tid=0x0000000000ff6000
>> nid=0x5502 runnable
>>
>> "Gang worker#8 (Parallel GC Threads)" prio=10 tid=0x0000000000ff7800
>> nid=0x5503 runnable
>>
>> "Gang worker#9 (Parallel GC Threads)" prio=10 tid=0x0000000000ff9800
>> nid=0x5504 runnable
>>
>> "Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00000000010f4000
>> nid=0x5508 runnable
>> "Gang worker#0 (Parallel CMS Threads)" prio=10 tid=0x00000000010ee000
>> nid=0x5505 runnable
>>
>> "Gang worker#1 (Parallel CMS Threads)" prio=10 tid=0x00000000010f0000
>> nid=0x5506 runnable
>>
>> "Gang worker#2 (Parallel CMS Threads)" prio=10 tid=0x00000000010f2000
>> nid=0x5507 runnable
>>
>> "VM Periodic Task Thread" prio=10 tid=0x000000000126f800 nid=0x5511
>> waiting on condition
>>
>> JNI global references: 906
>>
>>
>> -----Original Message-----
>> From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of
>> Stack
>> Sent: Sunday, September 11, 2011 2:27 PM
>> To: user@hbase.apache.org
>> Subject: Re: scanner deadlock?
>>
>> 0.90.4 fixes two deadlocks (HBASE-4101 and HBASE-4077). Since then,
>> there is HBASE-4367 (Which has a posted patch).
>>
>> Below sounds like slowness. Can you thread dump the particular
>> regionserver and see what its up to? Is there other loading on the
>> system at the time? For example, loading on hdfs? Anything in the
>> hdfs logs for the datanode running beside the slow regionserver?
>>
>> St.Ack
>>
>> On Sat, Sep 10, 2011 at 5:50 PM, Geoff Hendrey<gh...@decarta.com>
>> wrote:
>>> Hi all -
>>>
>>>
>>>
>>> I'm still dealing with the saga of ScannerTimeoutException,
>>> UnknownScannerException, etc. I rewrote my code, in the hope that
>> simply
>>> a different approach and some different code paths might yield better
>>> results. No change. I tried many variations (caching 1 row vs caching
>>> many rows, changing the regionserver's lease, increasing the number of
>>> allowed zookeeper connections, etc). I created a fresh table on the
>>> thought that maybe there was some problem with the table...no change.
>>>
>>>
>>>
>>> I am dealing with what appears to be some sort of scanner deadlock. I
>>> have a total order partitioned mapreduce job. In the reducer, as long
>> as
>>> I use just one reducer, the task finishes quickly. But as soon as more
>>> than one reducer opens a scanner, the tasks procede in what I would
>> call
>>> a "jittery" lock step. They both are able to do ResultScanner.next() a
>>> few times, but then the call to next() freezes for a long period, and
>>> ScannerTimeoutException. I catch the exception, and get a new
>>> ResultScanner, and the pattern repeats. Jittery lockstep consisting of
>>> being able to get a few successful next() calls, then a lock up.
>>>
>>>
>>>
>>> I have not yet tried upgrading from 90.1 to higher. Nor have I tried
>>> tsuna's async client. Can anyone think of anything else I can try to
>>> resolve this? I've sunk quite a few late nights into this, and would
>> be
>>> very excited to find a solution.
>>>
>>>
>>>
>>> -geoff
>>>
>>>
>>
--
Eric
http://about.echarles.net
Re: scanner deadlock?
Posted by Stack <st...@duboce.net>.
On Sun, Sep 11, 2011 at 2:40 PM, Geoff Hendrey <gh...@decarta.com> wrote:
> Hi,
>
> The cluster is definitely loaded. We are running a MR job with 130
> reducers, each using a scanner. Ganglia shows an overall CPU utilization
> of around 40%, with network pushing 800MB/s. The problem may correlate
> with a major_compact I initiated couple days ago. Even today, I still
> see a lot of compacting (continuous) on region servers.
>
> Here is the thread dump from jstack. It does not reveal any explicit
> deadlock. At the time I dumped the jstack, the region server logs are
> reporting "IPC Server handler...ClosedChannelException". As soon as this
> begins occurring, our network throughput, as seen on Ganglia drops from
> 800MB/s down to virtually zero. Then of course there is the cascade of
> failures, eventually leading to the regionservers shutting down.
>
Looking at the thread dump, as you say, few are 'BLOCKED' (There are a
few BLOCKED thread trying to go into a next). But most are deep in a
next stuck polling looking to read. If you thread dump a few times
in a row, do any seem to be making progress? (They may be making
progress but slowly, as you say)
So, is it possible you have a slow datanode in your cluster? Can you
tell who all are connected too... lsof'ing? That might give you a
clue. Perhaps there is a node with sick disks (check dmesg). Or
perhaps the machine's nework card is sick?
St.Ack
> I don't see anything unusual in the datanode logs on the same server.
> Just HDFS_READ and HDFS_WRITE for the most part.
>
> jstack here:
> ===================================================================
> 2011-09-11 14:28:52
> Full thread dump OpenJDK 64-Bit Server VM (14.0-b16 mixed mode):
>
> "IPC Client (47) connection to
> doop10.dt.sv4.decarta.com/10.241.8.230:54310 from hroot" daemon prio=10
> tid=0x0000000001492800 nid=0x75cb in Object.wait() [0x0000000047633000]
> Â java.lang.Thread.State: TIMED_WAITING (on object monitor)
> Â Â Â Â at java.lang.Object.wait(Native Method)
> Â Â Â Â at
> org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:676)
> Â Â Â Â - locked <0x00002aaaaf680000> (a
> org.apache.hadoop.ipc.Client$Connection)
> Â Â Â Â at org.apache.hadoop.ipc.Client$Connection.run(Client.java:719)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "ResponseProcessor for block blk_2613854253703670198_5032689" daemon
> prio=10 tid=0x0000000001988800 nid=0x75bb runnable [0x0000000046926000]
> Â java.lang.Thread.State: RUNNABLE
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
> Â Â Â Â at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
> Â Â Â Â at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> Â Â Â Â - locked <0x00002aab69337248> (a sun.nio.ch.Util$1)
> Â Â Â Â - locked <0x00002aab69337230> (a
> java.util.Collections$UnmodifiableSet)
> Â Â Â Â - locked <0x00002aaaf3897930> (a sun.nio.ch.EPollSelectorImpl)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
> thTimeout.java:332)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
> 157)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
> Â Â Â Â at java.io.DataInputStream.readFully(DataInputStream.java:195)
> Â Â Â Â at java.io.DataInputStream.readLong(DataInputStream.java:416)
> Â Â Â Â at
> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFie
> lds(DataTransferProtocol.java:120)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(D
> FSClient.java:2638)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "DataStreamer for file
> /hbase-20.6/NAM_CLUSTERKEYS3/60db9e0113c7578ed913dc37e0856954/.tmp/42977
> 03314038441248 block blk_2613854253703670198_5032689" daemon prio=10
> tid=0x00002aae40e8e800 nid=0x75b8 in Object.wait() [0x0000000046e2b000]
> Â java.lang.Thread.State: TIMED_WAITING (on object monitor)
> Â Â Â Â at java.lang.Object.wait(Native Method)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSCli
> ent.java:2483)
> Â Â Â Â - locked <0x00002aab6551a2d8> (a java.util.LinkedList)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "ResponseProcessor for block blk_8900384789253382699_5032549" daemon
> prio=10 tid=0x00000000013cd000 nid=0x71d0 runnable [0x0000000047431000]
> Â java.lang.Thread.State: RUNNABLE
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
> Â Â Â Â at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
> Â Â Â Â at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> Â Â Â Â - locked <0x00002aaae0569530> (a sun.nio.ch.Util$1)
> Â Â Â Â - locked <0x00002aaae0569518> (a
> java.util.Collections$UnmodifiableSet)
> Â Â Â Â - locked <0x00002aaadb4c5c68> (a sun.nio.ch.EPollSelectorImpl)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
> thTimeout.java:332)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
> 157)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
> Â Â Â Â at java.io.DataInputStream.readFully(DataInputStream.java:195)
> Â Â Â Â at java.io.DataInputStream.readLong(DataInputStream.java:416)
> Â Â Â Â at
> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFie
> lds(DataTransferProtocol.java:120)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(D
> FSClient.java:2638)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "DataStreamer for file
> /hbase-20.6/.logs/doop5.dt.sv4.decarta.com,60020,1315768059645/doop5.dt.
> sv4.decarta.com%3A60020.1315776307423 block
> blk_8900384789253382699_5032549" daemon prio=10 tid=0x00000000016c3800
> nid=0x71c8 in Object.wait() [0x000000004712e000]
> Â java.lang.Thread.State: TIMED_WAITING (on object monitor)
> Â Â Â Â at java.lang.Object.wait(Native Method)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSCli
> ent.java:2483)
> Â Â Â Â - locked <0x00002aab092a7f38> (a java.util.LinkedList)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "Attach Listener" daemon prio=10 tid=0x0000000001e32000 nid=0x50f4
> waiting on condition [0x0000000000000000]
> Â java.lang.Thread.State: RUNNABLE
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "RS_CLOSE_REGION-doop5.dt.sv4.decarta.com,60020,1315768059645-2" daemon
> prio=10 tid=0x00000000018cb800 nid=0x505 waiting on condition
> [0x0000000047330000]
> Â java.lang.Thread.State: WAITING (parking)
> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for  <0x00002aaabcefb428> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> Â Â Â Â at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> Â Â Â Â at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
> ait(AbstractQueuedSynchronizer.java:2043)
> Â Â Â Â at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
> 86)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
> 1043)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
> a:1103)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
> va:603)
> Â Â Â Â at java.lang.Thread.run(Thread.java:636)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "RS_CLOSE_REGION-doop5.dt.sv4.decarta.com,60020,1315768059645-1" daemon
> prio=10 tid=0x000000000163e000 nid=0x504 waiting on condition
> [0x0000000047532000]
> Â java.lang.Thread.State: WAITING (parking)
> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for  <0x00002aaabcefb428> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> Â Â Â Â at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> Â Â Â Â at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
> ait(AbstractQueuedSynchronizer.java:2043)
> Â Â Â Â at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
> 86)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
> 1043)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
> a:1103)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
> va:603)
> Â Â Â Â at java.lang.Thread.run(Thread.java:636)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "RS_CLOSE_REGION-doop5.dt.sv4.decarta.com,60020,1315768059645-0" daemon
> prio=10 tid=0x0000000001bff800 nid=0x503 waiting on condition
> [0x0000000046f2c000]
> Â java.lang.Thread.State: WAITING (parking)
> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for  <0x00002aaabcefb428> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> Â Â Â Â at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> Â Â Â Â at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
> ait(AbstractQueuedSynchronizer.java:2043)
> Â Â Â Â at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
> 86)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
> 1043)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
> a:1103)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
> va:603)
> Â Â Â Â at java.lang.Thread.run(Thread.java:636)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "sendParams-6" daemon prio=10 tid=0x00002aae401f1800 nid=0x6d3b waiting
> on condition [0x000000004702d000]
> Â java.lang.Thread.State: TIMED_WAITING (parking)
> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for  <0x00002aaabceea260> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
> Â Â Â Â at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> Â Â Â Â at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Synchro
> nousQueue.java:453)
> Â Â Â Â at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(Synchronous
> Queue.java:352)
> Â Â Â Â at
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:903)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
> 1043)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
> a:1103)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
> va:603)
> Â Â Â Â at java.lang.Thread.run(Thread.java:636)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "RS_OPEN_REGION-doop5.dt.sv4.decarta.com,60020,1315768059645-2" daemon
> prio=10 tid=0x0000000001a7f000 nid=0x5b56 waiting on condition
> [0x0000000046d2a000]
> Â java.lang.Thread.State: WAITING (parking)
> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for  <0x00002aaabcefb2e8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> Â Â Â Â at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> Â Â Â Â at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
> ait(AbstractQueuedSynchronizer.java:2043)
> Â Â Â Â at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
> 86)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
> 1043)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
> a:1103)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
> va:603)
> Â Â Â Â at java.lang.Thread.run(Thread.java:636)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "RS_OPEN_REGION-doop5.dt.sv4.decarta.com,60020,1315768059645-1" daemon
> prio=10 tid=0x0000000001af2800 nid=0x5b55 waiting on condition
> [0x0000000042deb000]
> Â java.lang.Thread.State: WAITING (parking)
> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for  <0x00002aaabcefb2e8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> Â Â Â Â at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> Â Â Â Â at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
> ait(AbstractQueuedSynchronizer.java:2043)
> Â Â Â Â at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
> 86)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
> 1043)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
> a:1103)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
> va:603)
> Â Â Â Â at java.lang.Thread.run(Thread.java:636)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "RS_OPEN_REGION-doop5.dt.sv4.decarta.com,60020,1315768059645-0" daemon
> prio=10 tid=0x00000000014b3800 nid=0x5b54 waiting on condition
> [0x0000000046c29000]
> Â java.lang.Thread.State: WAITING (parking)
> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for  <0x00002aaabcefb2e8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> Â Â Â Â at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> Â Â Â Â at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
> ait(AbstractQueuedSynchronizer.java:2043)
> Â Â Â Â at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
> 86)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
> 1043)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
> a:1103)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
> va:603)
> Â Â Â Â at java.lang.Thread.run(Thread.java:636)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "RS_OPEN_ROOT-doop5.dt.sv4.decarta.com,60020,1315768059645-0" daemon
> prio=10 tid=0x0000000001ad2800 nid=0x556e waiting on condition
> [0x0000000046a27000]
> Â java.lang.Thread.State: WAITING (parking)
> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for  <0x00002aaabcee8870> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> Â Â Â Â at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> Â Â Â Â at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
> ait(AbstractQueuedSynchronizer.java:2043)
> Â Â Â Â at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
> 86)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
> 1043)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
> a:1103)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
> va:603)
> Â Â Â Â at java.lang.Thread.run(Thread.java:636)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "LRU Statistics #0" prio=10 tid=0x00002aae4048a800 nid=0x5566 waiting on
> condition [0x0000000046825000]
> Â java.lang.Thread.State: TIMED_WAITING (parking)
> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for  <0x00002aaabd0695c0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> Â Â Â Â at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> Â Â Â Â at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
> aitNanos(AbstractQueuedSynchronizer.java:2081)
> Â Â Â Â at java.util.concurrent.DelayQueue.take(DelayQueue.java:193)
> Â Â Â Â at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(S
> cheduledThreadPoolExecutor.java:688)
> Â Â Â Â at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(S
> cheduledThreadPoolExecutor.java:681)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
> 1043)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
> a:1103)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
> va:603)
> Â Â Â Â at java.lang.Thread.run(Thread.java:636)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "LruBlockCache.EvictionThread" daemon prio=10 tid=0x00002aae4049f800
> nid=0x5565 in Object.wait() [0x0000000046724000]
> Â java.lang.Thread.State: WAITING (on object monitor)
> Â Â Â Â at java.lang.Object.wait(Native Method)
> Â Â Â Â at java.lang.Object.wait(Object.java:502)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlo
> ckCache.java:519)
> Â Â Â Â - locked <0x00002aaabce92f50> (a
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "PRI IPC Server handler 9 on 60020" daemon prio=10
> tid=0x00002aae4057e000 nid=0x5564 waiting on condition
> [0x0000000046623000]
> Â java.lang.Thread.State: WAITING (parking)
> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for  <0x00002aaabbb307e0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> Â Â Â Â at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> Â Â Â Â at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
> ait(AbstractQueuedSynchronizer.java:2043)
> Â Â Â Â at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
> 86)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
> 5)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "PRI IPC Server handler 8 on 60020" daemon prio=10
> tid=0x00002aae4057c800 nid=0x5563 waiting on condition
> [0x0000000046522000]
> Â java.lang.Thread.State: WAITING (parking)
> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for  <0x00002aaabbb307e0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> Â Â Â Â at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> Â Â Â Â at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
> ait(AbstractQueuedSynchronizer.java:2043)
> Â Â Â Â at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
> 86)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
> 5)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "PRI IPC Server handler 7 on 60020" daemon prio=10
> tid=0x00002aae40559000 nid=0x5562 waiting on condition
> [0x0000000046421000]
> Â java.lang.Thread.State: WAITING (parking)
> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for  <0x00002aaabbb307e0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> Â Â Â Â at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> Â Â Â Â at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
> ait(AbstractQueuedSynchronizer.java:2043)
> Â Â Â Â at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
> 86)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
> 5)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "PRI IPC Server handler 6 on 60020" daemon prio=10
> tid=0x00002aae40557800 nid=0x5561 waiting on condition
> [0x0000000046320000]
> Â java.lang.Thread.State: WAITING (parking)
> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for  <0x00002aaabbb307e0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> Â Â Â Â at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> Â Â Â Â at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
> ait(AbstractQueuedSynchronizer.java:2043)
> Â Â Â Â at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
> 86)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
> 5)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "PRI IPC Server handler 5 on 60020" daemon prio=10
> tid=0x00002aae40555800 nid=0x5560 waiting on condition
> [0x000000004621f000]
> Â java.lang.Thread.State: WAITING (parking)
> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for  <0x00002aaabbb307e0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> Â Â Â Â at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> Â Â Â Â at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
> ait(AbstractQueuedSynchronizer.java:2043)
> Â Â Â Â at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
> 86)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
> 5)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "PRI IPC Server handler 4 on 60020" daemon prio=10
> tid=0x00002aae40553800 nid=0x555f waiting on condition
> [0x000000004611e000]
> Â java.lang.Thread.State: WAITING (parking)
> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for  <0x00002aaabbb307e0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> Â Â Â Â at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> Â Â Â Â at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
> ait(AbstractQueuedSynchronizer.java:2043)
> Â Â Â Â at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
> 86)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
> 5)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "PRI IPC Server handler 3 on 60020" daemon prio=10
> tid=0x00002aae4054f000 nid=0x555e waiting on condition
> [0x000000004601d000]
> Â java.lang.Thread.State: WAITING (parking)
> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for  <0x00002aaabbb307e0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> Â Â Â Â at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> Â Â Â Â at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
> ait(AbstractQueuedSynchronizer.java:2043)
> Â Â Â Â at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
> 86)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
> 5)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "PRI IPC Server handler 2 on 60020" daemon prio=10
> tid=0x00000000012f8800 nid=0x555d waiting on condition
> [0x0000000045f1c000]
> Â java.lang.Thread.State: WAITING (parking)
> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for  <0x00002aaabbb307e0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> Â Â Â Â at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> Â Â Â Â at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
> ait(AbstractQueuedSynchronizer.java:2043)
> Â Â Â Â at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
> 86)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
> 5)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "PRI IPC Server handler 1 on 60020" daemon prio=10
> tid=0x0000000001991000 nid=0x555c waiting on condition
> [0x0000000045e1b000]
> Â java.lang.Thread.State: WAITING (parking)
> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for  <0x00002aaabbb307e0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> Â Â Â Â at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> Â Â Â Â at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
> ait(AbstractQueuedSynchronizer.java:2043)
> Â Â Â Â at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
> 86)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
> 5)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "PRI IPC Server handler 0 on 60020" daemon prio=10
> tid=0x000000000198e800 nid=0x555b waiting on condition
> [0x0000000045d1a000]
> Â java.lang.Thread.State: WAITING (parking)
> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for  <0x00002aaabbb307e0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> Â Â Â Â at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> Â Â Â Â at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
> ait(AbstractQueuedSynchronizer.java:2043)
> Â Â Â Â at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
> 86)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
> 5)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "IPC Server handler 29 on 60020" daemon prio=10 tid=0x000000000198c800
> nid=0x555a runnable [0x0000000045c18000]
> Â java.lang.Thread.State: RUNNABLE
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
> Â Â Â Â at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
> Â Â Â Â at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> Â Â Â Â - locked <0x00002aac053b8450> (a sun.nio.ch.Util$1)
> Â Â Â Â - locked <0x00002aac053b8438> (a
> java.util.Collections$UnmodifiableSet)
> Â Â Â Â - locked <0x00002aab1037afc8> (a sun.nio.ch.EPollSelectorImpl)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
> thTimeout.java:332)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
> 157)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
> Â Â Â Â at
> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
> Â Â Â Â - locked <0x00002aac92976a30> (a java.io.BufferedInputStream)
> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
> 50)
> Â Â Â Â - locked <0x00002aabc43be0a8> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
> a:237)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
> Â Â Â Â - locked <0x00002aabc43be0a8> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
> Â Â Â Â - locked <0x00002aabc43be0a8> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
> a:1899)
> Â Â Â Â - locked <0x00002aac1a4e0ea8> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
> )
> Â Â Â Â - locked <0x00002aac1a4e0ea8> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
> dRangeFileInputStream.java:105)
> Â Â Â Â - locked <0x00002aac237c37a8> (a
> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
> Â Â Â Â at
> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> Â Â Â Â - locked <0x00002aad39485080> (a java.io.BufferedInputStream)
> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
> )
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
> Â Â Â Â - locked <0x00002aab0b5ba8a8> (a [B)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
> 76)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
> ner.java:87)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
> :82)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
> :262)
> Â Â Â Â - locked <0x00002aab89f59aa8> (a
> org.apache.hadoop.hbase.regionserver.StoreScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
> :114)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
> HRegion.java:2344)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2301)
> Â Â Â Â - locked <0x00002aab016d6988> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2317)
> Â Â Â Â - locked <0x00002aab016d6988> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
> va:1832)
> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> Â Â Â Â at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:43)
> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
> 9)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "IPC Server handler 28 on 60020" daemon prio=10 tid=0x0000000001864000
> nid=0x5559 runnable [0x0000000045b17000]
> Â java.lang.Thread.State: RUNNABLE
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
> Â Â Â Â at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
> Â Â Â Â at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> Â Â Â Â - locked <0x00002aabaedb0608> (a sun.nio.ch.Util$1)
> Â Â Â Â - locked <0x00002aabaedb05f0> (a
> java.util.Collections$UnmodifiableSet)
> Â Â Â Â - locked <0x00002aaca096fa50> (a sun.nio.ch.EPollSelectorImpl)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
> thTimeout.java:332)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
> 157)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
> Â Â Â Â at
> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
> Â Â Â Â - locked <0x00002aac62a70b28> (a java.io.BufferedInputStream)
> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
> 50)
> Â Â Â Â - locked <0x00002aab856e6220> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
> a:237)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
> Â Â Â Â - locked <0x00002aab856e6220> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
> Â Â Â Â - locked <0x00002aab856e6220> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
> a:1899)
> Â Â Â Â - locked <0x00002aab0264afe8> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
> )
> Â Â Â Â - locked <0x00002aab0264afe8> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
> dRangeFileInputStream.java:105)
> Â Â Â Â - locked <0x00002aaae070be48> (a
> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
> Â Â Â Â at
> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> Â Â Â Â - locked <0x00002aac18961940> (a java.io.BufferedInputStream)
> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
> )
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
> Â Â Â Â - locked <0x00002aabad8605d0> (a [B)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
> 76)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
> ner.java:87)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
> :82)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
> :262)
> Â Â Â Â - locked <0x00002aab077e5d80> (a
> org.apache.hadoop.hbase.regionserver.StoreScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
> :114)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
> HRegion.java:2344)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2301)
> Â Â Â Â - locked <0x00002aad30d57c18> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2317)
> Â Â Â Â - locked <0x00002aad30d57c18> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
> va:1832)
> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> Â Â Â Â at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:43)
> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
> 9)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "IPC Server handler 27 on 60020" daemon prio=10 tid=0x0000000001862000
> nid=0x5558 runnable [0x0000000045a16000]
> Â java.lang.Thread.State: RUNNABLE
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
> Â Â Â Â at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
> Â Â Â Â at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> Â Â Â Â - locked <0x00002aab2ea61b70> (a sun.nio.ch.Util$1)
> Â Â Â Â - locked <0x00002aab2ea61b88> (a
> java.util.Collections$UnmodifiableSet)
> Â Â Â Â - locked <0x00002aacbb364578> (a sun.nio.ch.EPollSelectorImpl)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
> thTimeout.java:332)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
> 157)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
> Â Â Â Â at
> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
> Â Â Â Â - locked <0x00002aab1ffc0d00> (a java.io.BufferedInputStream)
> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
> 50)
> Â Â Â Â - locked <0x00002aad890f4640> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
> a:237)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
> Â Â Â Â - locked <0x00002aad890f4640> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
> Â Â Â Â - locked <0x00002aad890f4640> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
> a:1899)
> Â Â Â Â - locked <0x00002aaba1282b10> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
> )
> Â Â Â Â - locked <0x00002aaba1282b10> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
> dRangeFileInputStream.java:105)
> Â Â Â Â - locked <0x00002aac528d15d0> (a
> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
> Â Â Â Â at
> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> Â Â Â Â - locked <0x00002aaaaf68a630> (a java.io.BufferedInputStream)
> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
> )
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
> Â Â Â Â - locked <0x00002aabc5c948b8> (a [B)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
> 76)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
> anner.java:115)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
> va:255)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
> va:394)
> Â Â Â Â - locked <0x00002aab95832718> (a
> org.apache.hadoop.hbase.regionserver.StoreScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
> :293)
> Â Â Â Â - locked <0x00002aab95832718> (a
> org.apache.hadoop.hbase.regionserver.StoreScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
> :114)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
> HRegion.java:2344)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2301)
> Â Â Â Â - locked <0x00002aab28d7de90> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2317)
> Â Â Â Â - locked <0x00002aab28d7de90> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
> va:1832)
> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> Â Â Â Â at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:43)
> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
> 9)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "IPC Server handler 26 on 60020" daemon prio=10 tid=0x0000000001860000
> nid=0x5557 runnable [0x0000000045915000]
> Â java.lang.Thread.State: RUNNABLE
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
> Â Â Â Â at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
> Â Â Â Â at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> Â Â Â Â - locked <0x00002aabaaa6b808> (a sun.nio.ch.Util$1)
> Â Â Â Â - locked <0x00002aabaaa6b7f0> (a
> java.util.Collections$UnmodifiableSet)
> Â Â Â Â - locked <0x00002aaaf3899ca8> (a sun.nio.ch.EPollSelectorImpl)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
> thTimeout.java:332)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
> 157)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
> Â Â Â Â at
> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
> Â Â Â Â - locked <0x00002aaac1f59118> (a java.io.BufferedInputStream)
> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
> 50)
> Â Â Â Â - locked <0x00002aaac0e99db0> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
> a:237)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
> Â Â Â Â - locked <0x00002aaac0e99db0> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
> Â Â Â Â - locked <0x00002aaac0e99db0> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
> a:1899)
> Â Â Â Â - locked <0x00002aabb7221e88> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
> )
> Â Â Â Â - locked <0x00002aabb7221e88> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
> dRangeFileInputStream.java:105)
> Â Â Â Â - locked <0x00002aac28621da0> (a
> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
> Â Â Â Â at
> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> Â Â Â Â - locked <0x00002aac18962690> (a java.io.BufferedInputStream)
> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
> )
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
> Â Â Â Â - locked <0x00002aab6876b850> (a [B)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.loadBlock(HFile.ja
> va:1447)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.reseekTo(HFile.jav
> a:1325)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(St
> oreFileScanner.java:152)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
> anner.java:110)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
> va:255)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
> va:394)
> Â Â Â Â - locked <0x00002aaabe3468f8> (a
> org.apache.hadoop.hbase.regionserver.StoreScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
> :293)
> Â Â Â Â - locked <0x00002aaabe3468f8> (a
> org.apache.hadoop.hbase.regionserver.StoreScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
> :114)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
> HRegion.java:2344)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2301)
> Â Â Â Â - locked <0x00002aac8e37dbc8> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2317)
> Â Â Â Â - locked <0x00002aac8e37dbc8> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
> va:1832)
> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> Â Â Â Â at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:43)
> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
> 9)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "IPC Server handler 25 on 60020" daemon prio=10 tid=0x000000000173d800
> nid=0x5556 runnable [0x0000000045814000]
> Â java.lang.Thread.State: RUNNABLE
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
> Â Â Â Â at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
> Â Â Â Â at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> Â Â Â Â - locked <0x00002aab20178510> (a sun.nio.ch.Util$1)
> Â Â Â Â - locked <0x00002aab20178528> (a
> java.util.Collections$UnmodifiableSet)
> Â Â Â Â - locked <0x00002aaad6187af0> (a sun.nio.ch.EPollSelectorImpl)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
> thTimeout.java:332)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
> 157)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
> Â Â Â Â at
> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
> Â Â Â Â - locked <0x00002aac891a6610> (a java.io.BufferedInputStream)
> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
> 50)
> Â Â Â Â - locked <0x00002aac1de6d9f8> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
> a:237)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
> Â Â Â Â - locked <0x00002aac1de6d9f8> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
> Â Â Â Â - locked <0x00002aac1de6d9f8> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
> a:1899)
> Â Â Â Â - locked <0x00002aaac041c6c8> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
> )
> Â Â Â Â - locked <0x00002aaac041c6c8> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
> dRangeFileInputStream.java:105)
> Â Â Â Â - locked <0x00002aab25e717e0> (a
> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
> Â Â Â Â at
> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> Â Â Â Â - locked <0x00002aac3e6e4e28> (a java.io.BufferedInputStream)
> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
> )
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
> Â Â Â Â - locked <0x00002aaad4a425f0> (a [B)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
> 76)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
> ner.java:87)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
> :82)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
> :262)
> Â Â Â Â - locked <0x00002aaabd5eb7e0> (a
> org.apache.hadoop.hbase.regionserver.StoreScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
> :114)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
> HRegion.java:2344)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2301)
> Â Â Â Â - locked <0x00002aac8e37dc68> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2317)
> Â Â Â Â - locked <0x00002aac8e37dc68> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
> va:1832)
> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> Â Â Â Â at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:43)
> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
> 9)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "IPC Server handler 24 on 60020" daemon prio=10 tid=0x000000000173a000
> nid=0x5555 waiting for monitor entry [0x0000000045714000]
> Â java.lang.Thread.State: BLOCKED (on object monitor)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2317)
> Â Â Â Â - waiting to lock <0x00002aab016d6618> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
> va:1832)
> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> Â Â Â Â at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:43)
> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
> 9)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "IPC Server handler 23 on 60020" daemon prio=10 tid=0x0000000001444000
> nid=0x5554 runnable [0x0000000045612000]
> Â java.lang.Thread.State: RUNNABLE
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
> Â Â Â Â at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
> Â Â Â Â at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> Â Â Â Â - locked <0x00002aad70044d30> (a sun.nio.ch.Util$1)
> Â Â Â Â - locked <0x00002aad70044d48> (a
> java.util.Collections$UnmodifiableSet)
> Â Â Â Â - locked <0x00002aab328f1a40> (a sun.nio.ch.EPollSelectorImpl)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
> thTimeout.java:332)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
> 157)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
> Â Â Â Â at
> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
> Â Â Â Â - locked <0x00002aaac9d48e68> (a java.io.BufferedInputStream)
> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
> 50)
> Â Â Â Â - locked <0x00002aac1de6dd40> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
> a:237)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
> Â Â Â Â - locked <0x00002aac1de6dd40> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
> Â Â Â Â - locked <0x00002aac1de6dd40> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
> a:1899)
> Â Â Â Â - locked <0x00002aab39d88148> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
> )
> Â Â Â Â - locked <0x00002aab39d88148> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
> dRangeFileInputStream.java:105)
> Â Â Â Â - locked <0x00002aab7addd0d8> (a
> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
> Â Â Â Â at
> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> Â Â Â Â - locked <0x00002aab2811e8e8> (a java.io.BufferedInputStream)
> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
> )
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
> Â Â Â Â - locked <0x00002aabe56e8858> (a [B)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
> 76)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
> ner.java:87)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
> :82)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
> :262)
> Â Â Â Â - locked <0x00002aad15b7d360> (a
> org.apache.hadoop.hbase.regionserver.StoreScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
> :114)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
> HRegion.java:2344)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2301)
> Â Â Â Â - locked <0x00002aabc7a3d258> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2317)
> Â Â Â Â - locked <0x00002aabc7a3d258> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
> va:1832)
> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> Â Â Â Â at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:43)
> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
> 9)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "IPC Server handler 22 on 60020" daemon prio=10 tid=0x0000000001442000
> nid=0x5553 runnable [0x0000000045511000]
> Â java.lang.Thread.State: RUNNABLE
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
> Â Â Â Â at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
> Â Â Â Â at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> Â Â Â Â - locked <0x00002aabf6e38450> (a sun.nio.ch.Util$1)
> Â Â Â Â - locked <0x00002aabf6e38468> (a
> java.util.Collections$UnmodifiableSet)
> Â Â Â Â - locked <0x00002aabb9e99468> (a sun.nio.ch.EPollSelectorImpl)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
> thTimeout.java:332)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
> 157)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
> Â Â Â Â at
> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
> Â Â Â Â - locked <0x00002aaae13c0340> (a java.io.BufferedInputStream)
> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
> 50)
> Â Â Â Â - locked <0x00002aad427a91c0> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
> a:237)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
> Â Â Â Â - locked <0x00002aad427a91c0> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
> Â Â Â Â - locked <0x00002aad427a91c0> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
> a:1899)
> Â Â Â Â - locked <0x00002aab7c7ca0c8> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
> )
> Â Â Â Â - locked <0x00002aab7c7ca0c8> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
> dRangeFileInputStream.java:105)
> Â Â Â Â - locked <0x00002aab3748f1a8> (a
> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
> Â Â Â Â at
> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> Â Â Â Â - locked <0x00002aab52d56ed8> (a java.io.BufferedInputStream)
> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
> )
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
> Â Â Â Â - locked <0x00002aab60a86ca8> (a [B)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
> 76)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(St
> oreFileScanner.java:158)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
> anner.java:110)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
> va:255)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
> va:394)
> Â Â Â Â - locked <0x00002aab706a3e98> (a
> org.apache.hadoop.hbase.regionserver.StoreScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
> :293)
> Â Â Â Â - locked <0x00002aab706a3e98> (a
> org.apache.hadoop.hbase.regionserver.StoreScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
> :114)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
> HRegion.java:2344)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2301)
> Â Â Â Â - locked <0x00002aaae3489758> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2317)
> Â Â Â Â - locked <0x00002aaae3489758> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
> va:1832)
> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> Â Â Â Â at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:43)
> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
> 9)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "IPC Server handler 21 on 60020" daemon prio=10 tid=0x0000000001792000
> nid=0x5552 waiting for monitor entry [0x0000000045411000]
> Â java.lang.Thread.State: BLOCKED (on object monitor)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2317)
> Â Â Â Â - waiting to lock <0x00002aabc7a3d3e8> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
> va:1832)
> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> Â Â Â Â at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:43)
> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
> 9)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "IPC Server handler 20 on 60020" daemon prio=10 tid=0x000000000178e000
> nid=0x5551 runnable [0x000000004530f000]
> Â java.lang.Thread.State: RUNNABLE
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
> Â Â Â Â at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
> Â Â Â Â at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> Â Â Â Â - locked <0x00002aab3745e018> (a sun.nio.ch.Util$1)
> Â Â Â Â - locked <0x00002aab3745e030> (a
> java.util.Collections$UnmodifiableSet)
> Â Â Â Â - locked <0x00002aab7d5cc0c0> (a sun.nio.ch.EPollSelectorImpl)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
> thTimeout.java:332)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
> 157)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
> Â Â Â Â at
> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
> Â Â Â Â - locked <0x00002aac035e4990> (a java.io.BufferedInputStream)
> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
> 50)
> Â Â Â Â - locked <0x00002aad890f42f8> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
> a:237)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
> Â Â Â Â - locked <0x00002aad890f42f8> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
> Â Â Â Â - locked <0x00002aad890f42f8> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
> a:1899)
> Â Â Â Â - locked <0x00002aab1baffa50> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
> )
> Â Â Â Â - locked <0x00002aab1baffa50> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
> dRangeFileInputStream.java:105)
> Â Â Â Â - locked <0x00002aaadef99fe8> (a
> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
> Â Â Â Â at
> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> Â Â Â Â - locked <0x00002aac189625a0> (a java.io.BufferedInputStream)
> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
> )
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
> Â Â Â Â - locked <0x00002aabb88c7ce0> (a [B)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
> 76)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
> anner.java:115)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
> va:255)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
> va:394)
> Â Â Â Â - locked <0x00002aaadfa95ec0> (a
> org.apache.hadoop.hbase.regionserver.StoreScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
> :293)
> Â Â Â Â - locked <0x00002aaadfa95ec0> (a
> org.apache.hadoop.hbase.regionserver.StoreScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
> :114)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
> HRegion.java:2344)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2301)
> Â Â Â Â - locked <0x00002aabf6832658> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2317)
> Â Â Â Â - locked <0x00002aabf6832658> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
> va:1832)
> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> Â Â Â Â at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:43)
> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
> 9)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "IPC Server handler 19 on 60020" daemon prio=10 tid=0x0000000001b19800
> nid=0x5550 runnable [0x000000004520e000]
> Â java.lang.Thread.State: RUNNABLE
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
> Â Â Â Â at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
> Â Â Â Â at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> Â Â Â Â - locked <0x00002aab94af3848> (a sun.nio.ch.Util$1)
> Â Â Â Â - locked <0x00002aab94af3830> (a
> java.util.Collections$UnmodifiableSet)
> Â Â Â Â - locked <0x00002aac6ed3e1d8> (a sun.nio.ch.EPollSelectorImpl)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
> thTimeout.java:332)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
> 157)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
> Â Â Â Â at
> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
> Â Â Â Â - locked <0x00002aaac1194578> (a java.io.BufferedInputStream)
> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
> 50)
> Â Â Â Â - locked <0x00002aad890f2600> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
> a:237)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
> Â Â Â Â - locked <0x00002aad890f2600> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
> Â Â Â Â - locked <0x00002aad890f2600> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
> a:1899)
> Â Â Â Â - locked <0x00002aaae78be750> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
> )
> Â Â Â Â - locked <0x00002aaae78be750> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
> dRangeFileInputStream.java:105)
> Â Â Â Â - locked <0x00002aabba906cc0> (a
> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
> Â Â Â Â at
> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> Â Â Â Â - locked <0x00002aac18963ef0> (a java.io.BufferedInputStream)
> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
> )
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
> Â Â Â Â - locked <0x00002aaacd2974c0> (a [B)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
> 76)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
> anner.java:115)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
> va:255)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
> va:394)
> Â Â Â Â - locked <0x00002aab328ec218> (a
> org.apache.hadoop.hbase.regionserver.StoreScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
> :293)
> Â Â Â Â - locked <0x00002aab328ec218> (a
> org.apache.hadoop.hbase.regionserver.StoreScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
> :114)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
> HRegion.java:2344)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2301)
> Â Â Â Â - locked <0x00002aabbaf1c898> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2317)
> Â Â Â Â - locked <0x00002aabbaf1c898> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
> va:1832)
> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> Â Â Â Â at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:43)
> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
> 9)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "IPC Server handler 18 on 60020" daemon prio=10 tid=0x0000000001b17800
> nid=0x554f runnable [0x000000004510d000]
> Â java.lang.Thread.State: RUNNABLE
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
> Â Â Â Â at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
> Â Â Â Â at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> Â Â Â Â - locked <0x00002aaae98101b0> (a sun.nio.ch.Util$1)
> Â Â Â Â - locked <0x00002aaae9810198> (a
> java.util.Collections$UnmodifiableSet)
> Â Â Â Â - locked <0x00002aaaeba1b040> (a sun.nio.ch.EPollSelectorImpl)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
> thTimeout.java:332)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
> 157)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
> Â Â Â Â at
> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
> Â Â Â Â - locked <0x00002aab1ac7d288> (a java.io.BufferedInputStream)
> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
> 50)
> Â Â Â Â - locked <0x00002aad890f0db8> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
> a:237)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
> Â Â Â Â - locked <0x00002aad890f0db8> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
> Â Â Â Â - locked <0x00002aad890f0db8> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
> a:1899)
> Â Â Â Â - locked <0x00002aab8c671d30> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
> )
> Â Â Â Â - locked <0x00002aab8c671d30> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
> dRangeFileInputStream.java:105)
> Â Â Â Â - locked <0x00002aab8c4f3de0> (a
> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
> Â Â Â Â at
> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> Â Â Â Â - locked <0x00002aac18963dd0> (a java.io.BufferedInputStream)
> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
> )
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
> Â Â Â Â - locked <0x00002aab8d9fd7c8> (a [B)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
> 76)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
> anner.java:115)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
> va:255)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
> va:394)
> Â Â Â Â - locked <0x00002aaba69901b8> (a
> org.apache.hadoop.hbase.regionserver.StoreScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
> :293)
> Â Â Â Â - locked <0x00002aaba69901b8> (a
> org.apache.hadoop.hbase.regionserver.StoreScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
> :114)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
> HRegion.java:2344)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2301)
> Â Â Â Â - locked <0x00002aabbaf1e468> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2317)
> Â Â Â Â - locked <0x00002aabbaf1e468> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
> va:1832)
> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> Â Â Â Â at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:43)
> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
> 9)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "IPC Server handler 17 on 60020" daemon prio=10 tid=0x0000000001b15800
> nid=0x554e runnable [0x000000004500c000]
> Â java.lang.Thread.State: RUNNABLE
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
> Â Â Â Â at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
> Â Â Â Â at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> Â Â Â Â - locked <0x00002aaac2d0d348> (a sun.nio.ch.Util$1)
> Â Â Â Â - locked <0x00002aaac2d0d330> (a
> java.util.Collections$UnmodifiableSet)
> Â Â Â Â - locked <0x00002aacbb364b18> (a sun.nio.ch.EPollSelectorImpl)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
> thTimeout.java:332)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
> 157)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
> Â Â Â Â at
> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
> Â Â Â Â - locked <0x00002aaac2750e38> (a java.io.BufferedInputStream)
> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
> 50)
> Â Â Â Â - locked <0x00002aaac1028128> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
> a:237)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
> Â Â Â Â - locked <0x00002aaac1028128> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
> Â Â Â Â - locked <0x00002aaac1028128> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
> a:1899)
> Â Â Â Â - locked <0x00002aaae41f13d8> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
> )
> Â Â Â Â - locked <0x00002aaae41f13d8> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
> dRangeFileInputStream.java:105)
> Â Â Â Â - locked <0x00002aab3c08c348> (a
> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
> Â Â Â Â at
> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> Â Â Â Â - locked <0x00002aab52d54ef8> (a java.io.BufferedInputStream)
> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
> )
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
> Â Â Â Â - locked <0x00002aac0cb261b0> (a [B)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
> 76)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
> ner.java:87)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
> :82)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
> :262)
> Â Â Â Â - locked <0x00002aaabefb9c10> (a
> org.apache.hadoop.hbase.regionserver.StoreScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
> :114)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
> HRegion.java:2344)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2301)
> Â Â Â Â - locked <0x00002aad30d56688> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2317)
> Â Â Â Â - locked <0x00002aad30d56688> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
> va:1832)
> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> Â Â Â Â at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:43)
> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
> 9)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "IPC Server handler 16 on 60020" daemon prio=10 tid=0x00000000013fa000
> nid=0x554d runnable [0x0000000044f0b000]
> Â java.lang.Thread.State: RUNNABLE
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
> Â Â Â Â at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
> Â Â Â Â at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> Â Â Â Â - locked <0x00002aabec61aa08> (a sun.nio.ch.Util$1)
> Â Â Â Â - locked <0x00002aabec61a9f0> (a
> java.util.Collections$UnmodifiableSet)
> Â Â Â Â - locked <0x00002aaae7590720> (a sun.nio.ch.EPollSelectorImpl)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
> thTimeout.java:332)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
> 157)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
> Â Â Â Â at
> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
> Â Â Â Â - locked <0x00002aacf2246cb0> (a java.io.BufferedInputStream)
> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
> 50)
> Â Â Â Â - locked <0x00002aad06dd2c70> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
> a:237)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
> Â Â Â Â - locked <0x00002aad06dd2c70> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
> Â Â Â Â - locked <0x00002aad06dd2c70> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
> a:1899)
> Â Â Â Â - locked <0x00002aabe38abcd8> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
> )
> Â Â Â Â - locked <0x00002aabe38abcd8> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
> dRangeFileInputStream.java:105)
> Â Â Â Â - locked <0x00002aaad08ca208> (a
> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
> Â Â Â Â at
> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> Â Â Â Â - locked <0x00002aac189636e0> (a java.io.BufferedInputStream)
> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
> )
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
> Â Â Â Â - locked <0x00002aab84fe43c8> (a [B)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
> 76)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(St
> oreFileScanner.java:158)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
> anner.java:110)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
> va:255)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
> va:394)
> Â Â Â Â - locked <0x00002aad072dbd30> (a
> org.apache.hadoop.hbase.regionserver.StoreScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
> :293)
> Â Â Â Â - locked <0x00002aad072dbd30> (a
> org.apache.hadoop.hbase.regionserver.StoreScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
> :114)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
> HRegion.java:2344)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2301)
> Â Â Â Â - locked <0x00002aabdbaa0490> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2317)
> Â Â Â Â - locked <0x00002aabdbaa0490> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
> va:1832)
> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> Â Â Â Â at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:43)
> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
> 9)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "IPC Server handler 15 on 60020" daemon prio=10 tid=0x00000000013f7800
> nid=0x554c runnable [0x0000000044e0a000]
> Â java.lang.Thread.State: RUNNABLE
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
> Â Â Â Â at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
> Â Â Â Â at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> Â Â Â Â - locked <0x00002aac558cde50> (a sun.nio.ch.Util$1)
> Â Â Â Â - locked <0x00002aac558cde38> (a
> java.util.Collections$UnmodifiableSet)
> Â Â Â Â - locked <0x00002aac5cd41558> (a sun.nio.ch.EPollSelectorImpl)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
> thTimeout.java:332)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
> 157)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
> Â Â Â Â at
> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
> Â Â Â Â - locked <0x00002aadcf530468> (a java.io.BufferedInputStream)
> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
> 50)
> Â Â Â Â - locked <0x00002aad427a97a8> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
> a:237)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
> Â Â Â Â - locked <0x00002aad427a97a8> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
> Â Â Â Â - locked <0x00002aad427a97a8> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
> a:1899)
> Â Â Â Â - locked <0x00002aab9410bc00> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
> )
> Â Â Â Â - locked <0x00002aab9410bc00> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
> dRangeFileInputStream.java:105)
> Â Â Â Â - locked <0x00002aaca7cfcf70> (a
> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
> Â Â Â Â at
> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> Â Â Â Â - locked <0x00002aacf952a6c8> (a java.io.BufferedInputStream)
> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
> )
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
> Â Â Â Â - locked <0x00002aaac4472350> (a [B)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
> 76)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
> ner.java:87)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
> :82)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
> :297)
> Â Â Â Â - locked <0x00002aaaefe12070> (a
> org.apache.hadoop.hbase.regionserver.StoreScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
> :114)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
> HRegion.java:2344)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2301)
> Â Â Â Â - locked <0x00002aabbaf1d1a8> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2317)
> Â Â Â Â - locked <0x00002aabbaf1d1a8> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
> va:1832)
> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> Â Â Â Â at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:43)
> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
> 9)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "IPC Server handler 14 on 60020" daemon prio=10 tid=0x00000000013f5800
> nid=0x554b runnable [0x0000000044d09000]
> Â java.lang.Thread.State: RUNNABLE
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
> Â Â Â Â at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
> Â Â Â Â at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> Â Â Â Â - locked <0x00002aab28f91848> (a sun.nio.ch.Util$1)
> Â Â Â Â - locked <0x00002aab28f91860> (a
> java.util.Collections$UnmodifiableSet)
> Â Â Â Â - locked <0x00002aaaf5031970> (a sun.nio.ch.EPollSelectorImpl)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
> thTimeout.java:332)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
> 157)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
> Â Â Â Â at
> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
> Â Â Â Â - locked <0x00002aacec8cf590> (a java.io.BufferedInputStream)
> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
> 50)
> Â Â Â Â - locked <0x00002aaaf789b048> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
> a:237)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
> Â Â Â Â - locked <0x00002aaaf789b048> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
> Â Â Â Â - locked <0x00002aaaf789b048> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
> a:1899)
> Â Â Â Â - locked <0x00002aab9c1df470> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
> )
> Â Â Â Â - locked <0x00002aab9c1df470> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
> dRangeFileInputStream.java:105)
> Â Â Â Â - locked <0x00002aab7ec60418> (a
> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
> Â Â Â Â at
> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> Â Â Â Â - locked <0x00002aab44c84dd8> (a java.io.BufferedInputStream)
> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
> )
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
> Â Â Â Â - locked <0x00002aab227a2dc8> (a [B)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
> 76)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(St
> oreFileScanner.java:158)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
> anner.java:110)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
> va:255)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
> va:394)
> Â Â Â Â - locked <0x00002aab50d25f28> (a
> org.apache.hadoop.hbase.regionserver.StoreScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
> :293)
> Â Â Â Â - locked <0x00002aab50d25f28> (a
> org.apache.hadoop.hbase.regionserver.StoreScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
> :114)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
> HRegion.java:2344)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2301)
> Â Â Â Â - locked <0x00002aaae9efd9f0> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2317)
> Â Â Â Â - locked <0x00002aaae9efd9f0> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
> va:1832)
> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> Â Â Â Â at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:43)
> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
> 9)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "IPC Server handler 13 on 60020" daemon prio=10 tid=0x0000000001b61800
> nid=0x554a waiting for monitor entry [0x0000000044c09000]
> Â java.lang.Thread.State: BLOCKED (on object monitor)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2317)
> Â Â Â Â - waiting to lock <0x00002aacc684dad0> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
> va:1832)
> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> Â Â Â Â at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:43)
> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
> 9)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "IPC Server handler 12 on 60020" daemon prio=10 tid=0x0000000001b5f800
> nid=0x5549 runnable [0x0000000044b07000]
> Â java.lang.Thread.State: RUNNABLE
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
> Â Â Â Â at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
> Â Â Â Â at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> Â Â Â Â - locked <0x00002aab618bb060> (a sun.nio.ch.Util$1)
> Â Â Â Â - locked <0x00002aabe1d82718> (a
> java.util.Collections$UnmodifiableSet)
> Â Â Â Â - locked <0x00002aab76896238> (a sun.nio.ch.EPollSelectorImpl)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
> thTimeout.java:332)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
> 157)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
> Â Â Â Â at
> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
> Â Â Â Â - locked <0x00002aab016dc128> (a java.io.BufferedInputStream)
> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
> 50)
> Â Â Â Â - locked <0x00002aad03182340> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
> a:237)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
> Â Â Â Â - locked <0x00002aad03182340> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
> Â Â Â Â - locked <0x00002aad03182340> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
> a:1899)
> Â Â Â Â - locked <0x00002aab8b32e3e0> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
> )
> Â Â Â Â - locked <0x00002aab8b32e3e0> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
> dRangeFileInputStream.java:105)
> Â Â Â Â - locked <0x00002aab8b436170> (a
> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
> Â Â Â Â at
> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> Â Â Â Â - locked <0x00002aacd4fb8038> (a java.io.BufferedInputStream)
> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
> )
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
> Â Â Â Â - locked <0x00002aab8b4b4908> (a [B)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
> 76)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
> ner.java:87)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
> :82)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
> :262)
> Â Â Â Â - locked <0x00002aab480f55b8> (a
> org.apache.hadoop.hbase.regionserver.StoreScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
> :114)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
> HRegion.java:2344)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2301)
> Â Â Â Â - locked <0x00002aaae9efdb30> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2317)
> Â Â Â Â - locked <0x00002aaae9efdb30> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
> va:1832)
> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> Â Â Â Â at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:43)
> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
> 9)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "IPC Server handler 11 on 60020" daemon prio=10 tid=0x0000000001b5d000
> nid=0x5548 waiting for monitor entry [0x0000000044a07000]
> Â java.lang.Thread.State: BLOCKED (on object monitor)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2317)
> Â Â Â Â - waiting to lock <0x00002aacc684dad0> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
> va:1832)
> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> Â Â Â Â at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:43)
> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
> 9)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "IPC Server handler 10 on 60020" daemon prio=10 tid=0x000000000196a800
> nid=0x5547 runnable [0x0000000044905000]
> Â java.lang.Thread.State: RUNNABLE
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
> Â Â Â Â at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
> Â Â Â Â at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> Â Â Â Â - locked <0x00002aad7443dcd8> (a sun.nio.ch.Util$1)
> Â Â Â Â - locked <0x00002aad7a31cb00> (a
> java.util.Collections$UnmodifiableSet)
> Â Â Â Â - locked <0x00002aac9067d2d8> (a sun.nio.ch.EPollSelectorImpl)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
> thTimeout.java:332)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
> 157)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
> Â Â Â Â at
> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
> Â Â Â Â - locked <0x00002aad5103b0a0> (a java.io.BufferedInputStream)
> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
> 50)
> Â Â Â Â - locked <0x00002aadc79059c8> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
> a:237)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
> Â Â Â Â - locked <0x00002aadc79059c8> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
> Â Â Â Â - locked <0x00002aadc79059c8> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
> a:1899)
> Â Â Â Â - locked <0x00002aabf0f472b0> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
> )
> Â Â Â Â - locked <0x00002aabf0f472b0> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
> dRangeFileInputStream.java:105)
> Â Â Â Â - locked <0x00002aab8b2b29b0> (a
> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
> Â Â Â Â at
> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> Â Â Â Â - locked <0x00002aad0477a310> (a java.io.BufferedInputStream)
> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
> )
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
> Â Â Â Â - locked <0x00002aabb7b16fd0> (a [B)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
> 76)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
> ner.java:87)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
> :82)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
> :262)
> Â Â Â Â - locked <0x00002aab59165e10> (a
> org.apache.hadoop.hbase.regionserver.StoreScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
> :114)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
> HRegion.java:2344)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2301)
> Â Â Â Â - locked <0x00002aac8e37dd58> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2317)
> Â Â Â Â - locked <0x00002aac8e37dd58> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
> va:1832)
> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> Â Â Â Â at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:43)
> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
> 9)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "IPC Server handler 9 on 60020" daemon prio=10 tid=0x0000000001968000
> nid=0x5546 waiting for monitor entry [0x0000000044805000]
> Â java.lang.Thread.State: BLOCKED (on object monitor)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.isFilterDone(
> HRegion.java:2324)
> Â Â Â Â - waiting to lock <0x00002aacc684dad0> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
> va:1850)
> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> Â Â Â Â at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:43)
> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
> 9)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "IPC Server handler 8 on 60020" daemon prio=10 tid=0x0000000001967000
> nid=0x5545 runnable [0x0000000044703000]
> Â java.lang.Thread.State: RUNNABLE
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
> Â Â Â Â at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
> Â Â Â Â at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> Â Â Â Â - locked <0x00002aabb0b7b868> (a sun.nio.ch.Util$1)
> Â Â Â Â - locked <0x00002aabb0b7b880> (a
> java.util.Collections$UnmodifiableSet)
> Â Â Â Â - locked <0x00002aaac79442c8> (a sun.nio.ch.EPollSelectorImpl)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
> thTimeout.java:332)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
> 157)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
> Â Â Â Â at
> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
> Â Â Â Â - locked <0x00002aad0106a6b0> (a java.io.BufferedInputStream)
> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
> 50)
> Â Â Â Â - locked <0x00002aac108e6768> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
> a:237)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
> Â Â Â Â - locked <0x00002aac108e6768> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
> Â Â Â Â - locked <0x00002aac108e6768> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
> a:1899)
> Â Â Â Â - locked <0x00002aac4d4ecd40> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
> )
> Â Â Â Â - locked <0x00002aac4d4ecd40> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
> dRangeFileInputStream.java:105)
> Â Â Â Â - locked <0x00002aaad20c3b08> (a
> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
> Â Â Â Â at
> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> Â Â Â Â - locked <0x00002aac536bda88> (a java.io.BufferedInputStream)
> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
> )
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
> Â Â Â Â - locked <0x00002aaadfc86320> (a [B)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.loadBlock(HFile.ja
> va:1447)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.reseekTo(HFile.jav
> a:1325)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(St
> oreFileScanner.java:152)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
> anner.java:110)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
> va:255)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
> va:394)
> Â Â Â Â - locked <0x00002aad072dc1c8> (a
> org.apache.hadoop.hbase.regionserver.StoreScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
> :293)
> Â Â Â Â - locked <0x00002aad072dc1c8> (a
> org.apache.hadoop.hbase.regionserver.StoreScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
> :114)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
> HRegion.java:2344)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2301)
> Â Â Â Â - locked <0x00002aac8e37d808> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2317)
> Â Â Â Â - locked <0x00002aac8e37d808> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
> va:1832)
> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> Â Â Â Â at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:43)
> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
> 9)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "IPC Server handler 7 on 60020" daemon prio=10 tid=0x00000000016de800
> nid=0x5544 runnable [0x0000000044602000]
> Â java.lang.Thread.State: RUNNABLE
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
> Â Â Â Â at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
> Â Â Â Â at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> Â Â Â Â - locked <0x00002aab9f4dc588> (a sun.nio.ch.Util$1)
> Â Â Â Â - locked <0x00002aab9f4dc570> (a
> java.util.Collections$UnmodifiableSet)
> Â Â Â Â - locked <0x00002aac6ed3e250> (a sun.nio.ch.EPollSelectorImpl)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
> thTimeout.java:332)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
> 157)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
> Â Â Â Â at
> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
> Â Â Â Â - locked <0x00002aad003a0178> (a java.io.BufferedInputStream)
> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
> 50)
> Â Â Â Â - locked <0x00002aabbd698ec8> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
> a:237)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
> Â Â Â Â - locked <0x00002aabbd698ec8> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
> Â Â Â Â - locked <0x00002aabbd698ec8> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
> a:1899)
> Â Â Â Â - locked <0x00002aaad91ecfa8> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
> )
> Â Â Â Â - locked <0x00002aaad91ecfa8> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
> dRangeFileInputStream.java:105)
> Â Â Â Â - locked <0x00002aac237c5090> (a
> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
> Â Â Â Â at
> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> Â Â Â Â - locked <0x00002aad65d9a7f0> (a java.io.BufferedInputStream)
> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
> )
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
> Â Â Â Â - locked <0x00002aaaccc5d6c8> (a [B)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
> 76)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
> ner.java:87)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
> :82)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
> :262)
> Â Â Â Â - locked <0x00002aad15b7d2b8> (a
> org.apache.hadoop.hbase.regionserver.StoreScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
> :114)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
> HRegion.java:2344)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2301)
> Â Â Â Â - locked <0x00002aacc684dad0> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2317)
> Â Â Â Â - locked <0x00002aacc684dad0> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
> va:1832)
> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> Â Â Â Â at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:43)
> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
> 9)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "IPC Server handler 6 on 60020" daemon prio=10 tid=0x00000000016dd000
> nid=0x5543 runnable [0x0000000044501000]
> Â java.lang.Thread.State: RUNNABLE
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
> Â Â Â Â at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
> Â Â Â Â at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> Â Â Â Â - locked <0x00002aaaf7db82f0> (a sun.nio.ch.Util$1)
> Â Â Â Â - locked <0x00002aaaf7db82d8> (a
> java.util.Collections$UnmodifiableSet)
> Â Â Â Â - locked <0x00002aab6f3339a8> (a sun.nio.ch.EPollSelectorImpl)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
> thTimeout.java:332)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
> 157)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
> Â Â Â Â at
> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
> Â Â Â Â - locked <0x00002aab494cdcb8> (a java.io.BufferedInputStream)
> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
> 50)
> Â Â Â Â - locked <0x00002aad890f5018> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
> a:237)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
> Â Â Â Â - locked <0x00002aad890f5018> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
> Â Â Â Â - locked <0x00002aad890f5018> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
> a:1899)
> Â Â Â Â - locked <0x00002aaae4d56928> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
> )
> Â Â Â Â - locked <0x00002aaae4d56928> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
> dRangeFileInputStream.java:105)
> Â Â Â Â - locked <0x00002aaae8d98798> (a
> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
> Â Â Â Â at
> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> Â Â Â Â - locked <0x00002aaaaf686f40> (a java.io.BufferedInputStream)
> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
> )
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
> Â Â Â Â - locked <0x00002aaae77f0ad8> (a [B)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
> 76)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
> ner.java:87)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
> :82)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
> :262)
> Â Â Â Â - locked <0x00002aaae4cd85a0> (a
> org.apache.hadoop.hbase.regionserver.StoreScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
> :114)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
> HRegion.java:2344)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2301)
> Â Â Â Â - locked <0x00002aad30d55d28> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2317)
> Â Â Â Â - locked <0x00002aad30d55d28> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
> va:1832)
> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> Â Â Â Â at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:43)
> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
> 9)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "IPC Server handler 5 on 60020" daemon prio=10 tid=0x000000000194c800
> nid=0x5542 runnable [0x0000000044400000]
> Â java.lang.Thread.State: RUNNABLE
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
> Â Â Â Â at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
> Â Â Â Â at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> Â Â Â Â - locked <0x00002aaac5bf0ff0> (a sun.nio.ch.Util$1)
> Â Â Â Â - locked <0x00002aaac5bf10c8> (a
> java.util.Collections$UnmodifiableSet)
> Â Â Â Â - locked <0x00002aab0492d6d8> (a sun.nio.ch.EPollSelectorImpl)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
> thTimeout.java:332)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
> 157)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
> Â Â Â Â at
> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> Â Â Â Â at
> java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> Â Â Â Â - locked <0x00002aac451e5d20> (a java.io.BufferedInputStream)
> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
> 89)
> Â Â Â Â - locked <0x00002aabf9312408> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
> a:237)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
> Â Â Â Â - locked <0x00002aabf9312408> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
> Â Â Â Â - locked <0x00002aabf9312408> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
> a:1899)
> Â Â Â Â - locked <0x00002aab78dc6068> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
> )
> Â Â Â Â - locked <0x00002aab78dc6068> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
> dRangeFileInputStream.java:105)
> Â Â Â Â - locked <0x00002aab14390138> (a
> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
> Â Â Â Â at
> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> Â Â Â Â - locked <0x00002aab7fbb83e8> (a java.io.BufferedInputStream)
> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
> )
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
> Â Â Â Â - locked <0x00002aac2cbf46c8> (a [B)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
> 76)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
> anner.java:115)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
> va:255)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
> va:394)
> Â Â Â Â - locked <0x00002aad15b7da98> (a
> org.apache.hadoop.hbase.regionserver.StoreScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
> :293)
> Â Â Â Â - locked <0x00002aad15b7da98> (a
> org.apache.hadoop.hbase.regionserver.StoreScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
> :114)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
> HRegion.java:2344)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2301)
> Â Â Â Â - locked <0x00002aabc7a3d3e8> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2317)
> Â Â Â Â - locked <0x00002aabc7a3d3e8> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
> va:1832)
> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> Â Â Â Â at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:43)
> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
> 9)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "IPC Server handler 4 on 60020" daemon prio=10 tid=0x000000000194a800
> nid=0x5541 runnable [0x00000000442ff000]
> Â java.lang.Thread.State: RUNNABLE
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
> Â Â Â Â at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
> Â Â Â Â at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> Â Â Â Â - locked <0x00002aabbae85af8> (a sun.nio.ch.Util$1)
> Â Â Â Â - locked <0x00002aabbae85b10> (a
> java.util.Collections$UnmodifiableSet)
> Â Â Â Â - locked <0x00002aaae379da00> (a sun.nio.ch.EPollSelectorImpl)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
> thTimeout.java:332)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
> 157)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
> Â Â Â Â at
> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
> Â Â Â Â - locked <0x00002aac7ecf9bb8> (a java.io.BufferedInputStream)
> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
> 50)
> Â Â Â Â - locked <0x00002aaacf49d528> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
> a:237)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
> Â Â Â Â - locked <0x00002aaacf49d528> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
> Â Â Â Â - locked <0x00002aaacf49d528> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
> a:1899)
> Â Â Â Â - locked <0x00002aabdc652620> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
> )
> Â Â Â Â - locked <0x00002aabdc652620> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
> dRangeFileInputStream.java:105)
> Â Â Â Â - locked <0x00002aac357f4350> (a
> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
> Â Â Â Â at
> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> Â Â Â Â - locked <0x00002aab9fc22a58> (a java.io.BufferedInputStream)
> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
> )
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
> Â Â Â Â - locked <0x00002aabdbefbe20> (a [B)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
> 76)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
> ner.java:87)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
> :82)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
> :262)
> Â Â Â Â - locked <0x00002aad97b3b7e8> (a
> org.apache.hadoop.hbase.regionserver.StoreScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
> :114)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
> HRegion.java:2344)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2301)
> Â Â Â Â - locked <0x00002aab016d6618> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2317)
> Â Â Â Â - locked <0x00002aab016d6618> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
> va:1832)
> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> Â Â Â Â at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:43)
> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
> 9)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "IPC Server handler 3 on 60020" daemon prio=10 tid=0x00000000019c4000
> nid=0x5540 waiting for monitor entry [0x00000000441ff000]
> Â java.lang.Thread.State: BLOCKED (on object monitor)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2317)
> Â Â Â Â - waiting to lock <0x00002aab28d7de90> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
> va:1832)
> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> Â Â Â Â at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:43)
> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
> 9)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "IPC Server handler 2 on 60020" daemon prio=10 tid=0x00000000019c2000
> nid=0x553f runnable [0x00000000440fd000]
> Â java.lang.Thread.State: RUNNABLE
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
> Â Â Â Â at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
> Â Â Â Â at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> Â Â Â Â - locked <0x00002aaaeef929f8> (a sun.nio.ch.Util$1)
> Â Â Â Â - locked <0x00002aaaeef92a10> (a
> java.util.Collections$UnmodifiableSet)
> Â Â Â Â - locked <0x00002aaaffd9ef38> (a sun.nio.ch.EPollSelectorImpl)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
> thTimeout.java:332)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
> 157)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
> Â Â Â Â at
> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
> Â Â Â Â - locked <0x00002aab2ac42300> (a java.io.BufferedInputStream)
> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
> 50)
> Â Â Â Â - locked <0x00002aabc43bef18> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
> a:237)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
> Â Â Â Â - locked <0x00002aabc43bef18> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
> Â Â Â Â - locked <0x00002aabc43bef18> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
> a:1899)
> Â Â Â Â - locked <0x00002aabd53c1d58> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
> )
> Â Â Â Â - locked <0x00002aabd53c1d58> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
> dRangeFileInputStream.java:105)
> Â Â Â Â - locked <0x00002aabb602a180> (a
> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
> Â Â Â Â at
> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> Â Â Â Â - locked <0x00002aab44c85f18> (a java.io.BufferedInputStream)
> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
> )
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
> Â Â Â Â - locked <0x00002aac0d2be0b0> (a [B)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
> 76)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
> ner.java:87)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
> :82)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
> :262)
> Â Â Â Â - locked <0x00002aab4eb7a680> (a
> org.apache.hadoop.hbase.regionserver.StoreScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
> :114)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
> HRegion.java:2344)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2301)
> Â Â Â Â - locked <0x00002aabc7a3cc68> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2317)
> Â Â Â Â - locked <0x00002aabc7a3cc68> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
> va:1832)
> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> Â Â Â Â at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:43)
> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
> 9)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "IPC Server handler 1 on 60020" daemon prio=10 tid=0x0000000001535800
> nid=0x553e runnable [0x0000000043ffc000]
> Â java.lang.Thread.State: RUNNABLE
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
> Â Â Â Â at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
> Â Â Â Â at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> Â Â Â Â - locked <0x00002aad28449198> (a sun.nio.ch.Util$1)
> Â Â Â Â - locked <0x00002aad284491b0> (a
> java.util.Collections$UnmodifiableSet)
> Â Â Â Â - locked <0x00002aab3a682700> (a sun.nio.ch.EPollSelectorImpl)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
> thTimeout.java:332)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
> 157)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
> Â Â Â Â at
> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
> Â Â Â Â - locked <0x00002aac5b5cbd28> (a java.io.BufferedInputStream)
> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
> 50)
> Â Â Â Â - locked <0x00002aabc43bd778> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
> a:237)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
> Â Â Â Â - locked <0x00002aabc43bd778> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
> Â Â Â Â - locked <0x00002aabc43bd778> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
> a:1899)
> Â Â Â Â - locked <0x00002aac4d4f9b90> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
> )
> Â Â Â Â - locked <0x00002aac4d4f9b90> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
> dRangeFileInputStream.java:105)
> Â Â Â Â - locked <0x00002aac500ac780> (a
> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
> Â Â Â Â at
> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> Â Â Â Â - locked <0x00002aabe2211268> (a java.io.BufferedInputStream)
> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
> )
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
> Â Â Â Â - locked <0x00002aac537126f0> (a [B)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
> 76)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
> ner.java:87)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
> :82)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
> :262)
> Â Â Â Â - locked <0x00002aadb912fca8> (a
> org.apache.hadoop.hbase.regionserver.StoreScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
> :114)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
> HRegion.java:2344)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2301)
> Â Â Â Â - locked <0x00002aadcba8fe98> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2317)
> Â Â Â Â - locked <0x00002aadcba8fe98> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
> va:1832)
> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> Â Â Â Â at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:43)
> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
> 9)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "IPC Server handler 0 on 60020" daemon prio=10 tid=0x0000000001534000
> nid=0x553d waiting for monitor entry [0x0000000043efc000]
> Â java.lang.Thread.State: BLOCKED (on object monitor)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
> java:2317)
> Â Â Â Â - waiting to lock <0x00002aabc7a3d258> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
> va:1832)
> Â Â Â Â at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> Â Â Â Â at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:43)
> Â Â Â Â at java.lang.reflect.Method.invoke(Method.java:616)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
> 9)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "IPC Server listener on 60020" daemon prio=10 tid=0x00002aae4054e000
> nid=0x553c runnable [0x0000000043dfb000]
> Â java.lang.Thread.State: RUNNABLE
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
> Â Â Â Â at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
> Â Â Â Â at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> Â Â Â Â - locked <0x00002aaabc18bdf8> (a sun.nio.ch.Util$1)
> Â Â Â Â - locked <0x00002aaabc18bde0> (a
> java.util.Collections$UnmodifiableSet)
> Â Â Â Â - locked <0x00002aaabbb1ced8> (a sun.nio.ch.EPollSelectorImpl)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:41
> 4)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "IPC Server Responder" daemon prio=10 tid=0x00002aae4053d800 nid=0x553b
> runnable [0x0000000043cfa000]
> Â java.lang.Thread.State: RUNNABLE
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
> Â Â Â Â at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
> Â Â Â Â at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> Â Â Â Â - locked <0x00002aaabc2ceed8> (a sun.nio.ch.Util$1)
> Â Â Â Â - locked <0x00002aaabc2ceef0> (a
> java.util.Collections$UnmodifiableSet)
> Â Â Â Â - locked <0x00002aaabbb33fa8> (a sun.nio.ch.EPollSelectorImpl)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:5
> 88)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "Timer-0" daemon prio=10 tid=0x00002aae4053c800 nid=0x553a in
> Object.wait() [0x0000000043bf9000]
> Â java.lang.Thread.State: TIMED_WAITING (on object monitor)
> Â Â Â Â at java.lang.Object.wait(Native Method)
> Â Â Â Â at java.util.TimerThread.mainLoop(Timer.java:531)
> Â Â Â Â - locked <0x00002aaabd062850> (a java.util.TaskQueue)
> Â Â Â Â at java.util.TimerThread.run(Timer.java:484)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "983053578@qtp-1891789590-1 - Acceptor0
> SelectChannelConnector@0.0.0.0:60030" prio=10 tid=0x00002aae40381000
> nid=0x5539 runnable [0x0000000043af8000]
> Â java.lang.Thread.State: RUNNABLE
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
> Â Â Â Â at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
> Â Â Â Â at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> Â Â Â Â - locked <0x00002aaabd07c460> (a sun.nio.ch.Util$1)
> Â Â Â Â - locked <0x00002aaabd07c478> (a
> java.util.Collections$UnmodifiableSet)
> Â Â Â Â - locked <0x00002aaabbb26b78> (a sun.nio.ch.EPollSelectorImpl)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> Â Â Â Â at
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.ja
> va:498)
> Â Â Â Â at
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> Â Â Â Â at
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnect
> or.java:124)
> Â Â Â Â at
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:
> 708)
> Â Â Â Â at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java
> :582)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "1148428095@qtp-1891789590-0" prio=10 tid=0x00002aae403d2000 nid=0x5538
> in Object.wait() [0x00000000439f7000]
> Â java.lang.Thread.State: TIMED_WAITING (on object monitor)
> Â Â Â Â at java.lang.Object.wait(Native Method)
> Â Â Â Â at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java
> :626)
> Â Â Â Â - locked <0x00002aaabbc4c8f0> (a
> org.mortbay.thread.QueuedThreadPool$PoolThread)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "regionserver60020.leaseChecker" daemon prio=10 tid=0x00002aae402ee800
> nid=0x5537 waiting for monitor entry [0x00000000438f6000]
> Â java.lang.Thread.State: BLOCKED (on object monitor)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.close(HRegion
> .java:2409)
> Â Â Â Â - waiting to lock <0x00002aadcba8fe98> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegionServer$ScannerListener.lease
> Expired(HRegionServer.java:1892)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.Leases.run(Leases.java:99)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "regionserver60020.majorCompactionChecker" daemon prio=10
> tid=0x00002aae402ec800 nid=0x5536 in Object.wait() [0x00000000437f5000]
> Â java.lang.Thread.State: TIMED_WAITING (on object monitor)
> Â Â Â Â at java.lang.Object.wait(Native Method)
> Â Â Â Â - waiting on <0x00002aaabc1f7bf0> (a java.lang.Object)
> Â Â Â Â at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
> Â Â Â Â - locked <0x00002aaabc1f7bf0> (a java.lang.Object)
> Â Â Â Â at org.apache.hadoop.hbase.Chore.run(Chore.java:74)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "regionserver60020.compactor" daemon prio=10 tid=0x00002aae402eb000
> nid=0x5535 runnable [0x00000000436f4000]
> Â java.lang.Thread.State: RUNNABLE
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
> Â Â Â Â at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
> Â Â Â Â at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> Â Â Â Â - locked <0x00002aab26f462e0> (a sun.nio.ch.Util$1)
> Â Â Â Â - locked <0x00002aab26f462c8> (a
> java.util.Collections$UnmodifiableSet)
> Â Â Â Â - locked <0x00002aaaeba1ac80> (a sun.nio.ch.EPollSelectorImpl)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
> thTimeout.java:332)
> Â Â Â Â at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
> 157)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
> Â Â Â Â at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
> Â Â Â Â at
> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:254)
> Â Â Â Â - locked <0x00002aab44c85258> (a java.io.BufferedInputStream)
> Â Â Â Â at java.io.DataInputStream.readInt(DataInputStream.java:387)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
> 50)
> Â Â Â Â - locked <0x00002aaac6caf140> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
> a:237)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
> Â Â Â Â at
> org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
> Â Â Â Â - locked <0x00002aaac6caf140> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
> Â Â Â Â - locked <0x00002aaac6caf140> (a
> org.apache.hadoop.hdfs.DFSClient$BlockReader)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
> a:1899)
> Â Â Â Â - locked <0x00002aaabd63fbb0> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
> )
> Â Â Â Â - locked <0x00002aaabd63fbb0> (a
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
> Â Â Â Â at java.io.DataInputStream.read(DataInputStream.java:149)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
> dRangeFileInputStream.java:105)
> Â Â Â Â - locked <0x00002aab21b4e148> (a
> org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
> Â Â Â Â at
> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
> Â Â Â Â at
> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> Â Â Â Â - locked <0x00002aaaaedc39d0> (a java.io.BufferedInputStream)
> Â Â Â Â at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
> )
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
> Â Â Â Â - locked <0x00002aab458cc280> (a [B)
> Â Â Â Â at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
> 76)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
> ner.java:87)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
> :82)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
> :297)
> Â Â Â Â - locked <0x00002aaace390ec0> (a
> org.apache.hadoop.hbase.regionserver.StoreScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
> :326)
> Â Â Â Â - locked <0x00002aaace390ec0> (a
> org.apache.hadoop.hbase.regionserver.StoreScanner)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:927)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:733)
> Â Â Â Â - locked <0x00002aaacd20e840> (a java.lang.Object)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:
> 769)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:
> 714)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplit
> Thread.java:81)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - <0x00002aaabce779f8> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>
> "regionserver60020.cacheFlusher" daemon prio=10 tid=0x00002aae402d1800
> nid=0x5534 waiting on condition [0x00000000435f3000]
> Â java.lang.Thread.State: TIMED_WAITING (parking)
> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for  <0x00002aaabc241a10> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> Â Â Â Â at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> Â Â Â Â at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
> aitNanos(AbstractQueuedSynchronizer.java:2081)
> Â Â Â Â at java.util.concurrent.DelayQueue.poll(DelayQueue.java:230)
> Â Â Â Â at java.util.concurrent.DelayQueue.poll(DelayQueue.java:68)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher
> .java:216)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "regionserver60020.logRoller" daemon prio=10 tid=0x00002aae402d1000
> nid=0x5533 in Object.wait() [0x00000000434f2000]
> Â java.lang.Thread.State: TIMED_WAITING (on object monitor)
> Â Â Â Â at java.lang.Object.wait(Native Method)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:78)
> Â Â Â Â - locked <0x00002aaabcec18e0> (a
> java.util.concurrent.atomic.AtomicBoolean)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "Timer thread for monitoring jvm" daemon prio=10 tid=0x00002aae402b4000
> nid=0x5532 in Object.wait() [0x00000000433f1000]
> Â java.lang.Thread.State: TIMED_WAITING (on object monitor)
> Â Â Â Â at java.lang.Object.wait(Native Method)
> Â Â Â Â at java.util.TimerThread.mainLoop(Timer.java:531)
> Â Â Â Â - locked <0x00002aaabcedce48> (a java.util.TaskQueue)
> Â Â Â Â at java.util.TimerThread.run(Timer.java:484)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "Timer thread for monitoring hbase" daemon prio=10
> tid=0x00002aae402b2800 nid=0x5531 in Object.wait() [0x00000000432f0000]
> Â java.lang.Thread.State: TIMED_WAITING (on object monitor)
> Â Â Â Â at java.lang.Object.wait(Native Method)
> Â Â Â Â at java.util.TimerThread.mainLoop(Timer.java:531)
> Â Â Â Â - locked <0x00002aaabceca148> (a java.util.TaskQueue)
> Â Â Â Â at java.util.TimerThread.run(Timer.java:484)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "regionserver60020.logSyncer" daemon prio=10 tid=0x00002aae402b1000
> nid=0x5530 waiting on condition [0x00000000431ef000]
> Â java.lang.Thread.State: TIMED_WAITING (sleeping)
> Â Â Â Â at java.lang.Thread.sleep(Native Method)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:96
> 3)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "LeaseChecker" daemon prio=10 tid=0x00002aae4019a800 nid=0x552f
> sleeping[0x00000000430ee000]
> Â java.lang.Thread.State: TIMED_WAITING (sleeping)
> Â Â Â Â at java.lang.Thread.sleep(Native Method)
> Â Â Â Â at
> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1167)
> Â Â Â Â at java.lang.Thread.run(Thread.java:636)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "IPC Client (47) connection to
> doop10.dt.sv4.decarta.com/10.241.8.230:60000 from an unknown user"
> daemon prio=10 tid=0x00000000015c9800 nid=0x5528 in Object.wait()
> [0x0000000042cea000]
> Â java.lang.Thread.State: TIMED_WAITING (on object monitor)
> Â Â Â Â at java.lang.Object.wait(Native Method)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.waitForWork(HBaseClie
> nt.java:431)
> Â Â Â Â - locked <0x00002aaabc24dfc0> (a
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:
> 476)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "DestroyJavaVM" prio=10 tid=0x00002aae4015c800 nid=0x54fa waiting on
> condition [0x0000000000000000]
> Â java.lang.Thread.State: RUNNABLE
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "regionserver60020-EventThread" daemon prio=10 tid=0x00000000015c5000
> nid=0x5526 waiting on condition [0x0000000042be9000]
> Â java.lang.Thread.State: WAITING (parking)
> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for  <0x00002aaabc241a60> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> Â Â Â Â at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> Â Â Â Â at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
> ait(AbstractQueuedSynchronizer.java:2043)
> Â Â Â Â at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
> 86)
> Â Â Â Â at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "regionserver60020-SendThread(doop3.dt.sv4.decarta.com:2181)" daemon
> prio=10 tid=0x0000000001486800 nid=0x5525 runnable [0x0000000042ae8000]
> Â java.lang.Thread.State: RUNNABLE
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
> Â Â Â Â at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
> Â Â Â Â at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> Â Â Â Â - locked <0x00002aaabbb3e538> (a sun.nio.ch.Util$1)
> Â Â Â Â - locked <0x00002aaabbb3e550> (a
> java.util.Collections$UnmodifiableSet)
> Â Â Â Â - locked <0x00002aaabbb1db08> (a sun.nio.ch.EPollSelectorImpl)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> Â Â Â Â at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "regionserver60020" prio=10 tid=0x00002aae40187000 nid=0x5523 waiting on
> condition [0x000000004028d000]
> Â java.lang.Thread.State: TIMED_WAITING (parking)
> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for  <0x00002aaabc2faea0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> Â Â Â Â at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> Â Â Â Â at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
> aitNanos(AbstractQueuedSynchronizer.java:2081)
> Â Â Â Â at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:4
> 23)
> Â Â Â Â at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.jav
> a:621)
> Â Â Â Â at java.lang.Thread.run(Thread.java:636)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "Timer thread for monitoring rpc" daemon prio=10 tid=0x00002aae40135800
> nid=0x5522 in Object.wait() [0x00000000429e7000]
> Â java.lang.Thread.State: TIMED_WAITING (on object monitor)
> Â Â Â Â at java.lang.Object.wait(Native Method)
> Â Â Â Â at java.util.TimerThread.mainLoop(Timer.java:531)
> Â Â Â Â - locked <0x00002aaabbb3cb08> (a java.util.TaskQueue)
> Â Â Â Â at java.util.TimerThread.run(Timer.java:484)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "IPC Reader 9 on port 60020" prio=10 tid=0x00002aae40133000 nid=0x5521
> runnable [0x00000000428e6000]
> Â java.lang.Thread.State: RUNNABLE
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
> Â Â Â Â at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
> Â Â Â Â at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> Â Â Â Â - locked <0x00002aaabbb010a8> (a sun.nio.ch.Util$1)
> Â Â Â Â - locked <0x00002aaabbb010c0> (a
> java.util.Collections$UnmodifiableSet)
> Â Â Â Â - locked <0x00002aaabbb1e198> (a sun.nio.ch.EPollSelectorImpl)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
> java:305)
> Â Â Â Â - locked <0x00002aaabbb250e8> (a
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
> a:1110)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
> va:603)
> Â Â Â Â at java.lang.Thread.run(Thread.java:636)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - <0x00002aaabbb4b008> (a
> java.util.concurrent.ThreadPoolExecutor$Worker)
>
> "IPC Reader 8 on port 60020" prio=10 tid=0x00002aae40118000 nid=0x5520
> runnable [0x00000000427e5000]
> Â java.lang.Thread.State: RUNNABLE
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
> Â Â Â Â at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
> Â Â Â Â at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> Â Â Â Â - locked <0x00002aaabbb16118> (a sun.nio.ch.Util$1)
> Â Â Â Â - locked <0x00002aaabbb16130> (a
> java.util.Collections$UnmodifiableSet)
> Â Â Â Â - locked <0x00002aaabbb144f8> (a sun.nio.ch.EPollSelectorImpl)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
> java:305)
> Â Â Â Â - locked <0x00002aaabbb30308> (a
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
> a:1110)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
> va:603)
> Â Â Â Â at java.lang.Thread.run(Thread.java:636)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - <0x00002aaabbb32888> (a
> java.util.concurrent.ThreadPoolExecutor$Worker)
>
> "IPC Reader 7 on port 60020" prio=10 tid=0x00002aae400fd000 nid=0x551f
> runnable [0x00000000426e4000]
> Â java.lang.Thread.State: RUNNABLE
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
> Â Â Â Â at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
> Â Â Â Â at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> Â Â Â Â - locked <0x00002aaabbb2a228> (a sun.nio.ch.Util$1)
> Â Â Â Â - locked <0x00002aaabbb2a240> (a
> java.util.Collections$UnmodifiableSet)
> Â Â Â Â - locked <0x00002aaabbb28ab8> (a sun.nio.ch.EPollSelectorImpl)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
> java:305)
> Â Â Â Â - locked <0x00002aaabbb2a6d8> (a
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
> a:1110)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
> va:603)
> Â Â Â Â at java.lang.Thread.run(Thread.java:636)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - <0x00002aaabbb4a068> (a
> java.util.concurrent.ThreadPoolExecutor$Worker)
>
> "IPC Reader 6 on port 60020" prio=10 tid=0x00002aae400e2800 nid=0x551e
> runnable [0x00000000425e3000]
> Â java.lang.Thread.State: RUNNABLE
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
> Â Â Â Â at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
> Â Â Â Â at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> Â Â Â Â - locked <0x00002aaabbb27028> (a sun.nio.ch.Util$1)
> Â Â Â Â - locked <0x00002aaabbb27040> (a
> java.util.Collections$UnmodifiableSet)
> Â Â Â Â - locked <0x00002aaabbb258b8> (a sun.nio.ch.EPollSelectorImpl)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
> java:305)
> Â Â Â Â - locked <0x00002aaabbb282e8> (a
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
> a:1110)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
> va:603)
> Â Â Â Â at java.lang.Thread.run(Thread.java:636)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - <0x00002aaabbb4fc98> (a
> java.util.concurrent.ThreadPoolExecutor$Worker)
>
> "IPC Reader 5 on port 60020" prio=10 tid=0x00002aae400c7800 nid=0x551d
> runnable [0x00000000413e5000]
> Â java.lang.Thread.State: RUNNABLE
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
> Â Â Â Â at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
> Â Â Â Â at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> Â Â Â Â - locked <0x00002aaabbb2ea08> (a sun.nio.ch.Util$1)
> Â Â Â Â - locked <0x00002aaabbb2ea20> (a
> java.util.Collections$UnmodifiableSet)
> Â Â Â Â - locked <0x00002aaabbb2d298> (a sun.nio.ch.EPollSelectorImpl)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
> java:305)
> Â Â Â Â - locked <0x00002aaabbb50c38> (a
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
> a:1110)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
> va:603)
> Â Â Â Â at java.lang.Thread.run(Thread.java:636)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - <0x00002aaabbb523a8> (a
> java.util.concurrent.ThreadPoolExecutor$Worker)
>
> "IPC Reader 4 on port 60020" prio=10 tid=0x00002aae400ac800 nid=0x551c
> runnable [0x00000000412e4000]
> Â java.lang.Thread.State: RUNNABLE
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
> Â Â Â Â at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
> Â Â Â Â at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> Â Â Â Â - locked <0x00002aaabbb2aea8> (a sun.nio.ch.Util$1)
> Â Â Â Â - locked <0x00002aaabbb2aec0> (a
> java.util.Collections$UnmodifiableSet)
> Â Â Â Â - locked <0x00002aaabbb1ca28> (a sun.nio.ch.EPollSelectorImpl)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
> java:305)
> Â Â Â Â - locked <0x00002aaabbb2c168> (a
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
> a:1110)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
> va:603)
> Â Â Â Â at java.lang.Thread.run(Thread.java:636)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - <0x00002aaabbb48128> (a
> java.util.concurrent.ThreadPoolExecutor$Worker)
>
> "IPC Reader 3 on port 60020" prio=10 tid=0x00002aae40091800 nid=0x551b
> runnable [0x00000000411e3000]
> Â java.lang.Thread.State: RUNNABLE
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
> Â Â Â Â at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
> Â Â Â Â at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> Â Â Â Â - locked <0x00002aaabbb15c68> (a sun.nio.ch.Util$1)
> Â Â Â Â - locked <0x00002aaabbb15c80> (a
> java.util.Collections$UnmodifiableSet)
> Â Â Â Â - locked <0x00002aaabbb12d88> (a sun.nio.ch.EPollSelectorImpl)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
> java:305)
> Â Â Â Â - locked <0x00002aaabbb165c8> (a
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
> a:1110)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
> va:603)
> Â Â Â Â at java.lang.Thread.run(Thread.java:636)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - <0x00002aaabbb30ad8> (a
> java.util.concurrent.ThreadPoolExecutor$Worker)
>
> "IPC Reader 2 on port 60020" prio=10 tid=0x00002aae40077000 nid=0x551a
> runnable [0x00000000410e2000]
> Â java.lang.Thread.State: RUNNABLE
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
> Â Â Â Â at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
> Â Â Â Â at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> Â Â Â Â - locked <0x00002aaabbb3d148> (a sun.nio.ch.Util$1)
> Â Â Â Â - locked <0x00002aaabbb3d160> (a
> java.util.Collections$UnmodifiableSet)
> Â Â Â Â - locked <0x00002aaabbb33828> (a sun.nio.ch.EPollSelectorImpl)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
> java:305)
> Â Â Â Â - locked <0x00002aaabbb3ebd8> (a
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
> a:1110)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
> va:603)
> Â Â Â Â at java.lang.Thread.run(Thread.java:636)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - <0x00002aaabbb40028> (a
> java.util.concurrent.ThreadPoolExecutor$Worker)
>
> "IPC Reader 1 on port 60020" prio=10 tid=0x00002aae4005d000 nid=0x5519
> runnable [0x0000000040fe1000]
> Â java.lang.Thread.State: RUNNABLE
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
> Â Â Â Â at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
> Â Â Â Â at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> Â Â Â Â - locked <0x00002aaabbb15c98> (a sun.nio.ch.Util$1)
> Â Â Â Â - locked <0x00002aaabbb15cb0> (a
> java.util.Collections$UnmodifiableSet)
> Â Â Â Â - locked <0x00002aaabbb12e00> (a sun.nio.ch.EPollSelectorImpl)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
> java:305)
> Â Â Â Â - locked <0x00002aaabbb165f0> (a
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
> a:1110)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
> va:603)
> Â Â Â Â at java.lang.Thread.run(Thread.java:636)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - <0x00002aaabbb30b28> (a
> java.util.concurrent.ThreadPoolExecutor$Worker)
>
> "IPC Reader 0 on port 60020" prio=10 tid=0x00002aae40003000 nid=0x5518
> runnable [0x0000000040ee0000]
> Â java.lang.Thread.State: RUNNABLE
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
> Â Â Â Â at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
> Â Â Â Â at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> Â Â Â Â - locked <0x00002aaabbb16148> (a sun.nio.ch.Util$1)
> Â Â Â Â - locked <0x00002aaabbb16160> (a
> java.util.Collections$UnmodifiableSet)
> Â Â Â Â - locked <0x00002aaabbb14570> (a sun.nio.ch.EPollSelectorImpl)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
> Â Â Â Â at
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
> java:305)
> Â Â Â Â - locked <0x00002aaabbb30330> (a
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
> a:1110)
> Â Â Â Â at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
> va:603)
> Â Â Â Â at java.lang.Thread.run(Thread.java:636)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - <0x00002aaabbb328d8> (a
> java.util.concurrent.ThreadPoolExecutor$Worker)
>
> "main-EventThread" daemon prio=10 tid=0x00002aae3c1b6000 nid=0x5513
> waiting on condition [0x00000000424e2000]
> Â java.lang.Thread.State: WAITING (parking)
> Â Â Â Â at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for  <0x00002aaabbb16618> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> Â Â Â Â at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> Â Â Â Â at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
> ait(AbstractQueuedSynchronizer.java:2043)
> Â Â Â Â at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
> 86)
> Â Â Â Â at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "main-SendThread(doop7.dt.sv4.decarta.com:2181)" daemon prio=10
> tid=0x00002aae3c11b000 nid=0x5512 runnable [0x00000000423e1000]
> Â java.lang.Thread.State: RUNNABLE
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> Â Â Â Â at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
> Â Â Â Â at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
> Â Â Â Â at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> Â Â Â Â - locked <0x00002aaabbb16178> (a sun.nio.ch.Util$1)
> Â Â Â Â - locked <0x00002aaabbb16190> (a
> java.util.Collections$UnmodifiableSet)
> Â Â Â Â - locked <0x00002aaabbb145e8> (a sun.nio.ch.EPollSelectorImpl)
> Â Â Â Â at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> Â Â Â Â at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "Low Memory Detector" daemon prio=10 tid=0x000000000126c800 nid=0x5510
> runnable [0x0000000000000000]
> Â java.lang.Thread.State: RUNNABLE
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "CompilerThread1" daemon prio=10 tid=0x000000000126a800 nid=0x550f
> waiting on condition [0x0000000000000000]
> Â java.lang.Thread.State: RUNNABLE
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "CompilerThread0" daemon prio=10 tid=0x0000000001266000 nid=0x550e
> waiting on condition [0x0000000000000000]
> Â java.lang.Thread.State: RUNNABLE
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "Signal Dispatcher" daemon prio=10 tid=0x0000000001264000 nid=0x550d
> runnable [0x0000000000000000]
> Â java.lang.Thread.State: RUNNABLE
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "Surrogate Locker Thread (CMS)" daemon prio=10 tid=0x0000000001262000
> nid=0x550c waiting on condition [0x0000000000000000]
> Â java.lang.Thread.State: RUNNABLE
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "Finalizer" daemon prio=10 tid=0x000000000123b800 nid=0x550b in
> Object.wait() [0x00000000422e0000]
> Â java.lang.Thread.State: WAITING (on object monitor)
> Â Â Â Â at java.lang.Object.wait(Native Method)
> Â Â Â Â at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:133)
> Â Â Â Â - locked <0x00002aaabbb15cf8> (a
> java.lang.ref.ReferenceQueue$Lock)
> Â Â Â Â at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:149)
> Â Â Â Â at
> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "Reference Handler" daemon prio=10 tid=0x0000000001239800 nid=0x550a in
> Object.wait() [0x00000000421df000]
> Â java.lang.Thread.State: WAITING (on object monitor)
> Â Â Â Â at java.lang.Object.wait(Native Method)
> Â Â Â Â at java.lang.Object.wait(Object.java:502)
> Â Â Â Â at
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
> Â Â Â Â - locked <0x00002aaabbb16d98> (a java.lang.ref.Reference$Lock)
>
> Â Locked ownable synchronizers:
> Â Â Â Â - None
>
> "VM Thread" prio=10 tid=0x0000000001234800 nid=0x5509 runnable
>
> "Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x0000000000fe9000
> nid=0x54fb runnable
>
> "Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x0000000000feb000
> nid=0x54fc runnable
>
> "Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x0000000000fec800
> nid=0x54fd runnable
>
> "Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x0000000000fee800
> nid=0x54fe runnable
>
> "Gang worker#4 (Parallel GC Threads)" prio=10 tid=0x0000000000ff0800
> nid=0x54ff runnable
>
> "Gang worker#5 (Parallel GC Threads)" prio=10 tid=0x0000000000ff2000
> nid=0x5500 runnable
>
> "Gang worker#6 (Parallel GC Threads)" prio=10 tid=0x0000000000ff4000
> nid=0x5501 runnable
>
> "Gang worker#7 (Parallel GC Threads)" prio=10 tid=0x0000000000ff6000
> nid=0x5502 runnable
>
> "Gang worker#8 (Parallel GC Threads)" prio=10 tid=0x0000000000ff7800
> nid=0x5503 runnable
>
> "Gang worker#9 (Parallel GC Threads)" prio=10 tid=0x0000000000ff9800
> nid=0x5504 runnable
>
> "Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00000000010f4000
> nid=0x5508 runnable
> "Gang worker#0 (Parallel CMS Threads)" prio=10 tid=0x00000000010ee000
> nid=0x5505 runnable
>
> "Gang worker#1 (Parallel CMS Threads)" prio=10 tid=0x00000000010f0000
> nid=0x5506 runnable
>
> "Gang worker#2 (Parallel CMS Threads)" prio=10 tid=0x00000000010f2000
> nid=0x5507 runnable
>
> "VM Periodic Task Thread" prio=10 tid=0x000000000126f800 nid=0x5511
> waiting on condition
>
> JNI global references: 906
>
>
> -----Original Message-----
> From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of
> Stack
> Sent: Sunday, September 11, 2011 2:27 PM
> To: user@hbase.apache.org
> Subject: Re: scanner deadlock?
>
> 0.90.4 fixes two deadlocks (HBASE-4101 and HBASE-4077). Â Since then,
> there is HBASE-4367 (Which has a posted patch).
>
> Below sounds like slowness. Â Can you thread dump the particular
> regionserver and see what its up to? Â Is there other loading on the
> system at the time? Â For example, loading on hdfs? Â Anything in the
> hdfs logs for the datanode running beside the slow regionserver?
>
> St.Ack
>
> On Sat, Sep 10, 2011 at 5:50 PM, Geoff Hendrey <gh...@decarta.com>
> wrote:
>> Hi all -
>>
>>
>>
>> I'm still dealing with the saga of ScannerTimeoutException,
>> UnknownScannerException, etc. I rewrote my code, in the hope that
> simply
>> a different approach and some different code paths might yield better
>> results. No change. I tried many variations (caching 1 row vs caching
>> many rows, changing the regionserver's lease, increasing the number of
>> allowed zookeeper connections, etc). I created a fresh table on the
>> thought that maybe there was some problem with the table...no change.
>>
>>
>>
>> I am dealing with what appears to be some sort of scanner deadlock. I
>> have a total order partitioned mapreduce job. In the reducer, as long
> as
>> I use just one reducer, the task finishes quickly. But as soon as more
>> than one reducer opens a scanner, the tasks procede in what I would
> call
>> a "jittery" lock step. They both are able to do ResultScanner.next() a
>> few times, but then the call to next() freezes for a long period, and
>> ScannerTimeoutException. I catch the exception, and get a new
>> ResultScanner, and the pattern repeats. Jittery lockstep consisting of
>> being able to get a few successful next() calls, then a lock up.
>>
>>
>>
>> I have not yet tried upgrading from 90.1 to higher. Nor have I tried
>> tsuna's async client. Can anyone think of anything else I can try to
>> resolve this? I've sunk quite a few late nights into this, and would
> be
>> very excited to find a solution.
>>
>>
>>
>> -geoff
>>
>>
>
RE: scanner deadlock?
Posted by Geoff Hendrey <gh...@decarta.com>.
This is probably superfluous info, but when a regionserver begins
throwing ClosedChannelException, the client of course barfs as well, and
it's game over (as shown below by errors reported by client). If only
the master would realize that the regionserver is unavailable, but in
this case it seems the region server is only *effectively* unavailable,
so does not get taken out by the master.:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
contact region server doop5.dt.sv4.decarta.com:60020 for region
<REDACTED>, but failed after 10 attempts.
Exceptions:
java.io.IOException: java.net.SocketTimeoutException: Call to
doop5.dt.sv4.decarta.com/10.241.8.225:60020 failed on socket timeout
exception: java.net.SocketTimeoutException: 60000 millis timeout while
waiting for channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/10.241.8.222:34050
remote=doop5.dt.sv4.decarta.com/10.241.8.225:60020]
java.io.IOException: java.net.SocketTimeoutException: Call to
doop5.dt.sv4.decarta.com/10.241.8.225:60020 failed on socket timeout
exception: java.net.SocketTimeoutException: 60000 millis timeout while
waiting for channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/10.241.8.222:52609
remote=doop5.dt.sv4.decarta.com/10.241.8.225:60020]
java.io.IOException: java.net.SocketTimeoutException: Call to
doop5.dt.sv4.decarta.com/10.241.8.225:60020 failed on socket timeout
exception: java.net.SocketTimeoutException: 60000 millis timeout while
waiting for channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/10.241.8.222:52782
remote=doop5.dt.sv4.decarta.com/10.241.8.225:60020]
java.io.IOException: java.net.SocketTimeoutException: Call to
doop5.dt.sv4.decarta.com/10.241.8.225:60020 failed on socket timeout
exception: java.net.SocketTimeoutException: 60000 millis timeout while
waiting for channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/10.241.8.222:52930
remote=doop5.dt.sv4.decarta.com/10.241.8.225:60020]
java.io.IOException: java.net.SocketTimeoutException: Call to
doop5.dt.sv4.decarta.com/10.241.8.225:60020 failed on socket timeout
exception: java.net.SocketTimeoutException: 60000 millis timeout while
waiting for channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/10.241.8.222:53043
remote=doop5.dt.sv4.decarta.com/10.241.8.225:60020]
java.io.IOException: java.net.SocketTimeoutException: Call to
doop5.dt.sv4.decarta.com/10.241.8.225:60020 failed on socket timeout
exception: java.net.SocketTimeoutException: 60000 millis timeout while
waiting for channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/10.241.8.222:33404
remote=doop5.dt.sv4.decarta.com/10.241.8.225:60020]
java.io.IOException: java.net.SocketTimeoutException: Call to
doop5.dt.sv4.decarta.com/10.241.8.225:60020 failed on socket timeout
exception: java.net.SocketTimeoutException: 60000 millis timeout while
waiting for channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/10.241.8.222:33472
remote=doop5.dt.sv4.decarta.com/10.241.8.225:60020]
java.io.IOException: java.net.SocketTimeoutException: Call to
doop5.dt.sv4.decarta.com/10.241.8.225:60020 failed on socket timeout
exception: java.net.SocketTimeoutException: 60000 millis timeout while
waiting for channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/10.241.8.222:33540
remote=doop5.dt.sv4.decarta.com/10.241.8.225:60020]
java.io.IOException: java.net.SocketTimeoutException: Call to
doop5.dt.sv4.decarta.com/10.241.8.225:60020 failed on socket timeout
exception: java.net.SocketTimeoutException: 60000 millis timeout while
waiting for channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/10.241.8.222:33589
remote=doop5.dt.sv4.decarta.com/10.241.8.225:60020]
java.io.IOException: java.net.SocketTimeoutException: Call to
doop5.dt.sv4.decarta.com/10.241.8.225:60020 failed on socket timeout
exception: java.net.SocketTimeoutException: 60000 millis timeout while
waiting for channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/10.241.8.222:33640
remote=doop5.dt.sv4.decarta.com/10.241.8.225:60020]
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementat
ion.getRegionServerWithRetries(HConnectionManager.java:1008)
at
org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:110
0)
-----Original Message-----
From: Geoff Hendrey [mailto:ghendrey@decarta.com]
Sent: Sunday, September 11, 2011 2:40 PM
To: user@hbase.apache.org
Cc: Rohit Nigam; Tony Wang; Parmod Mehta; James Ladd
Subject: RE: scanner deadlock?
Hi,
The cluster is definitely loaded. We are running a MR job with 130
reducers, each using a scanner. Ganglia shows an overall CPU utilization
of around 40%, with network pushing 800MB/s. The problem may correlate
with a major_compact I initiated couple days ago. Even today, I still
see a lot of compacting (continuous) on region servers.
Here is the thread dump from jstack. It does not reveal any explicit
deadlock. At the time I dumped the jstack, the region server logs are
reporting "IPC Server handler...ClosedChannelException". As soon as this
begins occurring, our network throughput, as seen on Ganglia drops from
800MB/s down to virtually zero. Then of course there is the cascade of
failures, eventually leading to the regionservers shutting down.
I don't see anything unusual in the datanode logs on the same server.
Just HDFS_READ and HDFS_WRITE for the most part.
jstack here:
===================================================================
2011-09-11 14:28:52
Full thread dump OpenJDK 64-Bit Server VM (14.0-b16 mixed mode):
"IPC Client (47) connection to
doop10.dt.sv4.decarta.com/10.241.8.230:54310 from hroot" daemon prio=10
tid=0x0000000001492800 nid=0x75cb in Object.wait() [0x0000000047633000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:676)
- locked <0x00002aaaaf680000> (a
org.apache.hadoop.ipc.Client$Connection)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:719)
Locked ownable synchronizers:
- None
"ResponseProcessor for block blk_2613854253703670198_5032689" daemon
prio=10 tid=0x0000000001988800 nid=0x75bb runnable [0x0000000046926000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aab69337248> (a sun.nio.ch.Util$1)
- locked <0x00002aab69337230> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaaf3897930> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.DataInputStream.readFully(DataInputStream.java:195)
at java.io.DataInputStream.readLong(DataInputStream.java:416)
at
org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFie
lds(DataTransferProtocol.java:120)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(D
FSClient.java:2638)
Locked ownable synchronizers:
- None
"DataStreamer for file
/hbase-20.6/NAM_CLUSTERKEYS3/60db9e0113c7578ed913dc37e0856954/.tmp/42977
03314038441248 block blk_2613854253703670198_5032689" daemon prio=10
tid=0x00002aae40e8e800 nid=0x75b8 in Object.wait() [0x0000000046e2b000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSCli
ent.java:2483)
- locked <0x00002aab6551a2d8> (a java.util.LinkedList)
Locked ownable synchronizers:
- None
"ResponseProcessor for block blk_8900384789253382699_5032549" daemon
prio=10 tid=0x00000000013cd000 nid=0x71d0 runnable [0x0000000047431000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aaae0569530> (a sun.nio.ch.Util$1)
- locked <0x00002aaae0569518> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaadb4c5c68> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.DataInputStream.readFully(DataInputStream.java:195)
at java.io.DataInputStream.readLong(DataInputStream.java:416)
at
org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFie
lds(DataTransferProtocol.java:120)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(D
FSClient.java:2638)
Locked ownable synchronizers:
- None
"DataStreamer for file
/hbase-20.6/.logs/doop5.dt.sv4.decarta.com,60020,1315768059645/doop5.dt.
sv4.decarta.com%3A60020.1315776307423 block
blk_8900384789253382699_5032549" daemon prio=10 tid=0x00000000016c3800
nid=0x71c8 in Object.wait() [0x000000004712e000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSCli
ent.java:2483)
- locked <0x00002aab092a7f38> (a java.util.LinkedList)
Locked ownable synchronizers:
- None
"Attach Listener" daemon prio=10 tid=0x0000000001e32000 nid=0x50f4
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"RS_CLOSE_REGION-doop5.dt.sv4.decarta.com,60020,1315768059645-2" daemon
prio=10 tid=0x00000000018cb800 nid=0x505 waiting on condition
[0x0000000047330000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabcefb428> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
ait(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
86)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
1043)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
a:1103)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:603)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"RS_CLOSE_REGION-doop5.dt.sv4.decarta.com,60020,1315768059645-1" daemon
prio=10 tid=0x000000000163e000 nid=0x504 waiting on condition
[0x0000000047532000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabcefb428> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
ait(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
86)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
1043)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
a:1103)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:603)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"RS_CLOSE_REGION-doop5.dt.sv4.decarta.com,60020,1315768059645-0" daemon
prio=10 tid=0x0000000001bff800 nid=0x503 waiting on condition
[0x0000000046f2c000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabcefb428> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
ait(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
86)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
1043)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
a:1103)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:603)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"sendParams-6" daemon prio=10 tid=0x00002aae401f1800 nid=0x6d3b waiting
on condition [0x000000004702d000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabceea260> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Synchro
nousQueue.java:453)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(Synchronous
Queue.java:352)
at
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:903)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
1043)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
a:1103)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:603)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"RS_OPEN_REGION-doop5.dt.sv4.decarta.com,60020,1315768059645-2" daemon
prio=10 tid=0x0000000001a7f000 nid=0x5b56 waiting on condition
[0x0000000046d2a000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabcefb2e8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
ait(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
86)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
1043)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
a:1103)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:603)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"RS_OPEN_REGION-doop5.dt.sv4.decarta.com,60020,1315768059645-1" daemon
prio=10 tid=0x0000000001af2800 nid=0x5b55 waiting on condition
[0x0000000042deb000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabcefb2e8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
ait(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
86)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
1043)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
a:1103)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:603)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"RS_OPEN_REGION-doop5.dt.sv4.decarta.com,60020,1315768059645-0" daemon
prio=10 tid=0x00000000014b3800 nid=0x5b54 waiting on condition
[0x0000000046c29000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabcefb2e8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
ait(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
86)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
1043)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
a:1103)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:603)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"RS_OPEN_ROOT-doop5.dt.sv4.decarta.com,60020,1315768059645-0" daemon
prio=10 tid=0x0000000001ad2800 nid=0x556e waiting on condition
[0x0000000046a27000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabcee8870> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
ait(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
86)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
1043)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
a:1103)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:603)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"LRU Statistics #0" prio=10 tid=0x00002aae4048a800 nid=0x5566 waiting on
condition [0x0000000046825000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabd0695c0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
aitNanos(AbstractQueuedSynchronizer.java:2081)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:193)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(S
cheduledThreadPoolExecutor.java:688)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(S
cheduledThreadPoolExecutor.java:681)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
1043)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
a:1103)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:603)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"LruBlockCache.EvictionThread" daemon prio=10 tid=0x00002aae4049f800
nid=0x5565 in Object.wait() [0x0000000046724000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at
org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlo
ckCache.java:519)
- locked <0x00002aaabce92f50> (a
org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread)
Locked ownable synchronizers:
- None
"PRI IPC Server handler 9 on 60020" daemon prio=10
tid=0x00002aae4057e000 nid=0x5564 waiting on condition
[0x0000000046623000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabbb307e0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
ait(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
86)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
5)
Locked ownable synchronizers:
- None
"PRI IPC Server handler 8 on 60020" daemon prio=10
tid=0x00002aae4057c800 nid=0x5563 waiting on condition
[0x0000000046522000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabbb307e0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
ait(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
86)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
5)
Locked ownable synchronizers:
- None
"PRI IPC Server handler 7 on 60020" daemon prio=10
tid=0x00002aae40559000 nid=0x5562 waiting on condition
[0x0000000046421000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabbb307e0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
ait(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
86)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
5)
Locked ownable synchronizers:
- None
"PRI IPC Server handler 6 on 60020" daemon prio=10
tid=0x00002aae40557800 nid=0x5561 waiting on condition
[0x0000000046320000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabbb307e0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
ait(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
86)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
5)
Locked ownable synchronizers:
- None
"PRI IPC Server handler 5 on 60020" daemon prio=10
tid=0x00002aae40555800 nid=0x5560 waiting on condition
[0x000000004621f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabbb307e0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
ait(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
86)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
5)
Locked ownable synchronizers:
- None
"PRI IPC Server handler 4 on 60020" daemon prio=10
tid=0x00002aae40553800 nid=0x555f waiting on condition
[0x000000004611e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabbb307e0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
ait(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
86)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
5)
Locked ownable synchronizers:
- None
"PRI IPC Server handler 3 on 60020" daemon prio=10
tid=0x00002aae4054f000 nid=0x555e waiting on condition
[0x000000004601d000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabbb307e0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
ait(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
86)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
5)
Locked ownable synchronizers:
- None
"PRI IPC Server handler 2 on 60020" daemon prio=10
tid=0x00000000012f8800 nid=0x555d waiting on condition
[0x0000000045f1c000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabbb307e0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
ait(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
86)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
5)
Locked ownable synchronizers:
- None
"PRI IPC Server handler 1 on 60020" daemon prio=10
tid=0x0000000001991000 nid=0x555c waiting on condition
[0x0000000045e1b000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabbb307e0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
ait(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
86)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
5)
Locked ownable synchronizers:
- None
"PRI IPC Server handler 0 on 60020" daemon prio=10
tid=0x000000000198e800 nid=0x555b waiting on condition
[0x0000000045d1a000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabbb307e0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
ait(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
86)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
5)
Locked ownable synchronizers:
- None
"IPC Server handler 29 on 60020" daemon prio=10 tid=0x000000000198c800
nid=0x555a runnable [0x0000000045c18000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aac053b8450> (a sun.nio.ch.Util$1)
- locked <0x00002aac053b8438> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aab1037afc8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aac92976a30> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aabc43be0a8> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aabc43be0a8> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aabc43be0a8> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aac1a4e0ea8> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aac1a4e0ea8> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aac237c37a8> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aad39485080> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aab0b5ba8a8> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
ner.java:87)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:82)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:262)
- locked <0x00002aab89f59aa8> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aab016d6988> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aab016d6988> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 28 on 60020" daemon prio=10 tid=0x0000000001864000
nid=0x5559 runnable [0x0000000045b17000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aabaedb0608> (a sun.nio.ch.Util$1)
- locked <0x00002aabaedb05f0> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaca096fa50> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aac62a70b28> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aab856e6220> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aab856e6220> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aab856e6220> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aab0264afe8> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aab0264afe8> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aaae070be48> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aac18961940> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aabad8605d0> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
ner.java:87)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:82)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:262)
- locked <0x00002aab077e5d80> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aad30d57c18> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aad30d57c18> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 27 on 60020" daemon prio=10 tid=0x0000000001862000
nid=0x5558 runnable [0x0000000045a16000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aab2ea61b70> (a sun.nio.ch.Util$1)
- locked <0x00002aab2ea61b88> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aacbb364578> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aab1ffc0d00> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aad890f4640> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aad890f4640> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aad890f4640> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aaba1282b10> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aaba1282b10> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aac528d15d0> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aaaaf68a630> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aabc5c948b8> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
anner.java:115)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
va:255)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
va:394)
- locked <0x00002aab95832718> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:293)
- locked <0x00002aab95832718> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aab28d7de90> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aab28d7de90> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 26 on 60020" daemon prio=10 tid=0x0000000001860000
nid=0x5557 runnable [0x0000000045915000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aabaaa6b808> (a sun.nio.ch.Util$1)
- locked <0x00002aabaaa6b7f0> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaaf3899ca8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aaac1f59118> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aaac0e99db0> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aaac0e99db0> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aaac0e99db0> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aabb7221e88> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aabb7221e88> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aac28621da0> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aac18962690> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aab6876b850> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.loadBlock(HFile.ja
va:1447)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.reseekTo(HFile.jav
a:1325)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(St
oreFileScanner.java:152)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
anner.java:110)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
va:255)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
va:394)
- locked <0x00002aaabe3468f8> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:293)
- locked <0x00002aaabe3468f8> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aac8e37dbc8> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aac8e37dbc8> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 25 on 60020" daemon prio=10 tid=0x000000000173d800
nid=0x5556 runnable [0x0000000045814000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aab20178510> (a sun.nio.ch.Util$1)
- locked <0x00002aab20178528> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaad6187af0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aac891a6610> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aac1de6d9f8> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aac1de6d9f8> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aac1de6d9f8> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aaac041c6c8> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aaac041c6c8> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aab25e717e0> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aac3e6e4e28> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aaad4a425f0> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
ner.java:87)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:82)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:262)
- locked <0x00002aaabd5eb7e0> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aac8e37dc68> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aac8e37dc68> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 24 on 60020" daemon prio=10 tid=0x000000000173a000
nid=0x5555 waiting for monitor entry [0x0000000045714000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- waiting to lock <0x00002aab016d6618> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 23 on 60020" daemon prio=10 tid=0x0000000001444000
nid=0x5554 runnable [0x0000000045612000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aad70044d30> (a sun.nio.ch.Util$1)
- locked <0x00002aad70044d48> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aab328f1a40> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aaac9d48e68> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aac1de6dd40> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aac1de6dd40> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aac1de6dd40> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aab39d88148> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aab39d88148> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aab7addd0d8> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aab2811e8e8> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aabe56e8858> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
ner.java:87)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:82)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:262)
- locked <0x00002aad15b7d360> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aabc7a3d258> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aabc7a3d258> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 22 on 60020" daemon prio=10 tid=0x0000000001442000
nid=0x5553 runnable [0x0000000045511000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aabf6e38450> (a sun.nio.ch.Util$1)
- locked <0x00002aabf6e38468> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aabb9e99468> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aaae13c0340> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aad427a91c0> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aad427a91c0> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aad427a91c0> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aab7c7ca0c8> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aab7c7ca0c8> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aab3748f1a8> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aab52d56ed8> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aab60a86ca8> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(St
oreFileScanner.java:158)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
anner.java:110)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
va:255)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
va:394)
- locked <0x00002aab706a3e98> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:293)
- locked <0x00002aab706a3e98> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aaae3489758> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aaae3489758> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 21 on 60020" daemon prio=10 tid=0x0000000001792000
nid=0x5552 waiting for monitor entry [0x0000000045411000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- waiting to lock <0x00002aabc7a3d3e8> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 20 on 60020" daemon prio=10 tid=0x000000000178e000
nid=0x5551 runnable [0x000000004530f000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aab3745e018> (a sun.nio.ch.Util$1)
- locked <0x00002aab3745e030> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aab7d5cc0c0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aac035e4990> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aad890f42f8> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aad890f42f8> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aad890f42f8> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aab1baffa50> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aab1baffa50> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aaadef99fe8> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aac189625a0> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aabb88c7ce0> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
anner.java:115)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
va:255)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
va:394)
- locked <0x00002aaadfa95ec0> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:293)
- locked <0x00002aaadfa95ec0> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aabf6832658> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aabf6832658> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 19 on 60020" daemon prio=10 tid=0x0000000001b19800
nid=0x5550 runnable [0x000000004520e000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aab94af3848> (a sun.nio.ch.Util$1)
- locked <0x00002aab94af3830> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aac6ed3e1d8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aaac1194578> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aad890f2600> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aad890f2600> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aad890f2600> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aaae78be750> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aaae78be750> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aabba906cc0> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aac18963ef0> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aaacd2974c0> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
anner.java:115)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
va:255)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
va:394)
- locked <0x00002aab328ec218> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:293)
- locked <0x00002aab328ec218> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aabbaf1c898> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aabbaf1c898> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 18 on 60020" daemon prio=10 tid=0x0000000001b17800
nid=0x554f runnable [0x000000004510d000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aaae98101b0> (a sun.nio.ch.Util$1)
- locked <0x00002aaae9810198> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaaeba1b040> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aab1ac7d288> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aad890f0db8> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aad890f0db8> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aad890f0db8> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aab8c671d30> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aab8c671d30> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aab8c4f3de0> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aac18963dd0> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aab8d9fd7c8> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
anner.java:115)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
va:255)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
va:394)
- locked <0x00002aaba69901b8> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:293)
- locked <0x00002aaba69901b8> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aabbaf1e468> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aabbaf1e468> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 17 on 60020" daemon prio=10 tid=0x0000000001b15800
nid=0x554e runnable [0x000000004500c000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aaac2d0d348> (a sun.nio.ch.Util$1)
- locked <0x00002aaac2d0d330> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aacbb364b18> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aaac2750e38> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aaac1028128> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aaac1028128> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aaac1028128> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aaae41f13d8> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aaae41f13d8> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aab3c08c348> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aab52d54ef8> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aac0cb261b0> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
ner.java:87)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:82)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:262)
- locked <0x00002aaabefb9c10> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aad30d56688> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aad30d56688> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 16 on 60020" daemon prio=10 tid=0x00000000013fa000
nid=0x554d runnable [0x0000000044f0b000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aabec61aa08> (a sun.nio.ch.Util$1)
- locked <0x00002aabec61a9f0> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaae7590720> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aacf2246cb0> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aad06dd2c70> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aad06dd2c70> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aad06dd2c70> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aabe38abcd8> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aabe38abcd8> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aaad08ca208> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aac189636e0> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aab84fe43c8> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(St
oreFileScanner.java:158)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
anner.java:110)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
va:255)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
va:394)
- locked <0x00002aad072dbd30> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:293)
- locked <0x00002aad072dbd30> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aabdbaa0490> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aabdbaa0490> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 15 on 60020" daemon prio=10 tid=0x00000000013f7800
nid=0x554c runnable [0x0000000044e0a000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aac558cde50> (a sun.nio.ch.Util$1)
- locked <0x00002aac558cde38> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aac5cd41558> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aadcf530468> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aad427a97a8> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aad427a97a8> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aad427a97a8> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aab9410bc00> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aab9410bc00> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aaca7cfcf70> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aacf952a6c8> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aaac4472350> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
ner.java:87)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:82)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:297)
- locked <0x00002aaaefe12070> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aabbaf1d1a8> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aabbaf1d1a8> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 14 on 60020" daemon prio=10 tid=0x00000000013f5800
nid=0x554b runnable [0x0000000044d09000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aab28f91848> (a sun.nio.ch.Util$1)
- locked <0x00002aab28f91860> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaaf5031970> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aacec8cf590> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aaaf789b048> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aaaf789b048> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aaaf789b048> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aab9c1df470> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aab9c1df470> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aab7ec60418> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aab44c84dd8> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aab227a2dc8> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(St
oreFileScanner.java:158)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
anner.java:110)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
va:255)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
va:394)
- locked <0x00002aab50d25f28> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:293)
- locked <0x00002aab50d25f28> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aaae9efd9f0> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aaae9efd9f0> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 13 on 60020" daemon prio=10 tid=0x0000000001b61800
nid=0x554a waiting for monitor entry [0x0000000044c09000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- waiting to lock <0x00002aacc684dad0> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 12 on 60020" daemon prio=10 tid=0x0000000001b5f800
nid=0x5549 runnable [0x0000000044b07000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aab618bb060> (a sun.nio.ch.Util$1)
- locked <0x00002aabe1d82718> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aab76896238> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aab016dc128> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aad03182340> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aad03182340> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aad03182340> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aab8b32e3e0> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aab8b32e3e0> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aab8b436170> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aacd4fb8038> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aab8b4b4908> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
ner.java:87)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:82)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:262)
- locked <0x00002aab480f55b8> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aaae9efdb30> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aaae9efdb30> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 11 on 60020" daemon prio=10 tid=0x0000000001b5d000
nid=0x5548 waiting for monitor entry [0x0000000044a07000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- waiting to lock <0x00002aacc684dad0> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 10 on 60020" daemon prio=10 tid=0x000000000196a800
nid=0x5547 runnable [0x0000000044905000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aad7443dcd8> (a sun.nio.ch.Util$1)
- locked <0x00002aad7a31cb00> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aac9067d2d8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aad5103b0a0> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aadc79059c8> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aadc79059c8> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aadc79059c8> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aabf0f472b0> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aabf0f472b0> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aab8b2b29b0> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aad0477a310> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aabb7b16fd0> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
ner.java:87)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:82)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:262)
- locked <0x00002aab59165e10> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aac8e37dd58> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aac8e37dd58> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 9 on 60020" daemon prio=10 tid=0x0000000001968000
nid=0x5546 waiting for monitor entry [0x0000000044805000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.isFilterDone(
HRegion.java:2324)
- waiting to lock <0x00002aacc684dad0> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1850)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 8 on 60020" daemon prio=10 tid=0x0000000001967000
nid=0x5545 runnable [0x0000000044703000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aabb0b7b868> (a sun.nio.ch.Util$1)
- locked <0x00002aabb0b7b880> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaac79442c8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aad0106a6b0> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aac108e6768> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aac108e6768> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aac108e6768> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aac4d4ecd40> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aac4d4ecd40> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aaad20c3b08> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aac536bda88> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aaadfc86320> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.loadBlock(HFile.ja
va:1447)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.reseekTo(HFile.jav
a:1325)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(St
oreFileScanner.java:152)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
anner.java:110)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
va:255)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
va:394)
- locked <0x00002aad072dc1c8> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:293)
- locked <0x00002aad072dc1c8> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aac8e37d808> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aac8e37d808> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 7 on 60020" daemon prio=10 tid=0x00000000016de800
nid=0x5544 runnable [0x0000000044602000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aab9f4dc588> (a sun.nio.ch.Util$1)
- locked <0x00002aab9f4dc570> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aac6ed3e250> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aad003a0178> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aabbd698ec8> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aabbd698ec8> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aabbd698ec8> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aaad91ecfa8> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aaad91ecfa8> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aac237c5090> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aad65d9a7f0> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aaaccc5d6c8> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
ner.java:87)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:82)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:262)
- locked <0x00002aad15b7d2b8> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aacc684dad0> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aacc684dad0> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 6 on 60020" daemon prio=10 tid=0x00000000016dd000
nid=0x5543 runnable [0x0000000044501000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aaaf7db82f0> (a sun.nio.ch.Util$1)
- locked <0x00002aaaf7db82d8> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aab6f3339a8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aab494cdcb8> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aad890f5018> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aad890f5018> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aad890f5018> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aaae4d56928> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aaae4d56928> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aaae8d98798> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aaaaf686f40> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aaae77f0ad8> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
ner.java:87)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:82)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:262)
- locked <0x00002aaae4cd85a0> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aad30d55d28> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aad30d55d28> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 5 on 60020" daemon prio=10 tid=0x000000000194c800
nid=0x5542 runnable [0x0000000044400000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aaac5bf0ff0> (a sun.nio.ch.Util$1)
- locked <0x00002aaac5bf10c8> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aab0492d6d8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aac451e5d20> (a java.io.BufferedInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
89)
- locked <0x00002aabf9312408> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aabf9312408> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aabf9312408> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aab78dc6068> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aab78dc6068> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aab14390138> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aab7fbb83e8> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aac2cbf46c8> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
anner.java:115)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
va:255)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
va:394)
- locked <0x00002aad15b7da98> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:293)
- locked <0x00002aad15b7da98> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aabc7a3d3e8> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aabc7a3d3e8> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 4 on 60020" daemon prio=10 tid=0x000000000194a800
nid=0x5541 runnable [0x00000000442ff000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aabbae85af8> (a sun.nio.ch.Util$1)
- locked <0x00002aabbae85b10> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaae379da00> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aac7ecf9bb8> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aaacf49d528> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aaacf49d528> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aaacf49d528> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aabdc652620> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aabdc652620> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aac357f4350> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aab9fc22a58> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aabdbefbe20> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
ner.java:87)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:82)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:262)
- locked <0x00002aad97b3b7e8> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aab016d6618> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aab016d6618> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 3 on 60020" daemon prio=10 tid=0x00000000019c4000
nid=0x5540 waiting for monitor entry [0x00000000441ff000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- waiting to lock <0x00002aab28d7de90> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 2 on 60020" daemon prio=10 tid=0x00000000019c2000
nid=0x553f runnable [0x00000000440fd000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aaaeef929f8> (a sun.nio.ch.Util$1)
- locked <0x00002aaaeef92a10> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaaffd9ef38> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aab2ac42300> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aabc43bef18> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aabc43bef18> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aabc43bef18> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aabd53c1d58> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aabd53c1d58> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aabb602a180> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aab44c85f18> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aac0d2be0b0> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
ner.java:87)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:82)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:262)
- locked <0x00002aab4eb7a680> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aabc7a3cc68> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aabc7a3cc68> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 1 on 60020" daemon prio=10 tid=0x0000000001535800
nid=0x553e runnable [0x0000000043ffc000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aad28449198> (a sun.nio.ch.Util$1)
- locked <0x00002aad284491b0> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aab3a682700> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aac5b5cbd28> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aabc43bd778> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aabc43bd778> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aabc43bd778> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aac4d4f9b90> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aac4d4f9b90> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aac500ac780> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aabe2211268> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aac537126f0> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
ner.java:87)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:82)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:262)
- locked <0x00002aadb912fca8> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aadcba8fe98> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aadcba8fe98> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 0 on 60020" daemon prio=10 tid=0x0000000001534000
nid=0x553d waiting for monitor entry [0x0000000043efc000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- waiting to lock <0x00002aabc7a3d258> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server listener on 60020" daemon prio=10 tid=0x00002aae4054e000
nid=0x553c runnable [0x0000000043dfb000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aaabc18bdf8> (a sun.nio.ch.Util$1)
- locked <0x00002aaabc18bde0> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaabbb1ced8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:41
4)
Locked ownable synchronizers:
- None
"IPC Server Responder" daemon prio=10 tid=0x00002aae4053d800 nid=0x553b
runnable [0x0000000043cfa000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aaabc2ceed8> (a sun.nio.ch.Util$1)
- locked <0x00002aaabc2ceef0> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaabbb33fa8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:5
88)
Locked ownable synchronizers:
- None
"Timer-0" daemon prio=10 tid=0x00002aae4053c800 nid=0x553a in
Object.wait() [0x0000000043bf9000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:531)
- locked <0x00002aaabd062850> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:484)
Locked ownable synchronizers:
- None
"983053578@qtp-1891789590-1 - Acceptor0
SelectChannelConnector@0.0.0.0:60030" prio=10 tid=0x00002aae40381000
nid=0x5539 runnable [0x0000000043af8000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aaabd07c460> (a sun.nio.ch.Util$1)
- locked <0x00002aaabd07c478> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaabbb26b78> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.ja
va:498)
at
org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
at
org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnect
or.java:124)
at
org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:
708)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java
:582)
Locked ownable synchronizers:
- None
"1148428095@qtp-1891789590-0" prio=10 tid=0x00002aae403d2000 nid=0x5538
in Object.wait() [0x00000000439f7000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java
:626)
- locked <0x00002aaabbc4c8f0> (a
org.mortbay.thread.QueuedThreadPool$PoolThread)
Locked ownable synchronizers:
- None
"regionserver60020.leaseChecker" daemon prio=10 tid=0x00002aae402ee800
nid=0x5537 waiting for monitor entry [0x00000000438f6000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.close(HRegion
.java:2409)
- waiting to lock <0x00002aadcba8fe98> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer$ScannerListener.lease
Expired(HRegionServer.java:1892)
at
org.apache.hadoop.hbase.regionserver.Leases.run(Leases.java:99)
Locked ownable synchronizers:
- None
"regionserver60020.majorCompactionChecker" daemon prio=10
tid=0x00002aae402ec800 nid=0x5536 in Object.wait() [0x00000000437f5000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaabc1f7bf0> (a java.lang.Object)
at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
- locked <0x00002aaabc1f7bf0> (a java.lang.Object)
at org.apache.hadoop.hbase.Chore.run(Chore.java:74)
Locked ownable synchronizers:
- None
"regionserver60020.compactor" daemon prio=10 tid=0x00002aae402eb000
nid=0x5535 runnable [0x00000000436f4000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aab26f462e0> (a sun.nio.ch.Util$1)
- locked <0x00002aab26f462c8> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaaeba1ac80> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aab44c85258> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aaac6caf140> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aaac6caf140> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aaac6caf140> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aaabd63fbb0> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aaabd63fbb0> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aab21b4e148> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aaaaedc39d0> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aab458cc280> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
ner.java:87)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:82)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:297)
- locked <0x00002aaace390ec0> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:326)
- locked <0x00002aaace390ec0> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:927)
at
org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:733)
- locked <0x00002aaacd20e840> (a java.lang.Object)
at
org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:
769)
at
org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:
714)
at
org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplit
Thread.java:81)
Locked ownable synchronizers:
- <0x00002aaabce779f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
"regionserver60020.cacheFlusher" daemon prio=10 tid=0x00002aae402d1800
nid=0x5534 waiting on condition [0x00000000435f3000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabc241a10> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
aitNanos(AbstractQueuedSynchronizer.java:2081)
at java.util.concurrent.DelayQueue.poll(DelayQueue.java:230)
at java.util.concurrent.DelayQueue.poll(DelayQueue.java:68)
at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher
.java:216)
Locked ownable synchronizers:
- None
"regionserver60020.logRoller" daemon prio=10 tid=0x00002aae402d1000
nid=0x5533 in Object.wait() [0x00000000434f2000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:78)
- locked <0x00002aaabcec18e0> (a
java.util.concurrent.atomic.AtomicBoolean)
Locked ownable synchronizers:
- None
"Timer thread for monitoring jvm" daemon prio=10 tid=0x00002aae402b4000
nid=0x5532 in Object.wait() [0x00000000433f1000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:531)
- locked <0x00002aaabcedce48> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:484)
Locked ownable synchronizers:
- None
"Timer thread for monitoring hbase" daemon prio=10
tid=0x00002aae402b2800 nid=0x5531 in Object.wait() [0x00000000432f0000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:531)
- locked <0x00002aaabceca148> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:484)
Locked ownable synchronizers:
- None
"regionserver60020.logSyncer" daemon prio=10 tid=0x00002aae402b1000
nid=0x5530 waiting on condition [0x00000000431ef000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:96
3)
Locked ownable synchronizers:
- None
"LeaseChecker" daemon prio=10 tid=0x00002aae4019a800 nid=0x552f
sleeping[0x00000000430ee000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1167)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"IPC Client (47) connection to
doop10.dt.sv4.decarta.com/10.241.8.230:60000 from an unknown user"
daemon prio=10 tid=0x00000000015c9800 nid=0x5528 in Object.wait()
[0x0000000042cea000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.waitForWork(HBaseClie
nt.java:431)
- locked <0x00002aaabc24dfc0> (a
org.apache.hadoop.hbase.ipc.HBaseClient$Connection)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:
476)
Locked ownable synchronizers:
- None
"DestroyJavaVM" prio=10 tid=0x00002aae4015c800 nid=0x54fa waiting on
condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"regionserver60020-EventThread" daemon prio=10 tid=0x00000000015c5000
nid=0x5526 waiting on condition [0x0000000042be9000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabc241a60> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
ait(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
86)
at
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
Locked ownable synchronizers:
- None
"regionserver60020-SendThread(doop3.dt.sv4.decarta.com:2181)" daemon
prio=10 tid=0x0000000001486800 nid=0x5525 runnable [0x0000000042ae8000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aaabbb3e538> (a sun.nio.ch.Util$1)
- locked <0x00002aaabbb3e550> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaabbb1db08> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
Locked ownable synchronizers:
- None
"regionserver60020" prio=10 tid=0x00002aae40187000 nid=0x5523 waiting on
condition [0x000000004028d000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabc2faea0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
aitNanos(AbstractQueuedSynchronizer.java:2081)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:4
23)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.jav
a:621)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"Timer thread for monitoring rpc" daemon prio=10 tid=0x00002aae40135800
nid=0x5522 in Object.wait() [0x00000000429e7000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:531)
- locked <0x00002aaabbb3cb08> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:484)
Locked ownable synchronizers:
- None
"IPC Reader 9 on port 60020" prio=10 tid=0x00002aae40133000 nid=0x5521
runnable [0x00000000428e6000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aaabbb010a8> (a sun.nio.ch.Util$1)
- locked <0x00002aaabbb010c0> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaabbb1e198> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
java:305)
- locked <0x00002aaabbb250e8> (a
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
a:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:603)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- <0x00002aaabbb4b008> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"IPC Reader 8 on port 60020" prio=10 tid=0x00002aae40118000 nid=0x5520
runnable [0x00000000427e5000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aaabbb16118> (a sun.nio.ch.Util$1)
- locked <0x00002aaabbb16130> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaabbb144f8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
java:305)
- locked <0x00002aaabbb30308> (a
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
a:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:603)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- <0x00002aaabbb32888> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"IPC Reader 7 on port 60020" prio=10 tid=0x00002aae400fd000 nid=0x551f
runnable [0x00000000426e4000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aaabbb2a228> (a sun.nio.ch.Util$1)
- locked <0x00002aaabbb2a240> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaabbb28ab8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
java:305)
- locked <0x00002aaabbb2a6d8> (a
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
a:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:603)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- <0x00002aaabbb4a068> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"IPC Reader 6 on port 60020" prio=10 tid=0x00002aae400e2800 nid=0x551e
runnable [0x00000000425e3000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aaabbb27028> (a sun.nio.ch.Util$1)
- locked <0x00002aaabbb27040> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaabbb258b8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
java:305)
- locked <0x00002aaabbb282e8> (a
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
a:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:603)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- <0x00002aaabbb4fc98> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"IPC Reader 5 on port 60020" prio=10 tid=0x00002aae400c7800 nid=0x551d
runnable [0x00000000413e5000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aaabbb2ea08> (a sun.nio.ch.Util$1)
- locked <0x00002aaabbb2ea20> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaabbb2d298> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
java:305)
- locked <0x00002aaabbb50c38> (a
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
a:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:603)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- <0x00002aaabbb523a8> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"IPC Reader 4 on port 60020" prio=10 tid=0x00002aae400ac800 nid=0x551c
runnable [0x00000000412e4000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aaabbb2aea8> (a sun.nio.ch.Util$1)
- locked <0x00002aaabbb2aec0> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaabbb1ca28> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
java:305)
- locked <0x00002aaabbb2c168> (a
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
a:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:603)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- <0x00002aaabbb48128> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"IPC Reader 3 on port 60020" prio=10 tid=0x00002aae40091800 nid=0x551b
runnable [0x00000000411e3000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aaabbb15c68> (a sun.nio.ch.Util$1)
- locked <0x00002aaabbb15c80> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaabbb12d88> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
java:305)
- locked <0x00002aaabbb165c8> (a
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
a:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:603)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- <0x00002aaabbb30ad8> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"IPC Reader 2 on port 60020" prio=10 tid=0x00002aae40077000 nid=0x551a
runnable [0x00000000410e2000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aaabbb3d148> (a sun.nio.ch.Util$1)
- locked <0x00002aaabbb3d160> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaabbb33828> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
java:305)
- locked <0x00002aaabbb3ebd8> (a
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
a:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:603)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- <0x00002aaabbb40028> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"IPC Reader 1 on port 60020" prio=10 tid=0x00002aae4005d000 nid=0x5519
runnable [0x0000000040fe1000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aaabbb15c98> (a sun.nio.ch.Util$1)
- locked <0x00002aaabbb15cb0> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaabbb12e00> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
java:305)
- locked <0x00002aaabbb165f0> (a
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
a:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:603)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- <0x00002aaabbb30b28> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"IPC Reader 0 on port 60020" prio=10 tid=0x00002aae40003000 nid=0x5518
runnable [0x0000000040ee0000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aaabbb16148> (a sun.nio.ch.Util$1)
- locked <0x00002aaabbb16160> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaabbb14570> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
java:305)
- locked <0x00002aaabbb30330> (a
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
a:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:603)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- <0x00002aaabbb328d8> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"main-EventThread" daemon prio=10 tid=0x00002aae3c1b6000 nid=0x5513
waiting on condition [0x00000000424e2000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabbb16618> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
ait(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
86)
at
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
Locked ownable synchronizers:
- None
"main-SendThread(doop7.dt.sv4.decarta.com:2181)" daemon prio=10
tid=0x00002aae3c11b000 nid=0x5512 runnable [0x00000000423e1000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aaabbb16178> (a sun.nio.ch.Util$1)
- locked <0x00002aaabbb16190> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaabbb145e8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
Locked ownable synchronizers:
- None
"Low Memory Detector" daemon prio=10 tid=0x000000000126c800 nid=0x5510
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"CompilerThread1" daemon prio=10 tid=0x000000000126a800 nid=0x550f
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"CompilerThread0" daemon prio=10 tid=0x0000000001266000 nid=0x550e
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Signal Dispatcher" daemon prio=10 tid=0x0000000001264000 nid=0x550d
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Surrogate Locker Thread (CMS)" daemon prio=10 tid=0x0000000001262000
nid=0x550c waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Finalizer" daemon prio=10 tid=0x000000000123b800 nid=0x550b in
Object.wait() [0x00000000422e0000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:133)
- locked <0x00002aaabbb15cf8> (a
java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:149)
at
java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)
Locked ownable synchronizers:
- None
"Reference Handler" daemon prio=10 tid=0x0000000001239800 nid=0x550a in
Object.wait() [0x00000000421df000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at
java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
- locked <0x00002aaabbb16d98> (a java.lang.ref.Reference$Lock)
Locked ownable synchronizers:
- None
"VM Thread" prio=10 tid=0x0000000001234800 nid=0x5509 runnable
"Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x0000000000fe9000
nid=0x54fb runnable
"Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x0000000000feb000
nid=0x54fc runnable
"Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x0000000000fec800
nid=0x54fd runnable
"Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x0000000000fee800
nid=0x54fe runnable
"Gang worker#4 (Parallel GC Threads)" prio=10 tid=0x0000000000ff0800
nid=0x54ff runnable
"Gang worker#5 (Parallel GC Threads)" prio=10 tid=0x0000000000ff2000
nid=0x5500 runnable
"Gang worker#6 (Parallel GC Threads)" prio=10 tid=0x0000000000ff4000
nid=0x5501 runnable
"Gang worker#7 (Parallel GC Threads)" prio=10 tid=0x0000000000ff6000
nid=0x5502 runnable
"Gang worker#8 (Parallel GC Threads)" prio=10 tid=0x0000000000ff7800
nid=0x5503 runnable
"Gang worker#9 (Parallel GC Threads)" prio=10 tid=0x0000000000ff9800
nid=0x5504 runnable
"Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00000000010f4000
nid=0x5508 runnable
"Gang worker#0 (Parallel CMS Threads)" prio=10 tid=0x00000000010ee000
nid=0x5505 runnable
"Gang worker#1 (Parallel CMS Threads)" prio=10 tid=0x00000000010f0000
nid=0x5506 runnable
"Gang worker#2 (Parallel CMS Threads)" prio=10 tid=0x00000000010f2000
nid=0x5507 runnable
"VM Periodic Task Thread" prio=10 tid=0x000000000126f800 nid=0x5511
waiting on condition
JNI global references: 906
-----Original Message-----
From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of
Stack
Sent: Sunday, September 11, 2011 2:27 PM
To: user@hbase.apache.org
Subject: Re: scanner deadlock?
0.90.4 fixes two deadlocks (HBASE-4101 and HBASE-4077). Since then,
there is HBASE-4367 (Which has a posted patch).
Below sounds like slowness. Can you thread dump the particular
regionserver and see what its up to? Is there other loading on the
system at the time? For example, loading on hdfs? Anything in the
hdfs logs for the datanode running beside the slow regionserver?
St.Ack
On Sat, Sep 10, 2011 at 5:50 PM, Geoff Hendrey <gh...@decarta.com>
wrote:
> Hi all -
>
>
>
> I'm still dealing with the saga of ScannerTimeoutException,
> UnknownScannerException, etc. I rewrote my code, in the hope that
simply
> a different approach and some different code paths might yield better
> results. No change. I tried many variations (caching 1 row vs caching
> many rows, changing the regionserver's lease, increasing the number of
> allowed zookeeper connections, etc). I created a fresh table on the
> thought that maybe there was some problem with the table...no change.
>
>
>
> I am dealing with what appears to be some sort of scanner deadlock. I
> have a total order partitioned mapreduce job. In the reducer, as long
as
> I use just one reducer, the task finishes quickly. But as soon as more
> than one reducer opens a scanner, the tasks procede in what I would
call
> a "jittery" lock step. They both are able to do ResultScanner.next() a
> few times, but then the call to next() freezes for a long period, and
> ScannerTimeoutException. I catch the exception, and get a new
> ResultScanner, and the pattern repeats. Jittery lockstep consisting of
> being able to get a few successful next() calls, then a lock up.
>
>
>
> I have not yet tried upgrading from 90.1 to higher. Nor have I tried
> tsuna's async client. Can anyone think of anything else I can try to
> resolve this? I've sunk quite a few late nights into this, and would
be
> very excited to find a solution.
>
>
>
> -geoff
>
>
RE: scanner deadlock?
Posted by Geoff Hendrey <gh...@decarta.com>.
Hi,
The cluster is definitely loaded. We are running a MR job with 130
reducers, each using a scanner. Ganglia shows an overall CPU utilization
of around 40%, with network pushing 800MB/s. The problem may correlate
with a major_compact I initiated couple days ago. Even today, I still
see a lot of compacting (continuous) on region servers.
Here is the thread dump from jstack. It does not reveal any explicit
deadlock. At the time I dumped the jstack, the region server logs are
reporting "IPC Server handler...ClosedChannelException". As soon as this
begins occurring, our network throughput, as seen on Ganglia drops from
800MB/s down to virtually zero. Then of course there is the cascade of
failures, eventually leading to the regionservers shutting down.
I don't see anything unusual in the datanode logs on the same server.
Just HDFS_READ and HDFS_WRITE for the most part.
jstack here:
===================================================================
2011-09-11 14:28:52
Full thread dump OpenJDK 64-Bit Server VM (14.0-b16 mixed mode):
"IPC Client (47) connection to
doop10.dt.sv4.decarta.com/10.241.8.230:54310 from hroot" daemon prio=10
tid=0x0000000001492800 nid=0x75cb in Object.wait() [0x0000000047633000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:676)
- locked <0x00002aaaaf680000> (a
org.apache.hadoop.ipc.Client$Connection)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:719)
Locked ownable synchronizers:
- None
"ResponseProcessor for block blk_2613854253703670198_5032689" daemon
prio=10 tid=0x0000000001988800 nid=0x75bb runnable [0x0000000046926000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aab69337248> (a sun.nio.ch.Util$1)
- locked <0x00002aab69337230> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaaf3897930> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.DataInputStream.readFully(DataInputStream.java:195)
at java.io.DataInputStream.readLong(DataInputStream.java:416)
at
org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFie
lds(DataTransferProtocol.java:120)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(D
FSClient.java:2638)
Locked ownable synchronizers:
- None
"DataStreamer for file
/hbase-20.6/NAM_CLUSTERKEYS3/60db9e0113c7578ed913dc37e0856954/.tmp/42977
03314038441248 block blk_2613854253703670198_5032689" daemon prio=10
tid=0x00002aae40e8e800 nid=0x75b8 in Object.wait() [0x0000000046e2b000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSCli
ent.java:2483)
- locked <0x00002aab6551a2d8> (a java.util.LinkedList)
Locked ownable synchronizers:
- None
"ResponseProcessor for block blk_8900384789253382699_5032549" daemon
prio=10 tid=0x00000000013cd000 nid=0x71d0 runnable [0x0000000047431000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aaae0569530> (a sun.nio.ch.Util$1)
- locked <0x00002aaae0569518> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaadb4c5c68> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.DataInputStream.readFully(DataInputStream.java:195)
at java.io.DataInputStream.readLong(DataInputStream.java:416)
at
org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFie
lds(DataTransferProtocol.java:120)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(D
FSClient.java:2638)
Locked ownable synchronizers:
- None
"DataStreamer for file
/hbase-20.6/.logs/doop5.dt.sv4.decarta.com,60020,1315768059645/doop5.dt.
sv4.decarta.com%3A60020.1315776307423 block
blk_8900384789253382699_5032549" daemon prio=10 tid=0x00000000016c3800
nid=0x71c8 in Object.wait() [0x000000004712e000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSCli
ent.java:2483)
- locked <0x00002aab092a7f38> (a java.util.LinkedList)
Locked ownable synchronizers:
- None
"Attach Listener" daemon prio=10 tid=0x0000000001e32000 nid=0x50f4
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"RS_CLOSE_REGION-doop5.dt.sv4.decarta.com,60020,1315768059645-2" daemon
prio=10 tid=0x00000000018cb800 nid=0x505 waiting on condition
[0x0000000047330000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabcefb428> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
ait(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
86)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
1043)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
a:1103)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:603)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"RS_CLOSE_REGION-doop5.dt.sv4.decarta.com,60020,1315768059645-1" daemon
prio=10 tid=0x000000000163e000 nid=0x504 waiting on condition
[0x0000000047532000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabcefb428> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
ait(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
86)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
1043)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
a:1103)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:603)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"RS_CLOSE_REGION-doop5.dt.sv4.decarta.com,60020,1315768059645-0" daemon
prio=10 tid=0x0000000001bff800 nid=0x503 waiting on condition
[0x0000000046f2c000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabcefb428> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
ait(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
86)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
1043)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
a:1103)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:603)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"sendParams-6" daemon prio=10 tid=0x00002aae401f1800 nid=0x6d3b waiting
on condition [0x000000004702d000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabceea260> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Synchro
nousQueue.java:453)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(Synchronous
Queue.java:352)
at
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:903)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
1043)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
a:1103)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:603)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"RS_OPEN_REGION-doop5.dt.sv4.decarta.com,60020,1315768059645-2" daemon
prio=10 tid=0x0000000001a7f000 nid=0x5b56 waiting on condition
[0x0000000046d2a000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabcefb2e8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
ait(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
86)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
1043)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
a:1103)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:603)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"RS_OPEN_REGION-doop5.dt.sv4.decarta.com,60020,1315768059645-1" daemon
prio=10 tid=0x0000000001af2800 nid=0x5b55 waiting on condition
[0x0000000042deb000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabcefb2e8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
ait(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
86)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
1043)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
a:1103)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:603)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"RS_OPEN_REGION-doop5.dt.sv4.decarta.com,60020,1315768059645-0" daemon
prio=10 tid=0x00000000014b3800 nid=0x5b54 waiting on condition
[0x0000000046c29000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabcefb2e8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
ait(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
86)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
1043)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
a:1103)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:603)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"RS_OPEN_ROOT-doop5.dt.sv4.decarta.com,60020,1315768059645-0" daemon
prio=10 tid=0x0000000001ad2800 nid=0x556e waiting on condition
[0x0000000046a27000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabcee8870> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
ait(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
86)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
1043)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
a:1103)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:603)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"LRU Statistics #0" prio=10 tid=0x00002aae4048a800 nid=0x5566 waiting on
condition [0x0000000046825000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabd0695c0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
aitNanos(AbstractQueuedSynchronizer.java:2081)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:193)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(S
cheduledThreadPoolExecutor.java:688)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(S
cheduledThreadPoolExecutor.java:681)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:
1043)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
a:1103)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:603)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"LruBlockCache.EvictionThread" daemon prio=10 tid=0x00002aae4049f800
nid=0x5565 in Object.wait() [0x0000000046724000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at
org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlo
ckCache.java:519)
- locked <0x00002aaabce92f50> (a
org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread)
Locked ownable synchronizers:
- None
"PRI IPC Server handler 9 on 60020" daemon prio=10
tid=0x00002aae4057e000 nid=0x5564 waiting on condition
[0x0000000046623000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabbb307e0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
ait(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
86)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
5)
Locked ownable synchronizers:
- None
"PRI IPC Server handler 8 on 60020" daemon prio=10
tid=0x00002aae4057c800 nid=0x5563 waiting on condition
[0x0000000046522000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabbb307e0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
ait(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
86)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
5)
Locked ownable synchronizers:
- None
"PRI IPC Server handler 7 on 60020" daemon prio=10
tid=0x00002aae40559000 nid=0x5562 waiting on condition
[0x0000000046421000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabbb307e0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
ait(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
86)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
5)
Locked ownable synchronizers:
- None
"PRI IPC Server handler 6 on 60020" daemon prio=10
tid=0x00002aae40557800 nid=0x5561 waiting on condition
[0x0000000046320000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabbb307e0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
ait(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
86)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
5)
Locked ownable synchronizers:
- None
"PRI IPC Server handler 5 on 60020" daemon prio=10
tid=0x00002aae40555800 nid=0x5560 waiting on condition
[0x000000004621f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabbb307e0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
ait(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
86)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
5)
Locked ownable synchronizers:
- None
"PRI IPC Server handler 4 on 60020" daemon prio=10
tid=0x00002aae40553800 nid=0x555f waiting on condition
[0x000000004611e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabbb307e0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
ait(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
86)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
5)
Locked ownable synchronizers:
- None
"PRI IPC Server handler 3 on 60020" daemon prio=10
tid=0x00002aae4054f000 nid=0x555e waiting on condition
[0x000000004601d000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabbb307e0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
ait(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
86)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
5)
Locked ownable synchronizers:
- None
"PRI IPC Server handler 2 on 60020" daemon prio=10
tid=0x00000000012f8800 nid=0x555d waiting on condition
[0x0000000045f1c000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabbb307e0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
ait(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
86)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
5)
Locked ownable synchronizers:
- None
"PRI IPC Server handler 1 on 60020" daemon prio=10
tid=0x0000000001991000 nid=0x555c waiting on condition
[0x0000000045e1b000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabbb307e0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
ait(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
86)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
5)
Locked ownable synchronizers:
- None
"PRI IPC Server handler 0 on 60020" daemon prio=10
tid=0x000000000198e800 nid=0x555b waiting on condition
[0x0000000045d1a000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabbb307e0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
ait(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
86)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:102
5)
Locked ownable synchronizers:
- None
"IPC Server handler 29 on 60020" daemon prio=10 tid=0x000000000198c800
nid=0x555a runnable [0x0000000045c18000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aac053b8450> (a sun.nio.ch.Util$1)
- locked <0x00002aac053b8438> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aab1037afc8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aac92976a30> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aabc43be0a8> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aabc43be0a8> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aabc43be0a8> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aac1a4e0ea8> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aac1a4e0ea8> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aac237c37a8> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aad39485080> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aab0b5ba8a8> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
ner.java:87)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:82)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:262)
- locked <0x00002aab89f59aa8> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aab016d6988> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aab016d6988> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 28 on 60020" daemon prio=10 tid=0x0000000001864000
nid=0x5559 runnable [0x0000000045b17000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aabaedb0608> (a sun.nio.ch.Util$1)
- locked <0x00002aabaedb05f0> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaca096fa50> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aac62a70b28> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aab856e6220> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aab856e6220> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aab856e6220> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aab0264afe8> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aab0264afe8> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aaae070be48> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aac18961940> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aabad8605d0> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
ner.java:87)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:82)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:262)
- locked <0x00002aab077e5d80> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aad30d57c18> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aad30d57c18> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 27 on 60020" daemon prio=10 tid=0x0000000001862000
nid=0x5558 runnable [0x0000000045a16000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aab2ea61b70> (a sun.nio.ch.Util$1)
- locked <0x00002aab2ea61b88> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aacbb364578> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aab1ffc0d00> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aad890f4640> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aad890f4640> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aad890f4640> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aaba1282b10> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aaba1282b10> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aac528d15d0> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aaaaf68a630> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aabc5c948b8> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
anner.java:115)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
va:255)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
va:394)
- locked <0x00002aab95832718> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:293)
- locked <0x00002aab95832718> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aab28d7de90> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aab28d7de90> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 26 on 60020" daemon prio=10 tid=0x0000000001860000
nid=0x5557 runnable [0x0000000045915000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aabaaa6b808> (a sun.nio.ch.Util$1)
- locked <0x00002aabaaa6b7f0> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaaf3899ca8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aaac1f59118> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aaac0e99db0> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aaac0e99db0> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aaac0e99db0> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aabb7221e88> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aabb7221e88> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aac28621da0> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aac18962690> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aab6876b850> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.loadBlock(HFile.ja
va:1447)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.reseekTo(HFile.jav
a:1325)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(St
oreFileScanner.java:152)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
anner.java:110)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
va:255)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
va:394)
- locked <0x00002aaabe3468f8> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:293)
- locked <0x00002aaabe3468f8> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aac8e37dbc8> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aac8e37dbc8> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 25 on 60020" daemon prio=10 tid=0x000000000173d800
nid=0x5556 runnable [0x0000000045814000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aab20178510> (a sun.nio.ch.Util$1)
- locked <0x00002aab20178528> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaad6187af0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aac891a6610> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aac1de6d9f8> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aac1de6d9f8> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aac1de6d9f8> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aaac041c6c8> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aaac041c6c8> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aab25e717e0> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aac3e6e4e28> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aaad4a425f0> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
ner.java:87)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:82)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:262)
- locked <0x00002aaabd5eb7e0> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aac8e37dc68> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aac8e37dc68> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 24 on 60020" daemon prio=10 tid=0x000000000173a000
nid=0x5555 waiting for monitor entry [0x0000000045714000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- waiting to lock <0x00002aab016d6618> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 23 on 60020" daemon prio=10 tid=0x0000000001444000
nid=0x5554 runnable [0x0000000045612000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aad70044d30> (a sun.nio.ch.Util$1)
- locked <0x00002aad70044d48> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aab328f1a40> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aaac9d48e68> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aac1de6dd40> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aac1de6dd40> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aac1de6dd40> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aab39d88148> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aab39d88148> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aab7addd0d8> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aab2811e8e8> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aabe56e8858> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
ner.java:87)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:82)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:262)
- locked <0x00002aad15b7d360> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aabc7a3d258> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aabc7a3d258> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 22 on 60020" daemon prio=10 tid=0x0000000001442000
nid=0x5553 runnable [0x0000000045511000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aabf6e38450> (a sun.nio.ch.Util$1)
- locked <0x00002aabf6e38468> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aabb9e99468> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aaae13c0340> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aad427a91c0> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aad427a91c0> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aad427a91c0> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aab7c7ca0c8> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aab7c7ca0c8> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aab3748f1a8> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aab52d56ed8> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aab60a86ca8> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(St
oreFileScanner.java:158)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
anner.java:110)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
va:255)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
va:394)
- locked <0x00002aab706a3e98> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:293)
- locked <0x00002aab706a3e98> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aaae3489758> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aaae3489758> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 21 on 60020" daemon prio=10 tid=0x0000000001792000
nid=0x5552 waiting for monitor entry [0x0000000045411000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- waiting to lock <0x00002aabc7a3d3e8> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 20 on 60020" daemon prio=10 tid=0x000000000178e000
nid=0x5551 runnable [0x000000004530f000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aab3745e018> (a sun.nio.ch.Util$1)
- locked <0x00002aab3745e030> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aab7d5cc0c0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aac035e4990> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aad890f42f8> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aad890f42f8> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aad890f42f8> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aab1baffa50> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aab1baffa50> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aaadef99fe8> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aac189625a0> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aabb88c7ce0> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
anner.java:115)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
va:255)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
va:394)
- locked <0x00002aaadfa95ec0> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:293)
- locked <0x00002aaadfa95ec0> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aabf6832658> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aabf6832658> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 19 on 60020" daemon prio=10 tid=0x0000000001b19800
nid=0x5550 runnable [0x000000004520e000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aab94af3848> (a sun.nio.ch.Util$1)
- locked <0x00002aab94af3830> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aac6ed3e1d8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aaac1194578> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aad890f2600> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aad890f2600> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aad890f2600> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aaae78be750> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aaae78be750> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aabba906cc0> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aac18963ef0> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aaacd2974c0> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
anner.java:115)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
va:255)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
va:394)
- locked <0x00002aab328ec218> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:293)
- locked <0x00002aab328ec218> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aabbaf1c898> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aabbaf1c898> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 18 on 60020" daemon prio=10 tid=0x0000000001b17800
nid=0x554f runnable [0x000000004510d000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aaae98101b0> (a sun.nio.ch.Util$1)
- locked <0x00002aaae9810198> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaaeba1b040> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aab1ac7d288> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aad890f0db8> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aad890f0db8> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aad890f0db8> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aab8c671d30> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aab8c671d30> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aab8c4f3de0> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aac18963dd0> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aab8d9fd7c8> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
anner.java:115)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
va:255)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
va:394)
- locked <0x00002aaba69901b8> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:293)
- locked <0x00002aaba69901b8> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aabbaf1e468> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aabbaf1e468> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 17 on 60020" daemon prio=10 tid=0x0000000001b15800
nid=0x554e runnable [0x000000004500c000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aaac2d0d348> (a sun.nio.ch.Util$1)
- locked <0x00002aaac2d0d330> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aacbb364b18> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aaac2750e38> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aaac1028128> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aaac1028128> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aaac1028128> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aaae41f13d8> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aaae41f13d8> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aab3c08c348> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aab52d54ef8> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aac0cb261b0> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
ner.java:87)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:82)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:262)
- locked <0x00002aaabefb9c10> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aad30d56688> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aad30d56688> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 16 on 60020" daemon prio=10 tid=0x00000000013fa000
nid=0x554d runnable [0x0000000044f0b000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aabec61aa08> (a sun.nio.ch.Util$1)
- locked <0x00002aabec61a9f0> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaae7590720> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aacf2246cb0> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aad06dd2c70> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aad06dd2c70> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aad06dd2c70> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aabe38abcd8> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aabe38abcd8> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aaad08ca208> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aac189636e0> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aab84fe43c8> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(St
oreFileScanner.java:158)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
anner.java:110)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
va:255)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
va:394)
- locked <0x00002aad072dbd30> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:293)
- locked <0x00002aad072dbd30> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aabdbaa0490> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aabdbaa0490> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 15 on 60020" daemon prio=10 tid=0x00000000013f7800
nid=0x554c runnable [0x0000000044e0a000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aac558cde50> (a sun.nio.ch.Util$1)
- locked <0x00002aac558cde38> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aac5cd41558> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aadcf530468> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aad427a97a8> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aad427a97a8> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aad427a97a8> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aab9410bc00> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aab9410bc00> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aaca7cfcf70> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aacf952a6c8> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aaac4472350> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
ner.java:87)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:82)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:297)
- locked <0x00002aaaefe12070> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aabbaf1d1a8> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aabbaf1d1a8> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 14 on 60020" daemon prio=10 tid=0x00000000013f5800
nid=0x554b runnable [0x0000000044d09000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aab28f91848> (a sun.nio.ch.Util$1)
- locked <0x00002aab28f91860> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaaf5031970> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aacec8cf590> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aaaf789b048> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aaaf789b048> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aaaf789b048> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aab9c1df470> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aab9c1df470> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aab7ec60418> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aab44c84dd8> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aab227a2dc8> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(St
oreFileScanner.java:158)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
anner.java:110)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
va:255)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
va:394)
- locked <0x00002aab50d25f28> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:293)
- locked <0x00002aab50d25f28> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aaae9efd9f0> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aaae9efd9f0> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 13 on 60020" daemon prio=10 tid=0x0000000001b61800
nid=0x554a waiting for monitor entry [0x0000000044c09000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- waiting to lock <0x00002aacc684dad0> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 12 on 60020" daemon prio=10 tid=0x0000000001b5f800
nid=0x5549 runnable [0x0000000044b07000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aab618bb060> (a sun.nio.ch.Util$1)
- locked <0x00002aabe1d82718> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aab76896238> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aab016dc128> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aad03182340> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aad03182340> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aad03182340> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aab8b32e3e0> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aab8b32e3e0> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aab8b436170> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aacd4fb8038> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aab8b4b4908> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
ner.java:87)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:82)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:262)
- locked <0x00002aab480f55b8> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aaae9efdb30> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aaae9efdb30> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 11 on 60020" daemon prio=10 tid=0x0000000001b5d000
nid=0x5548 waiting for monitor entry [0x0000000044a07000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- waiting to lock <0x00002aacc684dad0> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 10 on 60020" daemon prio=10 tid=0x000000000196a800
nid=0x5547 runnable [0x0000000044905000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aad7443dcd8> (a sun.nio.ch.Util$1)
- locked <0x00002aad7a31cb00> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aac9067d2d8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aad5103b0a0> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aadc79059c8> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aadc79059c8> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aadc79059c8> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aabf0f472b0> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aabf0f472b0> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aab8b2b29b0> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aad0477a310> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aabb7b16fd0> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
ner.java:87)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:82)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:262)
- locked <0x00002aab59165e10> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aac8e37dd58> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aac8e37dd58> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 9 on 60020" daemon prio=10 tid=0x0000000001968000
nid=0x5546 waiting for monitor entry [0x0000000044805000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.isFilterDone(
HRegion.java:2324)
- waiting to lock <0x00002aacc684dad0> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1850)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 8 on 60020" daemon prio=10 tid=0x0000000001967000
nid=0x5545 runnable [0x0000000044703000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aabb0b7b868> (a sun.nio.ch.Util$1)
- locked <0x00002aabb0b7b880> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaac79442c8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aad0106a6b0> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aac108e6768> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aac108e6768> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aac108e6768> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aac4d4ecd40> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aac4d4ecd40> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aaad20c3b08> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aac536bda88> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aaadfc86320> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.loadBlock(HFile.ja
va:1447)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.reseekTo(HFile.jav
a:1325)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(St
oreFileScanner.java:152)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
anner.java:110)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
va:255)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
va:394)
- locked <0x00002aad072dc1c8> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:293)
- locked <0x00002aad072dc1c8> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aac8e37d808> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aac8e37d808> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 7 on 60020" daemon prio=10 tid=0x00000000016de800
nid=0x5544 runnable [0x0000000044602000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aab9f4dc588> (a sun.nio.ch.Util$1)
- locked <0x00002aab9f4dc570> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aac6ed3e250> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aad003a0178> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aabbd698ec8> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aabbd698ec8> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aabbd698ec8> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aaad91ecfa8> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aaad91ecfa8> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aac237c5090> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aad65d9a7f0> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aaaccc5d6c8> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
ner.java:87)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:82)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:262)
- locked <0x00002aad15b7d2b8> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aacc684dad0> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aacc684dad0> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 6 on 60020" daemon prio=10 tid=0x00000000016dd000
nid=0x5543 runnable [0x0000000044501000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aaaf7db82f0> (a sun.nio.ch.Util$1)
- locked <0x00002aaaf7db82d8> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aab6f3339a8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aab494cdcb8> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aad890f5018> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aad890f5018> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aad890f5018> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aaae4d56928> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aaae4d56928> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aaae8d98798> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aaaaf686f40> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aaae77f0ad8> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
ner.java:87)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:82)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:262)
- locked <0x00002aaae4cd85a0> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aad30d55d28> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aad30d55d28> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 5 on 60020" daemon prio=10 tid=0x000000000194c800
nid=0x5542 runnable [0x0000000044400000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aaac5bf0ff0> (a sun.nio.ch.Util$1)
- locked <0x00002aaac5bf10c8> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aab0492d6d8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aac451e5d20> (a java.io.BufferedInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
89)
- locked <0x00002aabf9312408> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aabf9312408> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aabf9312408> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aab78dc6068> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aab78dc6068> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aab14390138> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aab7fbb83e8> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aac2cbf46c8> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileSc
anner.java:115)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.ja
va:255)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.ja
va:394)
- locked <0x00002aad15b7da98> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:293)
- locked <0x00002aad15b7da98> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aabc7a3d3e8> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aabc7a3d3e8> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 4 on 60020" daemon prio=10 tid=0x000000000194a800
nid=0x5541 runnable [0x00000000442ff000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aabbae85af8> (a sun.nio.ch.Util$1)
- locked <0x00002aabbae85b10> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaae379da00> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aac7ecf9bb8> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aaacf49d528> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aaacf49d528> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aaacf49d528> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aabdc652620> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aabdc652620> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aac357f4350> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aab9fc22a58> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aabdbefbe20> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
ner.java:87)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:82)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:262)
- locked <0x00002aad97b3b7e8> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aab016d6618> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aab016d6618> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 3 on 60020" daemon prio=10 tid=0x00000000019c4000
nid=0x5540 waiting for monitor entry [0x00000000441ff000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- waiting to lock <0x00002aab28d7de90> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 2 on 60020" daemon prio=10 tid=0x00000000019c2000
nid=0x553f runnable [0x00000000440fd000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aaaeef929f8> (a sun.nio.ch.Util$1)
- locked <0x00002aaaeef92a10> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaaffd9ef38> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aab2ac42300> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aabc43bef18> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aabc43bef18> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aabc43bef18> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aabd53c1d58> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aabd53c1d58> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aabb602a180> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aab44c85f18> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aac0d2be0b0> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
ner.java:87)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:82)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:262)
- locked <0x00002aab4eb7a680> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aabc7a3cc68> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aabc7a3cc68> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 1 on 60020" daemon prio=10 tid=0x0000000001535800
nid=0x553e runnable [0x0000000043ffc000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aad28449198> (a sun.nio.ch.Util$1)
- locked <0x00002aad284491b0> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aab3a682700> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aac5b5cbd28> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aabc43bd778> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aabc43bd778> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aabc43bd778> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aac4d4f9b90> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aac4d4f9b90> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aac500ac780> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aabe2211268> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aac537126f0> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
ner.java:87)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:82)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:262)
- locked <0x00002aadb912fca8> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:114)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(
HRegion.java:2344)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2301)
- locked <0x00002aadcba8fe98> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- locked <0x00002aadcba8fe98> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server handler 0 on 60020" daemon prio=10 tid=0x0000000001534000
nid=0x553d waiting for monitor entry [0x0000000043efc000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.
java:2317)
- waiting to lock <0x00002aabc7a3d258> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1832)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:103
9)
Locked ownable synchronizers:
- None
"IPC Server listener on 60020" daemon prio=10 tid=0x00002aae4054e000
nid=0x553c runnable [0x0000000043dfb000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aaabc18bdf8> (a sun.nio.ch.Util$1)
- locked <0x00002aaabc18bde0> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaabbb1ced8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:41
4)
Locked ownable synchronizers:
- None
"IPC Server Responder" daemon prio=10 tid=0x00002aae4053d800 nid=0x553b
runnable [0x0000000043cfa000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aaabc2ceed8> (a sun.nio.ch.Util$1)
- locked <0x00002aaabc2ceef0> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaabbb33fa8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:5
88)
Locked ownable synchronizers:
- None
"Timer-0" daemon prio=10 tid=0x00002aae4053c800 nid=0x553a in
Object.wait() [0x0000000043bf9000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:531)
- locked <0x00002aaabd062850> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:484)
Locked ownable synchronizers:
- None
"983053578@qtp-1891789590-1 - Acceptor0
SelectChannelConnector@0.0.0.0:60030" prio=10 tid=0x00002aae40381000
nid=0x5539 runnable [0x0000000043af8000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aaabd07c460> (a sun.nio.ch.Util$1)
- locked <0x00002aaabd07c478> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaabbb26b78> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.ja
va:498)
at
org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
at
org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnect
or.java:124)
at
org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:
708)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java
:582)
Locked ownable synchronizers:
- None
"1148428095@qtp-1891789590-0" prio=10 tid=0x00002aae403d2000 nid=0x5538
in Object.wait() [0x00000000439f7000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java
:626)
- locked <0x00002aaabbc4c8f0> (a
org.mortbay.thread.QueuedThreadPool$PoolThread)
Locked ownable synchronizers:
- None
"regionserver60020.leaseChecker" daemon prio=10 tid=0x00002aae402ee800
nid=0x5537 waiting for monitor entry [0x00000000438f6000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.close(HRegion
.java:2409)
- waiting to lock <0x00002aadcba8fe98> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer$ScannerListener.lease
Expired(HRegionServer.java:1892)
at
org.apache.hadoop.hbase.regionserver.Leases.run(Leases.java:99)
Locked ownable synchronizers:
- None
"regionserver60020.majorCompactionChecker" daemon prio=10
tid=0x00002aae402ec800 nid=0x5536 in Object.wait() [0x00000000437f5000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaabc1f7bf0> (a java.lang.Object)
at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
- locked <0x00002aaabc1f7bf0> (a java.lang.Object)
at org.apache.hadoop.hbase.Chore.run(Chore.java:74)
Locked ownable synchronizers:
- None
"regionserver60020.compactor" daemon prio=10 tid=0x00002aae402eb000
nid=0x5535 runnable [0x00000000436f4000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aab26f462e0> (a sun.nio.ch.Util$1)
- locked <0x00002aab26f462c8> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaaeba1ac80> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWi
thTimeout.java:332)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:
157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00002aab44c85258> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:13
50)
- locked <0x00002aaac6caf140> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.jav
a:237)
at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at
org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
- locked <0x00002aaac6caf140> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1249)
- locked <0x00002aaac6caf140> (a
org.apache.hadoop.hdfs.DFSClient$BlockReader)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.jav
a:1899)
- locked <0x00002aaabd63fbb0> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1951
)
- locked <0x00002aaabd63fbb0> (a
org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounde
dRangeFileInputStream.java:105)
- locked <0x00002aab21b4e148> (a
org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
at
java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00002aaaaedc39d0> (a java.io.BufferedInputStream)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x00002aab458cc280> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:12
76)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScan
ner.java:87)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java
:82)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:297)
- locked <0x00002aaace390ec0> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java
:326)
- locked <0x00002aaace390ec0> (a
org.apache.hadoop.hbase.regionserver.StoreScanner)
at
org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:927)
at
org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:733)
- locked <0x00002aaacd20e840> (a java.lang.Object)
at
org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:
769)
at
org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:
714)
at
org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplit
Thread.java:81)
Locked ownable synchronizers:
- <0x00002aaabce779f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
"regionserver60020.cacheFlusher" daemon prio=10 tid=0x00002aae402d1800
nid=0x5534 waiting on condition [0x00000000435f3000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabc241a10> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
aitNanos(AbstractQueuedSynchronizer.java:2081)
at java.util.concurrent.DelayQueue.poll(DelayQueue.java:230)
at java.util.concurrent.DelayQueue.poll(DelayQueue.java:68)
at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher
.java:216)
Locked ownable synchronizers:
- None
"regionserver60020.logRoller" daemon prio=10 tid=0x00002aae402d1000
nid=0x5533 in Object.wait() [0x00000000434f2000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:78)
- locked <0x00002aaabcec18e0> (a
java.util.concurrent.atomic.AtomicBoolean)
Locked ownable synchronizers:
- None
"Timer thread for monitoring jvm" daemon prio=10 tid=0x00002aae402b4000
nid=0x5532 in Object.wait() [0x00000000433f1000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:531)
- locked <0x00002aaabcedce48> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:484)
Locked ownable synchronizers:
- None
"Timer thread for monitoring hbase" daemon prio=10
tid=0x00002aae402b2800 nid=0x5531 in Object.wait() [0x00000000432f0000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:531)
- locked <0x00002aaabceca148> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:484)
Locked ownable synchronizers:
- None
"regionserver60020.logSyncer" daemon prio=10 tid=0x00002aae402b1000
nid=0x5530 waiting on condition [0x00000000431ef000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:96
3)
Locked ownable synchronizers:
- None
"LeaseChecker" daemon prio=10 tid=0x00002aae4019a800 nid=0x552f
sleeping[0x00000000430ee000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1167)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"IPC Client (47) connection to
doop10.dt.sv4.decarta.com/10.241.8.230:60000 from an unknown user"
daemon prio=10 tid=0x00000000015c9800 nid=0x5528 in Object.wait()
[0x0000000042cea000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.waitForWork(HBaseClie
nt.java:431)
- locked <0x00002aaabc24dfc0> (a
org.apache.hadoop.hbase.ipc.HBaseClient$Connection)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:
476)
Locked ownable synchronizers:
- None
"DestroyJavaVM" prio=10 tid=0x00002aae4015c800 nid=0x54fa waiting on
condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"regionserver60020-EventThread" daemon prio=10 tid=0x00000000015c5000
nid=0x5526 waiting on condition [0x0000000042be9000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabc241a60> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
ait(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
86)
at
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
Locked ownable synchronizers:
- None
"regionserver60020-SendThread(doop3.dt.sv4.decarta.com:2181)" daemon
prio=10 tid=0x0000000001486800 nid=0x5525 runnable [0x0000000042ae8000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aaabbb3e538> (a sun.nio.ch.Util$1)
- locked <0x00002aaabbb3e550> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaabbb1db08> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
Locked ownable synchronizers:
- None
"regionserver60020" prio=10 tid=0x00002aae40187000 nid=0x5523 waiting on
condition [0x000000004028d000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabc2faea0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
aitNanos(AbstractQueuedSynchronizer.java:2081)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:4
23)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.jav
a:621)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"Timer thread for monitoring rpc" daemon prio=10 tid=0x00002aae40135800
nid=0x5522 in Object.wait() [0x00000000429e7000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:531)
- locked <0x00002aaabbb3cb08> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:484)
Locked ownable synchronizers:
- None
"IPC Reader 9 on port 60020" prio=10 tid=0x00002aae40133000 nid=0x5521
runnable [0x00000000428e6000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aaabbb010a8> (a sun.nio.ch.Util$1)
- locked <0x00002aaabbb010c0> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaabbb1e198> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
java:305)
- locked <0x00002aaabbb250e8> (a
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
a:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:603)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- <0x00002aaabbb4b008> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"IPC Reader 8 on port 60020" prio=10 tid=0x00002aae40118000 nid=0x5520
runnable [0x00000000427e5000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aaabbb16118> (a sun.nio.ch.Util$1)
- locked <0x00002aaabbb16130> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaabbb144f8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
java:305)
- locked <0x00002aaabbb30308> (a
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
a:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:603)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- <0x00002aaabbb32888> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"IPC Reader 7 on port 60020" prio=10 tid=0x00002aae400fd000 nid=0x551f
runnable [0x00000000426e4000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aaabbb2a228> (a sun.nio.ch.Util$1)
- locked <0x00002aaabbb2a240> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaabbb28ab8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
java:305)
- locked <0x00002aaabbb2a6d8> (a
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
a:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:603)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- <0x00002aaabbb4a068> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"IPC Reader 6 on port 60020" prio=10 tid=0x00002aae400e2800 nid=0x551e
runnable [0x00000000425e3000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aaabbb27028> (a sun.nio.ch.Util$1)
- locked <0x00002aaabbb27040> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaabbb258b8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
java:305)
- locked <0x00002aaabbb282e8> (a
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
a:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:603)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- <0x00002aaabbb4fc98> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"IPC Reader 5 on port 60020" prio=10 tid=0x00002aae400c7800 nid=0x551d
runnable [0x00000000413e5000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aaabbb2ea08> (a sun.nio.ch.Util$1)
- locked <0x00002aaabbb2ea20> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaabbb2d298> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
java:305)
- locked <0x00002aaabbb50c38> (a
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
a:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:603)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- <0x00002aaabbb523a8> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"IPC Reader 4 on port 60020" prio=10 tid=0x00002aae400ac800 nid=0x551c
runnable [0x00000000412e4000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aaabbb2aea8> (a sun.nio.ch.Util$1)
- locked <0x00002aaabbb2aec0> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaabbb1ca28> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
java:305)
- locked <0x00002aaabbb2c168> (a
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
a:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:603)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- <0x00002aaabbb48128> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"IPC Reader 3 on port 60020" prio=10 tid=0x00002aae40091800 nid=0x551b
runnable [0x00000000411e3000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aaabbb15c68> (a sun.nio.ch.Util$1)
- locked <0x00002aaabbb15c80> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaabbb12d88> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
java:305)
- locked <0x00002aaabbb165c8> (a
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
a:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:603)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- <0x00002aaabbb30ad8> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"IPC Reader 2 on port 60020" prio=10 tid=0x00002aae40077000 nid=0x551a
runnable [0x00000000410e2000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aaabbb3d148> (a sun.nio.ch.Util$1)
- locked <0x00002aaabbb3d160> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaabbb33828> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
java:305)
- locked <0x00002aaabbb3ebd8> (a
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
a:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:603)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- <0x00002aaabbb40028> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"IPC Reader 1 on port 60020" prio=10 tid=0x00002aae4005d000 nid=0x5519
runnable [0x0000000040fe1000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aaabbb15c98> (a sun.nio.ch.Util$1)
- locked <0x00002aaabbb15cb0> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaabbb12e00> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
java:305)
- locked <0x00002aaabbb165f0> (a
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
a:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:603)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- <0x00002aaabbb30b28> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"IPC Reader 0 on port 60020" prio=10 tid=0x00002aae40003000 nid=0x5518
runnable [0x0000000040ee0000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aaabbb16148> (a sun.nio.ch.Util$1)
- locked <0x00002aaabbb16160> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaabbb14570> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.
java:305)
- locked <0x00002aaabbb30330> (a
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
a:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:603)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- <0x00002aaabbb328d8> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"main-EventThread" daemon prio=10 tid=0x00002aae3c1b6000 nid=0x5513
waiting on condition [0x00000000424e2000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaabbb16618> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw
ait(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:3
86)
at
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
Locked ownable synchronizers:
- None
"main-SendThread(doop7.dt.sv4.decarta.com:2181)" daemon prio=10
tid=0x00002aae3c11b000 nid=0x5512 runnable [0x00000000423e1000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:230)
at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00002aaabbb16178> (a sun.nio.ch.Util$1)
- locked <0x00002aaabbb16190> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaabbb145e8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
Locked ownable synchronizers:
- None
"Low Memory Detector" daemon prio=10 tid=0x000000000126c800 nid=0x5510
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"CompilerThread1" daemon prio=10 tid=0x000000000126a800 nid=0x550f
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"CompilerThread0" daemon prio=10 tid=0x0000000001266000 nid=0x550e
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Signal Dispatcher" daemon prio=10 tid=0x0000000001264000 nid=0x550d
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Surrogate Locker Thread (CMS)" daemon prio=10 tid=0x0000000001262000
nid=0x550c waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Finalizer" daemon prio=10 tid=0x000000000123b800 nid=0x550b in
Object.wait() [0x00000000422e0000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:133)
- locked <0x00002aaabbb15cf8> (a
java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:149)
at
java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)
Locked ownable synchronizers:
- None
"Reference Handler" daemon prio=10 tid=0x0000000001239800 nid=0x550a in
Object.wait() [0x00000000421df000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at
java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
- locked <0x00002aaabbb16d98> (a java.lang.ref.Reference$Lock)
Locked ownable synchronizers:
- None
"VM Thread" prio=10 tid=0x0000000001234800 nid=0x5509 runnable
"Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x0000000000fe9000
nid=0x54fb runnable
"Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x0000000000feb000
nid=0x54fc runnable
"Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x0000000000fec800
nid=0x54fd runnable
"Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x0000000000fee800
nid=0x54fe runnable
"Gang worker#4 (Parallel GC Threads)" prio=10 tid=0x0000000000ff0800
nid=0x54ff runnable
"Gang worker#5 (Parallel GC Threads)" prio=10 tid=0x0000000000ff2000
nid=0x5500 runnable
"Gang worker#6 (Parallel GC Threads)" prio=10 tid=0x0000000000ff4000
nid=0x5501 runnable
"Gang worker#7 (Parallel GC Threads)" prio=10 tid=0x0000000000ff6000
nid=0x5502 runnable
"Gang worker#8 (Parallel GC Threads)" prio=10 tid=0x0000000000ff7800
nid=0x5503 runnable
"Gang worker#9 (Parallel GC Threads)" prio=10 tid=0x0000000000ff9800
nid=0x5504 runnable
"Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00000000010f4000
nid=0x5508 runnable
"Gang worker#0 (Parallel CMS Threads)" prio=10 tid=0x00000000010ee000
nid=0x5505 runnable
"Gang worker#1 (Parallel CMS Threads)" prio=10 tid=0x00000000010f0000
nid=0x5506 runnable
"Gang worker#2 (Parallel CMS Threads)" prio=10 tid=0x00000000010f2000
nid=0x5507 runnable
"VM Periodic Task Thread" prio=10 tid=0x000000000126f800 nid=0x5511
waiting on condition
JNI global references: 906
-----Original Message-----
From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of
Stack
Sent: Sunday, September 11, 2011 2:27 PM
To: user@hbase.apache.org
Subject: Re: scanner deadlock?
0.90.4 fixes two deadlocks (HBASE-4101 and HBASE-4077). Since then,
there is HBASE-4367 (Which has a posted patch).
Below sounds like slowness. Can you thread dump the particular
regionserver and see what its up to? Is there other loading on the
system at the time? For example, loading on hdfs? Anything in the
hdfs logs for the datanode running beside the slow regionserver?
St.Ack
On Sat, Sep 10, 2011 at 5:50 PM, Geoff Hendrey <gh...@decarta.com>
wrote:
> Hi all -
>
>
>
> I'm still dealing with the saga of ScannerTimeoutException,
> UnknownScannerException, etc. I rewrote my code, in the hope that
simply
> a different approach and some different code paths might yield better
> results. No change. I tried many variations (caching 1 row vs caching
> many rows, changing the regionserver's lease, increasing the number of
> allowed zookeeper connections, etc). I created a fresh table on the
> thought that maybe there was some problem with the table...no change.
>
>
>
> I am dealing with what appears to be some sort of scanner deadlock. I
> have a total order partitioned mapreduce job. In the reducer, as long
as
> I use just one reducer, the task finishes quickly. But as soon as more
> than one reducer opens a scanner, the tasks procede in what I would
call
> a "jittery" lock step. They both are able to do ResultScanner.next() a
> few times, but then the call to next() freezes for a long period, and
> ScannerTimeoutException. I catch the exception, and get a new
> ResultScanner, and the pattern repeats. Jittery lockstep consisting of
> being able to get a few successful next() calls, then a lock up.
>
>
>
> I have not yet tried upgrading from 90.1 to higher. Nor have I tried
> tsuna's async client. Can anyone think of anything else I can try to
> resolve this? I've sunk quite a few late nights into this, and would
be
> very excited to find a solution.
>
>
>
> -geoff
>
>
Re: scanner deadlock?
Posted by Stack <st...@duboce.net>.
0.90.4 fixes two deadlocks (HBASE-4101 and HBASE-4077). Since then,
there is HBASE-4367 (Which has a posted patch).
Below sounds like slowness. Can you thread dump the particular
regionserver and see what its up to? Is there other loading on the
system at the time? For example, loading on hdfs? Anything in the
hdfs logs for the datanode running beside the slow regionserver?
St.Ack
On Sat, Sep 10, 2011 at 5:50 PM, Geoff Hendrey <gh...@decarta.com> wrote:
> Hi all -
>
>
>
> I'm still dealing with the saga of ScannerTimeoutException,
> UnknownScannerException, etc. I rewrote my code, in the hope that simply
> a different approach and some different code paths might yield better
> results. No change. I tried many variations (caching 1 row vs caching
> many rows, changing the regionserver's lease, increasing the number of
> allowed zookeeper connections, etc). I created a fresh table on the
> thought that maybe there was some problem with the table...no change.
>
>
>
> I am dealing with what appears to be some sort of scanner deadlock. I
> have a total order partitioned mapreduce job. In the reducer, as long as
> I use just one reducer, the task finishes quickly. But as soon as more
> than one reducer opens a scanner, the tasks procede in what I would call
> a "jittery" lock step. They both are able to do ResultScanner.next() a
> few times, but then the call to next() freezes for a long period, and
> ScannerTimeoutException. I catch the exception, and get a new
> ResultScanner, and the pattern repeats. Jittery lockstep consisting of
> being able to get a few successful next() calls, then a lock up.
>
>
>
> I have not yet tried upgrading from 90.1 to higher. Nor have I tried
> tsuna's async client. Can anyone think of anything else I can try to
> resolve this? I've sunk quite a few late nights into this, and would be
> very excited to find a solution.
>
>
>
> -geoff
>
>