You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Johannes Schaback <jo...@visual-meta.com> on 2014/08/22 19:28:57 UTC
single RegionServer stuck, causing cluster to hang
Dear HBase-Pros,
we face a serious issue with our HBase production cluster for two days now.
Every couple minutes, a random RegionServer gets stuck and does not process
any requests. In addition this causes the other RegionServers to
freeze within a minute which brings down the entire cluster. Stopping the
affected RegionServer unblocks the cluster and everything comes back to
normal.
We run 27 RegionServers, each having 31 GB JVM memory. The HBase Version is
0.98.5 on Hadoop 2.4.1. We basically have two tables, the first having
about 4,500 Regions and holding 8 TB with 1000 requests per second, the
second table is around 200 Regions with about 50,000 to 120,000 requests
per sec over all Regions, 800 GB worth of data and with IN_MEMORY enabled.
While investigating the problem, I found out, that every healthy
RegionServer has the following thread:
Thread 12 (RpcServer.listener,port=60020):
State: RUNNABLE
Blocked count: 35
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
org.apache.hadoop.hbase.ipc.RpcServer$Listener.run(RpcServer.java:684)
When suddenly becoming a blocked RegionServer, this particular thread then
looks like
Thread 12 (RpcServer.listener,port=60020):
State: BLOCKED
Blocked count: 2889
Waited count: 0
Blocked on org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader@38cba1a1
Blocked by 14 (RpcServer.reader=1,port=60020)
Stack:
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.registerChannel(RpcServer.java:619)
org.apache.hadoop.hbase.ipc.RpcServer$Listener.doAccept(RpcServer.java:774)
org.apache.hadoop.hbase.ipc.RpcServer$Listener.run(RpcServer.java:692)
Also, JMX shows for an unhealthy RegionServer that
- "queueSize" grows quickly and constantly to values greater than 60k,
and
- "numCallsInGeneralQueue" quickly reaches 300
Both values are usually very small or 0 under normal circumstances, but in
case of a RS "getting stuck" they explode, which leads me to believe that
the IPC-queue does not get processed properly causing the RegionServer to
become "deaf".
These two symptoms appear to bring down the entire cluster. When killign
that RS, everyhing goes back to normal.
I could not find any correlation between this phenomenon and compactions,
load or other factors. hbck says it is all fine as well.
The servers are all 3.2.0-4-amd64 Debian, 12 cores, 96 GB RAM. Besides the
RS and a DataNode, there isn't too much running on the boxes so the load
(top) is usually around 5 to 10 and bandwidth does not exceed 10 MB on
average.
We currently survive by polling /jmx of all RegionServers constantly and
restarting those off that show the symptioms :(
Do you have any idea what could be causing this?
Thank you very much in advance!
Johannes
Re: single RegionServer stuck, causing cluster to hang
Posted by Johannes Schaback <jo...@visual-meta.com>.
Hi Michael,
thank you for your quick reply!
I paste an entire debug dump of a stuck RegionServer, including stacks etc.
I guess we have little to hide considering our problem :)
Master status for r3c7s4.colo.vm,60020,1408707352291 as of Fri Aug 22
15:15:46 CEST 2014
Version Info:
===========================================================
HBase 0.98.5-hadoop2
Subversion file:///var/tmp/0.98.5RC0/hbase-0.98.5 -r Unknown
Compiled by apurtell on Mon Aug 4 23:58:06 PDT 2014
Hadoop 2.4.1
Subversion http://svn.apache.org/repos/asf/hadoop/common -r 1604318
Compiled by jenkins on 2014-06-21T05:43Z
Tasks:
===========================================================
Task: RpcServer.reader=1,port=60020
Status: WAITING:Waiting for a call
Running for 5950s
Task: RpcServer.reader=2,port=60020
Status: WAITING:Waiting for a call
Running for 5920s
Task: RpcServer.reader=3,port=60020
Status: WAITING:Waiting for a call
Running for 5880s
Task: RpcServer.reader=4,port=60020
Status: WAITING:Waiting for a call
Running for 5850s
Task: RpcServer.reader=5,port=60020
Status: WAITING:Waiting for a call
Running for 5802s
Task: RpcServer.reader=6,port=60020
Status: WAITING:Waiting for a call
Running for 5707s
Task: RpcServer.reader=7,port=60020
Status: WAITING:Waiting for a call
Running for 5707s
Task: RpcServer.reader=8,port=60020
Status: WAITING:Waiting for a call
Running for 5700s
Task: RpcServer.reader=9,port=60020
Status: WAITING:Waiting for a call
Running for 5697s
Task: RpcServer.reader=0,port=60020
Status: WAITING:Waiting for a call
Running for 5697s
Executors:
===========================================================
Status for executor: Executor-2-RS_OPEN_META-r3c7s4:60020
=======================================
0 events queued, 0 running
Status for executor: Executor-5-RS_LOG_REPLAY_OPS-r3c7s4:60020
=======================================
0 events queued, 0 running
Status for executor: Executor-1-RS_OPEN_REGION-r3c7s4:60020
=======================================
0 events queued, 0 running
Status for executor: Executor-4-RS_CLOSE_META-r3c7s4:60020
=======================================
0 events queued, 0 running
Status for executor: Executor-3-RS_CLOSE_REGION-r3c7s4:60020
=======================================
0 events queued, 0 running
Stacks:
===========================================================
Process Thread Dump:
85 active threads
Thread 2120 (1878278300@qtp-1123595403-7):
State: RUNNABLE
Blocked count: 887
Waited count: 890
Stack:
sun.management.ThreadImpl.getThreadInfo1(Native Method)
sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:174)
sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:139)
org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:165)
org.apache.hadoop.hbase.regionserver.RSDumpServlet.doGet(RSDumpServlet.java:81)
javax.servlet.http.HttpServlet.service(HttpServlet.java:735)
javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1089)
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
Thread 1609 (RS_CLOSE_REGION-r3c7s4:60020-2):
State: WAITING
Blocked count: 254
Waited count: 406
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7dd5615b
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
Thread 1607 (RS_CLOSE_REGION-r3c7s4:60020-1):
State: WAITING
Blocked count: 210
Waited count: 343
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7dd5615b
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
Thread 1604 (RS_CLOSE_REGION-r3c7s4:60020-0):
State: WAITING
Blocked count: 233
Waited count: 374
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7dd5615b
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
Thread 1355 (ResponseProcessor for block
BP-1157637685-192.168.3.192-1382642140917:blk_1294960049_255276569):
State: RUNNABLE
Blocked count: 1
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
java.io.FilterInputStream.read(FilterInputStream.java:83)
java.io.FilterInputStream.read(FilterInputStream.java:83)
org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1984)
org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:796)
Thread 1354 (DataStreamer for file
/hbase/WALs/r3c7s4.colo.vm,60020,1408707352291/r3c7s4.colo.vm%2C60020%2C1408707352291.1408710955280
block BP-1157637685-192.168.3.192-1382642140917:blk_1294960049_255276569):
State: TIMED_WAITING
Blocked count: 1061
Waited count: 1088
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:502)
Thread 181 (ReplicationExecutor-0):
State: WAITING
Blocked count: 15
Waited count: 45
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@18a8507c
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
Thread 144 (regionserver60020-splits-1408707430778):
State: WAITING
Blocked count: 39
Waited count: 53
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@46b1c7f4
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
Thread 131 (regionserver60020-smallCompactions-1408707426607):
State: WAITING
Blocked count: 923
Waited count: 1365
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@41be886e
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:539)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
Thread 122 (RS_OPEN_REGION-r3c7s4:60020-1):
State: WAITING
Blocked count: 1493
Waited count: 2245
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@43ef0f7a
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
Thread 123 (RS_OPEN_REGION-r3c7s4:60020-2):
State: WAITING
Blocked count: 1520
Waited count: 2305
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@43ef0f7a
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
Thread 121 (RS_OPEN_REGION-r3c7s4:60020-0):
State: WAITING
Blocked count: 1529
Waited count: 2317
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@43ef0f7a
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
Thread 120 (org.apache.hadoop.hdfs.PeerCache@51bd2431):
State: TIMED_WAITING
Blocked count: 0
Waited count: 1973
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
java.lang.Thread.run(Thread.java:744)
Thread 116 (SplitLogWorker-r3c7s4.colo.vm,60020,1408707352291-EventThread):
State: WAITING
Blocked count: 0
Waited count: 3
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3142a13
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
Thread 115
(SplitLogWorker-r3c7s4.colo.vm,60020,1408707352291-SendThread(192.168.3.191:2181
)):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
Thread 113 (SplitLogWorker-r3c7s4.colo.vm,60020,1408707352291):
State: TIMED_WAITING
Blocked count: 5
Waited count: 1203
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:261)
org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:205)
java.lang.Thread.run(Thread.java:744)
Thread 112 (ReplicationRpcServer.handler=2,queue=0,port=60020):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@53cd9646
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:111)
org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
java.lang.Thread.run(Thread.java:744)
Thread 111 (ReplicationRpcServer.handler=1,queue=0,port=60020):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@53cd9646
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:111)
org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
java.lang.Thread.run(Thread.java:744)
Thread 110 (ReplicationRpcServer.handler=0,queue=0,port=60020):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@53cd9646
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:111)
org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
java.lang.Thread.run(Thread.java:744)
Thread 109 (PriorityRpcServer.handler=9,queue=0,port=60020):
State: WAITING
Blocked count: 56
Waited count: 159
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5e44f8a5
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:111)
org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
java.lang.Thread.run(Thread.java:744)
Thread 108 (PriorityRpcServer.handler=8,queue=0,port=60020):
State: WAITING
Blocked count: 70
Waited count: 180
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5e44f8a5
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:111)
org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
java.lang.Thread.run(Thread.java:744)
Thread 107 (PriorityRpcServer.handler=7,queue=0,port=60020):
State: WAITING
Blocked count: 65
Waited count: 170
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5e44f8a5
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:111)
org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
java.lang.Thread.run(Thread.java:744)
Thread 106 (PriorityRpcServer.handler=6,queue=0,port=60020):
State: WAITING
Blocked count: 38
Waited count: 145
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5e44f8a5
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:111)
org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
java.lang.Thread.run(Thread.java:744)
Thread 105 (PriorityRpcServer.handler=5,queue=0,port=60020):
State: WAITING
Blocked count: 86
Waited count: 194
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5e44f8a5
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:111)
org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
java.lang.Thread.run(Thread.java:744)
Thread 104 (PriorityRpcServer.handler=4,queue=0,port=60020):
State: WAITING
Blocked count: 81
Waited count: 193
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5e44f8a5
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:111)
org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
java.lang.Thread.run(Thread.java:744)
Thread 103 (PriorityRpcServer.handler=3,queue=0,port=60020):
State: WAITING
Blocked count: 76
Waited count: 167
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5e44f8a5
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:111)
org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
java.lang.Thread.run(Thread.java:744)
Thread 102 (PriorityRpcServer.handler=2,queue=0,port=60020):
State: WAITING
Blocked count: 92
Waited count: 171
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5e44f8a5
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:111)
org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
java.lang.Thread.run(Thread.java:744)
Thread 101 (PriorityRpcServer.handler=1,queue=0,port=60020):
State: WAITING
Blocked count: 132
Waited count: 207
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5e44f8a5
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:111)
org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
java.lang.Thread.run(Thread.java:744)
Thread 100 (PriorityRpcServer.handler=0,queue=0,port=60020):
State: WAITING
Blocked count: 55
Waited count: 153
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5e44f8a5
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:111)
org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
java.lang.Thread.run(Thread.java:744)
Thread 97 (defaultRpcServer.handler=27,queue=0,port=60020):
State: WAITING
Blocked count: 176
Waited count: 3136
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24b3a041
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:111)
org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
java.lang.Thread.run(Thread.java:744)
Thread 94 (defaultRpcServer.handler=24,queue=0,port=60020):
State: WAITING
Blocked count: 147
Waited count: 3093
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24b3a041
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:111)
org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
java.lang.Thread.run(Thread.java:744)
Thread 91 (defaultRpcServer.handler=21,queue=0,port=60020):
State: WAITING
Blocked count: 163
Waited count: 3107
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24b3a041
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:111)
org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
java.lang.Thread.run(Thread.java:744)
Thread 90 (defaultRpcServer.handler=20,queue=2,port=60020):
State: WAITING
Blocked count: 227
Waited count: 3792
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22048fd
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:111)
org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
java.lang.Thread.run(Thread.java:744)
Thread 88 (defaultRpcServer.handler=18,queue=0,port=60020):
State: WAITING
Blocked count: 175
Waited count: 3104
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24b3a041
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:111)
org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
java.lang.Thread.run(Thread.java:744)
Thread 82 (defaultRpcServer.handler=12,queue=0,port=60020):
State: WAITING
Blocked count: 173
Waited count: 3131
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24b3a041
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:111)
org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
java.lang.Thread.run(Thread.java:744)
Thread 81 (defaultRpcServer.handler=11,queue=2,port=60020):
State: WAITING
Blocked count: 214
Waited count: 3832
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22048fd
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:111)
org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
java.lang.Thread.run(Thread.java:744)
Thread 76 (defaultRpcServer.handler=6,queue=0,port=60020):
State: WAITING
Blocked count: 208
Waited count: 3171
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24b3a041
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:111)
org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
java.lang.Thread.run(Thread.java:744)
Thread 12 (RpcServer.listener,port=60020):
State: BLOCKED
Blocked count: 2889
Waited count: 0
Blocked on org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader@38cba1a1
Blocked by 14 (RpcServer.reader=1,port=60020)
Stack:
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.registerChannel(RpcServer.java:619)
org.apache.hadoop.hbase.ipc.RpcServer$Listener.doAccept(RpcServer.java:774)
org.apache.hadoop.hbase.ipc.RpcServer$Listener.run(RpcServer.java:692)
Thread 24 (RpcServer.responder):
State: RUNNABLE
Blocked count: 159
Waited count: 149
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.hadoop.hbase.ipc.RpcServer$Responder.doRunLoop(RpcServer.java:874)
org.apache.hadoop.hbase.ipc.RpcServer$Responder.run(RpcServer.java:857)
Thread 69 (r3c7s4:60020Replication Statistics #0):
State: TIMED_WAITING
Blocked count: 0
Waited count: 20
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
Thread 67 (regionserver60020-EventThread):
State: WAITING
Blocked count: 0
Waited count: 3
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4a92786f
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
Thread 66 (regionserver60020-SendThread(192.168.3.191:2181)):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
Thread 42 (regionserver60020.leaseChecker):
State: TIMED_WAITING
Blocked count: 0
Waited count: 600
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hbase.regionserver.Leases.run(Leases.java:95)
java.lang.Thread.run(Thread.java:744)
Thread 44 (regionserver60020.nonceCleaner):
State: TIMED_WAITING
Blocked count: 0
Waited count: 17
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:92)
org.apache.hadoop.hbase.Chore.run(Chore.java:88)
java.lang.Thread.run(Thread.java:744)
Thread 41 (regionserver60020.periodicFlusher):
State: TIMED_WAITING
Blocked count: 0
Waited count: 600
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:92)
org.apache.hadoop.hbase.Chore.run(Chore.java:88)
java.lang.Thread.run(Thread.java:744)
Thread 40 (regionserver60020.compactionChecker):
State: TIMED_WAITING
Blocked count: 0
Waited count: 600
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:92)
org.apache.hadoop.hbase.Chore.run(Chore.java:88)
java.lang.Thread.run(Thread.java:744)
Thread 63 (MemStoreFlusher.1):
State: TIMED_WAITING
Blocked count: 124
Waited count: 828
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
java.util.concurrent.DelayQueue.poll(DelayQueue.java:256)
java.util.concurrent.DelayQueue.poll(DelayQueue.java:68)
org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:233)
java.lang.Thread.run(Thread.java:744)
Thread 61 (MemStoreFlusher.0):
State: TIMED_WAITING
Blocked count: 87
Waited count: 775
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
java.util.concurrent.DelayQueue.poll(DelayQueue.java:256)
java.util.concurrent.DelayQueue.poll(DelayQueue.java:68)
org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:233)
java.lang.Thread.run(Thread.java:744)
Thread 49 (regionserver60020.logRoller):
State: TIMED_WAITING
Blocked count: 11
Waited count: 616
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:78)
java.lang.Thread.run(Thread.java:744)
Thread 60 (HBase-Metrics2-1):
State: TIMED_WAITING
Blocked count: 0
Waited count: 22808
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
Thread 59 (regionserver60020-WAL.AsyncNotifier):
State: WAITING
Blocked count: 2737
Waited count: 2738
Waiting on java.lang.Object@22104a3e
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:503)
org.apache.hadoop.hbase.regionserver.wal.FSHLog$AsyncNotifier.run(FSHLog.java:1319)
java.lang.Thread.run(Thread.java:744)
Thread 58 (regionserver60020-WAL.AsyncSyncer4):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.lang.Object@5b7a3dea
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:503)
org.apache.hadoop.hbase.regionserver.wal.FSHLog$AsyncSyncer.run(FSHLog.java:1199)
java.lang.Thread.run(Thread.java:744)
Thread 57 (regionserver60020-WAL.AsyncSyncer3):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.lang.Object@3676fc91
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:503)
org.apache.hadoop.hbase.regionserver.wal.FSHLog$AsyncSyncer.run(FSHLog.java:1199)
java.lang.Thread.run(Thread.java:744)
Thread 56 (regionserver60020-WAL.AsyncSyncer2):
State: WAITING
Blocked count: 5
Waited count: 6
Waiting on java.lang.Object@3dbfaf86
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:503)
org.apache.hadoop.hbase.regionserver.wal.FSHLog$AsyncSyncer.run(FSHLog.java:1199)
java.lang.Thread.run(Thread.java:744)
Thread 55 (regionserver60020-WAL.AsyncSyncer1):
State: WAITING
Blocked count: 48
Waited count: 49
Waiting on java.lang.Object@739ca615
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:503)
org.apache.hadoop.hbase.regionserver.wal.FSHLog$AsyncSyncer.run(FSHLog.java:1199)
java.lang.Thread.run(Thread.java:744)
Thread 54 (regionserver60020-WAL.AsyncSyncer0):
State: WAITING
Blocked count: 8235
Waited count: 8234
Waiting on java.lang.Object@315be7ff
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:503)
org.apache.hadoop.hbase.regionserver.wal.FSHLog$AsyncSyncer.run(FSHLog.java:1199)
java.lang.Thread.run(Thread.java:744)
Thread 53 (regionserver60020-WAL.AsyncWriter):
State: WAITING
Blocked count: 2737
Waited count: 2737
Waiting on java.lang.Object@8dc2faa
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:503)
org.apache.hadoop.hbase.regionserver.wal.FSHLog$AsyncWriter.run(FSHLog.java:1106)
java.lang.Thread.run(Thread.java:744)
Thread 51 (LeaseRenewer:visualmeta@visualmeta):
State: TIMED_WAITING
Blocked count: 199
Waited count: 6380
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438)
org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
java.lang.Thread.run(Thread.java:744)
Thread 48 (IPC Parameter Sending Thread #0):
State: TIMED_WAITING
Blocked count: 0
Waited count: 7996
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
Thread 46 (IPC Client (1773132304) connection to r2c3s1.colo.vm/
192.168.3.171:60000 from visualmeta):
State: TIMED_WAITING
Blocked count: 1996
Waited count: 1996
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hbase.ipc.RpcClient$Connection.waitForWork(RpcClient.java:678)
org.apache.hadoop.hbase.ipc.RpcClient$Connection.run(RpcClient.java:726)
Thread 45 (JvmPauseMonitor):
State: TIMED_WAITING
Blocked count: 0
Waited count: 11953
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hbase.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:159)
java.lang.Thread.run(Thread.java:744)
Thread 37 (regionserver60020-EventThread):
State: WAITING
Blocked count: 0
Waited count: 3
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@378c93c2
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
Thread 36 (regionserver60020-SendThread(192.168.3.171:2181)):
State: RUNNABLE
Blocked count: 24
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
Thread 34 (regionserver60020-EventThread):
State: WAITING
Blocked count: 936
Waited count: 967
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6e38c20b
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
Thread 33 (regionserver60020-SendThread(192.168.3.171:2181)):
State: RUNNABLE
Blocked count: 348
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
Thread 32 (regionserver60020):
State: TIMED_WAITING
Blocked count: 2068
Waited count: 4060
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:92)
org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:56)
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:900)
java.lang.Thread.run(Thread.java:744)
Thread 31 (Timer-0):
State: TIMED_WAITING
Blocked count: 0
Waited count: 200
Stack:
java.lang.Object.wait(Native Method)
java.util.TimerThread.mainLoop(Timer.java:552)
java.util.TimerThread.run(Timer.java:505)
Thread 30 (1702920744@qtp-1123595403-1 - Acceptor0
SelectChannelConnector@0.0.0.0:60030):
State: RUNNABLE
Blocked count: 156
Waited count: 1
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Thread 27 (LruStats #0):
State: TIMED_WAITING
Blocked count: 0
Waited count: 20
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
Thread 25 (main.LruBlockCache.EvictionThread):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on
org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread@39a19ad8
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:503)
org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:705)
java.lang.Thread.run(Thread.java:744)
Thread 23 (Timer for 'HBase' metrics system):
State: TIMED_WAITING
Blocked count: 0
Waited count: 600
Stack:
java.lang.Object.wait(Native Method)
java.util.TimerThread.mainLoop(Timer.java:552)
java.util.TimerThread.run(Timer.java:505)
Thread 22 (RpcServer.reader=9,port=60020):
State: WAITING
Blocked count: 289
Waited count: 290
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@40e06e28
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:349)
org.apache.hadoop.hbase.ipc.BalancedQueueRpcExecutor.dispatch(BalancedQueueRpcExecutor.java:67)
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.dispatch(SimpleRpcScheduler.java:129)
org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequest(RpcServer.java:1747)
org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneRpc(RpcServer.java:1651)
org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1511)
org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:798)
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:589)
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:564)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
Thread 21 (RpcServer.reader=8,port=60020):
State: WAITING
Blocked count: 289
Waited count: 291
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@40e06e28
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:349)
org.apache.hadoop.hbase.ipc.BalancedQueueRpcExecutor.dispatch(BalancedQueueRpcExecutor.java:67)
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.dispatch(SimpleRpcScheduler.java:129)
org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequest(RpcServer.java:1747)
org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneRpc(RpcServer.java:1651)
org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1511)
org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:798)
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:589)
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:564)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
Thread 20 (RpcServer.reader=7,port=60020):
State: RUNNABLE
Blocked count: 289
Waited count: 289
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:578)
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:564)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
Thread 19 (RpcServer.reader=6,port=60020):
State: RUNNABLE
Blocked count: 289
Waited count: 289
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:578)
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:564)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
Thread 18 (RpcServer.reader=5,port=60020):
State: WAITING
Blocked count: 290
Waited count: 290
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@40e06e28
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:349)
org.apache.hadoop.hbase.ipc.BalancedQueueRpcExecutor.dispatch(BalancedQueueRpcExecutor.java:67)
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.dispatch(SimpleRpcScheduler.java:129)
org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequest(RpcServer.java:1747)
org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneRpc(RpcServer.java:1651)
org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1511)
org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:798)
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:589)
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:564)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
Thread 17 (RpcServer.reader=4,port=60020):
State: WAITING
Blocked count: 289
Waited count: 290
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@40e06e28
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:349)
org.apache.hadoop.hbase.ipc.BalancedQueueRpcExecutor.dispatch(BalancedQueueRpcExecutor.java:67)
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.dispatch(SimpleRpcScheduler.java:129)
org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequest(RpcServer.java:1747)
org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneRpc(RpcServer.java:1651)
org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1511)
org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:798)
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:589)
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:564)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
Thread 16 (RpcServer.reader=3,port=60020):
State: WAITING
Blocked count: 289
Waited count: 290
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@40e06e28
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:349)
org.apache.hadoop.hbase.ipc.BalancedQueueRpcExecutor.dispatch(BalancedQueueRpcExecutor.java:67)
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.dispatch(SimpleRpcScheduler.java:129)
org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequest(RpcServer.java:1747)
org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneRpc(RpcServer.java:1651)
org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1511)
org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:798)
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:589)
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:564)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
Thread 15 (RpcServer.reader=2,port=60020):
State: WAITING
Blocked count: 289
Waited count: 290
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@40e06e28
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:349)
org.apache.hadoop.hbase.ipc.BalancedQueueRpcExecutor.dispatch(BalancedQueueRpcExecutor.java:67)
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.dispatch(SimpleRpcScheduler.java:129)
org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequest(RpcServer.java:1747)
org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneRpc(RpcServer.java:1651)
org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1511)
org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:798)
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:589)
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:564)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
Thread 14 (RpcServer.reader=1,port=60020):
State: WAITING
Blocked count: 289
Waited count: 290
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@40e06e28
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:349)
org.apache.hadoop.hbase.ipc.BalancedQueueRpcExecutor.dispatch(BalancedQueueRpcExecutor.java:67)
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.dispatch(SimpleRpcScheduler.java:129)
org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequest(RpcServer.java:1747)
org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneRpc(RpcServer.java:1651)
org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1511)
org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:798)
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:589)
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:564)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
Thread 13 (RpcServer.reader=0,port=60020):
State: WAITING
Blocked count: 289
Waited count: 290
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@40e06e28
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:349)
org.apache.hadoop.hbase.ipc.BalancedQueueRpcExecutor.dispatch(BalancedQueueRpcExecutor.java:67)
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.dispatch(SimpleRpcScheduler.java:129)
org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequest(RpcServer.java:1747)
org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneRpc(RpcServer.java:1651)
org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1511)
org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:798)
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:589)
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:564)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
Thread 5 (Signal Dispatcher):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
Thread 3 (Finalizer):
State: WAITING
Blocked count: 442
Waited count: 107
Waiting on java.lang.ref.ReferenceQueue$Lock@3c3c270c
Stack:
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:189)
Thread 2 (Reference Handler):
State: WAITING
Blocked count: 131
Waited count: 112
Waiting on java.lang.ref.Reference$Lock@1127a3b9
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:503)
java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
Thread 1 (main):
State: WAITING
Blocked count: 3
Waited count: 2
Waiting on java.lang.Thread@20bcf877
Stack:
java.lang.Object.wait(Native Method)
java.lang.Thread.join(Thread.java:1280)
java.lang.Thread.join(Thread.java:1354)
org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:64)
org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:85)
org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2422)
RS Configuration:
===========================================================
<?xml version="1.0" encoding="UTF-8" standalone="no"?><configuration>
[snipp] no fancy stuff, all default, except absolute necessary settings
[snipp]
</configuration>
Logs
===========================================================
+++++++++++++++++++++++++++++++
/var/vm/apps/hbase-0.98.5-hadoop2/bin/../logs/hbase-visualmeta-regionserver-r3c7s4.log
+++++++++++++++++++++++++++++++
2014-08-22 14:52:46,121 INFO
[PriorityRpcServer.handler=4,queue=0,port=60020]
regionserver.HRegionServer: Close a07ecf6735212703fe83a93678a86225, via
zk=yes, znode version=0, on r3c3s4.colo.vm,60020,1408711455393
2014-08-22 14:52:46,131 INFO
[StoreCloserThread-image,nl\x00\x00\x00\x005\x1F\x1BN,1408356163124.682b252fdde4bb3bec3613e53bc9bec9.-1]
regionserver.HStore: Closed cf
2014-08-22 14:52:46,131 INFO [RS_CLOSE_REGION-r3c7s4:60020-0]
regionserver.HRegion: Closed
image,nl\x00\x00\x00\x005\x1F\x1BN,1408356163124.682b252fdde4bb3bec3613e53bc9bec9.
2014-08-22 14:52:46,131 INFO [RS_CLOSE_REGION-r3c7s4:60020-0]
regionserver.HRegionServer: Adding moved region record:
682b252fdde4bb3bec3613e53bc9bec9 to
r3c2s1.colo.vm,60020,1408711451959:60020 as of 857256765
2014-08-22 14:52:46,156 INFO
[PriorityRpcServer.handler=6,queue=0,port=60020]
regionserver.HRegionServer: Close 2dd5d2212b5e767aa2823c3af78b2ca7, via
zk=yes, znode version=0, on r2c9s4.colo.vm,60020,1408711451080
2014-08-22 14:52:46,161 INFO
[StoreCloserThread-image,de\x00\x00\x00\x02\x82\xA2z\xA3,1408191336257.a07ecf6735212703fe83a93678a86225.-1]
regionserver.HStore: Closed cf
2014-08-22 14:52:46,162 INFO [RS_CLOSE_REGION-r3c7s4:60020-1]
regionserver.HRegion: Closed
image,de\x00\x00\x00\x02\x82\xA2z\xA3,1408191336257.a07ecf6735212703fe83a93678a86225.
2014-08-22 14:52:46,162 INFO [RS_CLOSE_REGION-r3c7s4:60020-1]
regionserver.HRegionServer: Adding moved region record:
a07ecf6735212703fe83a93678a86225 to
r3c3s4.colo.vm,60020,1408711455393:60020 as of 859701098
2014-08-22 14:52:46,174 INFO
[PriorityRpcServer.handler=7,queue=0,port=60020]
regionserver.HRegionServer: Close db0fae8af8fcd271f2911387327df40a, via
zk=yes, znode version=0, on r2c10s2.colo.vm,60020,1408711444387
2014-08-22 14:52:46,183 INFO
[StoreCloserThread-image,de\x00\x00\x00\x02\x82\xAD\x9F],1408195161494.2dd5d2212b5e767aa2823c3af78b2ca7.-1]
regionserver.HStore: Closed cf
2014-08-22 14:52:46,183 INFO [RS_CLOSE_REGION-r3c7s4:60020-2]
regionserver.HRegion: Closed
image,de\x00\x00\x00\x02\x82\xAD\x9F],1408195161494.2dd5d2212b5e767aa2823c3af78b2ca7.
2014-08-22 14:52:46,183 INFO [RS_CLOSE_REGION-r3c7s4:60020-2]
regionserver.HRegionServer: Adding moved region record:
2dd5d2212b5e767aa2823c3af78b2ca7 to
r2c9s4.colo.vm,60020,1408711451080:60020 as of 860177691
2014-08-22 14:52:46,192 INFO
[PriorityRpcServer.handler=3,queue=0,port=60020]
regionserver.HRegionServer: Close e786e7c447b1614e69d46418c685a2cc, via
zk=yes, znode version=0, on r2c9s3.colo.vm,60020,1408711449726
2014-08-22 14:52:46,208 INFO
[StoreCloserThread-image,be\x00\x00\x00\x00\x01xK%,1397196499096.db0fae8af8fcd271f2911387327df40a.-1]
regionserver.HStore: Closed cf
2014-08-22 14:52:46,208 INFO [RS_CLOSE_REGION-r3c7s4:60020-0]
regionserver.HRegion: Closed
image,be\x00\x00\x00\x00\x01xK%,1397196499096.db0fae8af8fcd271f2911387327df40a.
2014-08-22 14:52:46,208 INFO [RS_CLOSE_REGION-r3c7s4:60020-0]
regionserver.HRegionServer: Adding moved region record:
db0fae8af8fcd271f2911387327df40a to
r2c10s2.colo.vm,60020,1408711444387:60020 as of 847254032
2014-08-22 14:52:46,228 INFO
[PriorityRpcServer.handler=9,queue=0,port=60020]
regionserver.HRegionServer: Close 72fb360847f0aa83ade3e54ca9061d5d, via
zk=yes, znode version=0, on r3c5s1.colo.vm,60020,1408711460148
2014-08-22 14:52:46,236 INFO
[StoreCloserThread-image,de\x00\x00\x00\x02G\x06\xF2\xF6,1399131972275.e786e7c447b1614e69d46418c685a2cc.-1]
regionserver.HStore: Closed cf
2014-08-22 14:52:46,236 INFO [RS_CLOSE_REGION-r3c7s4:60020-1]
regionserver.HRegion: Closed
image,de\x00\x00\x00\x02G\x06\xF2\xF6,1399131972275.e786e7c447b1614e69d46418c685a2cc.
2014-08-22 14:52:46,236 INFO [RS_CLOSE_REGION-r3c7s4:60020-1]
regionserver.HRegionServer: Adding moved region record:
e786e7c447b1614e69d46418c685a2cc to
r2c9s3.colo.vm,60020,1408711449726:60020 as of 821132653
2014-08-22 14:52:46,246 INFO
[PriorityRpcServer.handler=1,queue=0,port=60020]
regionserver.HRegionServer: Close 1335c0778d21018fb0c8c4c54f9213c1, via
zk=yes, znode version=0, on r3c5s3.colo.vm,60020,1408711481735
2014-08-22 14:52:46,261 INFO
[StoreCloserThread-image,de\x00\x00\x00\x02}`}\xD6,1406846742363.72fb360847f0aa83ade3e54ca9061d5d.-1]
regionserver.HStore: Closed cf
2014-08-22 14:52:46,261 INFO [RS_CLOSE_REGION-r3c7s4:60020-2]
regionserver.HRegion: Closed
image,de\x00\x00\x00\x02}`}\xD6,1406846742363.72fb360847f0aa83ade3e54ca9061d5d.
2014-08-22 14:52:46,261 INFO [RS_CLOSE_REGION-r3c7s4:60020-2]
regionserver.HRegionServer: Adding moved region record:
72fb360847f0aa83ade3e54ca9061d5d to
r3c5s1.colo.vm,60020,1408711460148:60020 as of 809285789
2014-08-22 14:52:46,277 INFO
[PriorityRpcServer.handler=8,queue=0,port=60020]
regionserver.HRegionServer: Close bba160a46f0a581c5c7cf9360e167d7e, via
zk=yes, znode version=0, on r2c9s4.colo.vm,60020,1408711451080
2014-08-22 14:52:46,278 INFO
[StoreCloserThread-image,de\x00\x00\x00\x02w\xCAn\x0A,1405579293102.1335c0778d21018fb0c8c4c54f9213c1.-1]
regionserver.HStore: Closed cf
2014-08-22 14:52:46,278 INFO [RS_CLOSE_REGION-r3c7s4:60020-0]
regionserver.HRegion: Closed
image,de\x00\x00\x00\x02w\xCAn\x0A,1405579293102.1335c0778d21018fb0c8c4c54f9213c1.
2014-08-22 14:52:46,278 INFO [RS_CLOSE_REGION-r3c7s4:60020-0]
regionserver.HRegionServer: Adding moved region record:
1335c0778d21018fb0c8c4c54f9213c1 to
r3c5s3.colo.vm,60020,1408711481735:60020 as of 826458371
2014-08-22 14:52:46,296 INFO
[PriorityRpcServer.handler=0,queue=0,port=60020]
regionserver.HRegionServer: Close 3ad114f3546150f1b2caee963e7d2618, via
zk=yes, znode version=0, on r3c7s2.colo.vm,60020,1408711484277
2014-08-22 14:52:46,313 INFO
[StoreCloserThread-image,de\x00\x00\x00\x02\x82\x03\x05M,1408069202259.bba160a46f0a581c5c7cf9360e167d7e.-1]
regionserver.HStore: Closed cf
2014-08-22 14:52:46,313 INFO [RS_CLOSE_REGION-r3c7s4:60020-1]
regionserver.HRegion: Closed
image,de\x00\x00\x00\x02\x82\x03\x05M,1408069202259.bba160a46f0a581c5c7cf9360e167d7e.
2014-08-22 14:52:46,313 INFO [RS_CLOSE_REGION-r3c7s4:60020-1]
regionserver.HRegionServer: Adding moved region record:
bba160a46f0a581c5c7cf9360e167d7e to
r2c9s4.colo.vm,60020,1408711451080:60020 as of 857601277
2014-08-22 14:52:46,353 INFO
[PriorityRpcServer.handler=2,queue=0,port=60020]
regionserver.HRegionServer: Close 14372e73c33d9b8c9848519ad628123e, via
zk=yes, znode version=0, on r3c2s1.colo.vm,60020,1408711451959
2014-08-22 14:52:46,357 INFO
[StoreCloserThread-image,de\x00\x00\x00\x02Gq\xBF\xCC,1399165548009.3ad114f3546150f1b2caee963e7d2618.-1]
regionserver.HStore: Closed cf
2014-08-22 14:52:46,358 INFO [RS_CLOSE_REGION-r3c7s4:60020-2]
regionserver.HRegion: Closed
image,de\x00\x00\x00\x02Gq\xBF\xCC,1399165548009.3ad114f3546150f1b2caee963e7d2618.
2014-08-22 14:52:46,358 INFO [RS_CLOSE_REGION-r3c7s4:60020-2]
regionserver.HRegionServer: Adding moved region record:
3ad114f3546150f1b2caee963e7d2618 to
r3c7s2.colo.vm,60020,1408711484277:60020 as of 826418599
2014-08-22 14:52:46,395 INFO
[PriorityRpcServer.handler=5,queue=0,port=60020]
regionserver.HRegionServer: Close 23ba7e5b9354d05a71c526d2fa0ecd47, via
zk=yes, znode version=0, on r3c5s3.colo.vm,60020,1408711481735
2014-08-22 14:52:46,408 INFO
[StoreCloserThread-image,it\x00\x00\x00\x00;\xBC\x18\x98,1405232618304.14372e73c33d9b8c9848519ad628123e.-1]
regionserver.HStore: Closed cf
2014-08-22 14:52:46,408 INFO [RS_CLOSE_REGION-r3c7s4:60020-0]
regionserver.HRegion: Closed
image,it\x00\x00\x00\x00;\xBC\x18\x98,1405232618304.14372e73c33d9b8c9848519ad628123e.
2014-08-22 14:52:46,409 INFO [RS_CLOSE_REGION-r3c7s4:60020-0]
regionserver.HRegionServer: Adding moved region record:
14372e73c33d9b8c9848519ad628123e to
r3c2s1.colo.vm,60020,1408711451959:60020 as of 857293958
2014-08-22 14:52:46,410 INFO
[StoreCloserThread-image,ru\x00\x00\x00\x00\x00\xF9\xB9\x9C,1400121006136.23ba7e5b9354d05a71c526d2fa0ecd47.-1]
regionserver.HStore: Closed cf
2014-08-22 14:52:46,411 INFO [RS_CLOSE_REGION-r3c7s4:60020-1]
regionserver.HRegion: Closed
image,ru\x00\x00\x00\x00\x00\xF9\xB9\x9C,1400121006136.23ba7e5b9354d05a71c526d2fa0ecd47.
2014-08-22 14:52:46,411 INFO [RS_CLOSE_REGION-r3c7s4:60020-1]
regionserver.HRegionServer: Adding moved region record:
23ba7e5b9354d05a71c526d2fa0ecd47 to
r3c5s3.colo.vm,60020,1408711481735:60020 as of 755730346
2014-08-22 14:52:46,419 INFO
[PriorityRpcServer.handler=4,queue=0,port=60020]
regionserver.HRegionServer: Close b26a52eeb129185aa3e2b7f5c6c7268c, via
zk=yes, znode version=0, on r2c10s2.colo.vm,60020,1408711444387
2014-08-22 14:52:46,450 INFO
[PriorityRpcServer.handler=6,queue=0,port=60020]
regionserver.HRegionServer: Close 9e6746f82fb374f0d74d57c1edcd8b79, via
zk=yes, znode version=0, on r3c7s3.colo.vm,60020,1408711499973
2014-08-22 14:52:46,459 INFO
[StoreCloserThread-image,fi\x00\x00\x00\x00\x09\xDC\x80o,1398909121598.b26a52eeb129185aa3e2b7f5c6c7268c.-1]
regionserver.HStore: Closed cf
2014-08-22 14:52:46,459 INFO [RS_CLOSE_REGION-r3c7s4:60020-2]
regionserver.HRegion: Closed
image,fi\x00\x00\x00\x00\x09\xDC\x80o,1398909121598.b26a52eeb129185aa3e2b7f5c6c7268c.
2014-08-22 14:52:46,459 INFO [RS_CLOSE_REGION-r3c7s4:60020-2]
regionserver.HRegionServer: Adding moved region record:
b26a52eeb129185aa3e2b7f5c6c7268c to
r2c10s2.colo.vm,60020,1408711444387:60020 as of 800831945
2014-08-22 14:52:46,482 INFO
[StoreCloserThread-image,de\x00\x00\x00\x02D\x0F\xE9\x94,1398936500039.9e6746f82fb374f0d74d57c1edcd8b79.-1]
regionserver.HStore: Closed cf
2014-08-22 14:52:46,482 INFO [RS_CLOSE_REGION-r3c7s4:60020-0]
regionserver.HRegion: Closed
image,de\x00\x00\x00\x02D\x0F\xE9\x94,1398936500039.9e6746f82fb374f0d74d57c1edcd8b79.
2014-08-22 14:52:46,482 INFO [RS_CLOSE_REGION-r3c7s4:60020-0]
regionserver.HRegionServer: Adding moved region record:
9e6746f82fb374f0d74d57c1edcd8b79 to
r3c7s3.colo.vm,60020,1408711499973:60020 as of 824230259
2014-08-22 14:52:55,284 INFO [regionserver60020.periodicFlusher]
regionserver.HRegionServer: regionserver60020.periodicFlusher requesting
flush for region
item,fr\x00\x00\x00\x00\x01\x14\xFB\xC6,1408559156558.357acd681a22c550ca1696dbe1e65b15.
after a delay of 17900
2014-08-22 14:53:05,284 INFO [regionserver60020.periodicFlusher]
regionserver.HRegionServer: regionserver60020.periodicFlusher requesting
flush for region
item,fr\x00\x00\x00\x00\x01\x14\xFB\xC6,1408559156558.357acd681a22c550ca1696dbe1e65b15.
after a delay of 20873
2014-08-22 14:53:13,304 INFO [MemStoreFlusher.0]
regionserver.DefaultStoreFlusher: Flushed, sequenceid=857612072,
memsize=18.4 K, hasBloomFilter=true, into tmp file
hdfs://visualmeta/hbase/data/default/item/357acd681a22c550ca1696dbe1e65b15/.tmp/3bb4beaea04c497d9fa4410247557ac3
2014-08-22 14:53:13,324 INFO [MemStoreFlusher.0] regionserver.HStore:
Added
hdfs://visualmeta/hbase/data/default/item/357acd681a22c550ca1696dbe1e65b15/cf_feature_vector/3bb4beaea04c497d9fa4410247557ac3,
entries=100, sequenceid=857612072, filesize=6.8 K
2014-08-22 14:53:13,325 INFO [MemStoreFlusher.0] regionserver.HRegion:
Finished memstore flush of ~18.4 K/18800, currentsize=0/0 for region
item,fr\x00\x00\x00\x00\x01\x14\xFB\xC6,1408559156558.357acd681a22c550ca1696dbe1e65b15.
in 141ms, sequenceid=857612072, compaction requested=true
2014-08-22 14:54:05,284 INFO [regionserver60020.periodicFlusher]
regionserver.HRegionServer: regionserver60020.periodicFlusher requesting
flush for region
classifier,\xE9\x02-\xF0\xA2\xDF\x17*,1408110936897.3129db940c9f94729b986b742f0161c2.
after a delay of 12130
2014-08-22 14:54:05,285 INFO [regionserver60020.periodicFlusher]
regionserver.HRegionServer: regionserver60020.periodicFlusher requesting
flush for region
classifier,%\xFD\xA3,1401198483217.82f8098871398b59ead1c6963781685d. after
a delay of 14728
2014-08-22 14:54:15,284 INFO [regionserver60020.periodicFlusher]
regionserver.HRegionServer: regionserver60020.periodicFlusher requesting
flush for region
classifier,\xE9\x02-\xF0\xA2\xDF\x17*,1408110936897.3129db940c9f94729b986b742f0161c2.
after a delay of 19426
2014-08-22 14:54:15,284 INFO [regionserver60020.periodicFlusher]
regionserver.HRegionServer: regionserver60020.periodicFlusher requesting
flush for region
classifier,%\xFD\xA3,1401198483217.82f8098871398b59ead1c6963781685d. after
a delay of 10422
2014-08-22 14:54:17,490 INFO [MemStoreFlusher.0]
regionserver.DefaultStoreFlusher: Flushed, sequenceid=830369741,
memsize=337.9 K, hasBloomFilter=true, into tmp file
hdfs://visualmeta/hbase/data/default/classifier/3129db940c9f94729b986b742f0161c2/.tmp/0d82d522fcd14c00bd91a8a964e7decf
2014-08-22 14:54:17,511 INFO [MemStoreFlusher.0] regionserver.HStore:
Added
hdfs://visualmeta/hbase/data/default/classifier/3129db940c9f94729b986b742f0161c2/cf/0d82d522fcd14c00bd91a8a964e7decf,
entries=5, sequenceid=830369741, filesize=338.4 K
2014-08-22 14:54:17,511 INFO [MemStoreFlusher.0] regionserver.HRegion:
Finished memstore flush of ~337.9 K/346008, currentsize=0/0 for region
classifier,\xE9\x02-\xF0\xA2\xDF\x17*,1408110936897.3129db940c9f94729b986b742f0161c2.
in 97ms, sequenceid=830369741, compaction requested=true
2014-08-22 14:54:20,124 INFO [MemStoreFlusher.1]
regionserver.DefaultStoreFlusher: Flushed, sequenceid=857453886,
memsize=661.7 K, hasBloomFilter=true, into tmp file
hdfs://visualmeta/hbase/data/default/classifier/82f8098871398b59ead1c6963781685d/.tmp/d28487d6829e400c87879b1e666e469c
2014-08-22 14:54:20,199 INFO [MemStoreFlusher.1] regionserver.HStore:
Added
hdfs://visualmeta/hbase/data/default/classifier/82f8098871398b59ead1c6963781685d/cf/d28487d6829e400c87879b1e666e469c,
entries=7, sequenceid=857453886, filesize=662.1 K
2014-08-22 14:54:20,199 INFO [MemStoreFlusher.1] regionserver.HRegion:
Finished memstore flush of ~661.7 K/677544, currentsize=0/0 for region
classifier,%\xFD\xA3,1401198483217.82f8098871398b59ead1c6963781685d. in
186ms, sequenceid=857453886, compaction requested=true
2014-08-22 14:54:20,200 INFO
[regionserver60020-smallCompactions-1408707426607] regionserver.HRegion:
Starting compaction on cf in region
classifier,%\xFD\xA3,1401198483217.82f8098871398b59ead1c6963781685d.
2014-08-22 14:54:20,200 INFO
[regionserver60020-smallCompactions-1408707426607] regionserver.HStore:
Starting compaction of 3 file(s) in cf of
classifier,%\xFD\xA3,1401198483217.82f8098871398b59ead1c6963781685d. into
tmpdir=hdfs://visualmeta/hbase/data/default/classifier/82f8098871398b59ead1c6963781685d/.tmp,
totalSize=46.1 M
2014-08-22 14:54:21,242 INFO
[regionserver60020-smallCompactions-1408707426607] regionserver.HStore:
Completed compaction of 3 file(s) in cf of
classifier,%\xFD\xA3,1401198483217.82f8098871398b59ead1c6963781685d. into
22f1215d31f34490aaf2465a0414c16f(size=45.4 M), total size for store is
633.1 M. This selection was in queue for 0sec, and took 1sec to execute.
2014-08-22 14:54:21,242 INFO
[regionserver60020-smallCompactions-1408707426607]
regionserver.CompactSplitThread: Completed compaction: Request =
regionName=classifier,%\xFD\xA3,1401198483217.82f8098871398b59ead1c6963781685d.,
storeName=cf, fileCount=3, fileSize=46.1 M, priority=5,
time=2525894161984378; duration=1sec
2014-08-22 15:00:53,222 INFO [regionserver60020-EventThread]
replication.ReplicationTrackerZKImpl:
/hbase/rs/r1c14s3.colo.vm,60020,1408688739222 znode expired, triggering
replicatorRemoved event
2014-08-22 15:00:53,744 INFO
[PriorityRpcServer.handler=7,queue=0,port=60020]
regionserver.HRegionServer: Open
classifier,\xCA\x90\xCD,1408632541341.c4b8745f4427e86ec1668eabe6adea1a.
2014-08-22 15:00:53,769 INFO
[PriorityRpcServer.handler=7,queue=0,port=60020]
regionserver.HRegionServer: Open
image,de\x00\x00\x00\x02Q\xA4\xA4\xCF,1400622870851.0606a108cfb00a6954eae8c84bc39669.
2014-08-22 15:00:53,829 INFO
[PriorityRpcServer.handler=7,queue=0,port=60020]
regionserver.HRegionServer: Open
image,de\x00\x00\x00\x02{\x04>=,1406099001127.7d62f7f7139aea672d70e0e78f6edfda.
2014-08-22 15:00:53,859 INFO
[PriorityRpcServer.handler=7,queue=0,port=60020]
regionserver.HRegionServer: Open
image,dk\x00\x00\x00\x00u\xB5\xB4\x1E,1405615660019.02aa0832f54f795aad7f4016a3a87a99.
2014-08-22 15:00:53,885 INFO
[PriorityRpcServer.handler=7,queue=0,port=60020]
regionserver.HRegionServer: Open
image,fi\x00\x00\x00\x00\x0A~\xFA\x08,1400494510921.2b16fc52a132709296646ebc00b2f616.
2014-08-22 15:00:53,900 INFO
[StoreOpener-c4b8745f4427e86ec1668eabe6adea1a-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:00:53,901 INFO
[StoreOpener-0606a108cfb00a6954eae8c84bc39669-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:00:53,911 INFO [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for cd296eb45a6e42698057e83c21c8a44c
2014-08-22 15:00:53,928 INFO [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for cf2bbb2640d549d0aaab73150303c9af
2014-08-22 15:00:53,929 INFO [RS_OPEN_REGION-r3c7s4:60020-1]
regionserver.HRegion: Onlined c4b8745f4427e86ec1668eabe6adea1a; next
sequenceid=854334330
2014-08-22 15:00:53,935 INFO [RS_OPEN_REGION-r3c7s4:60020-2]
regionserver.HRegion: Onlined 0606a108cfb00a6954eae8c84bc39669; next
sequenceid=826450913
2014-08-22 15:00:53,980 INFO
[PriorityRpcServer.handler=7,queue=0,port=60020]
regionserver.HRegionServer: Open
image,fr\x00\x00\x00\x00\xA4x6\x0B,1405342105031.f4de800d88fb4bd41eb746f9d4b747be.
2014-08-22 15:00:53,983 INFO
[StoreOpener-7d62f7f7139aea672d70e0e78f6edfda-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:00:53,987 INFO [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 7423e1fa193a4ee9858acc0b44d5cabd
2014-08-22 15:00:53,989 INFO
[PostOpenDeployTasks:c4b8745f4427e86ec1668eabe6adea1a]
regionserver.HRegionServer: Post open deploy tasks for
region=classifier,\xCA\x90\xCD,1408632541341.c4b8745f4427e86ec1668eabe6adea1a.
2014-08-22 15:00:53,989 INFO
[PostOpenDeployTasks:0606a108cfb00a6954eae8c84bc39669]
regionserver.HRegionServer: Post open deploy tasks for
region=image,de\x00\x00\x00\x02Q\xA4\xA4\xCF,1400622870851.0606a108cfb00a6954eae8c84bc39669.
2014-08-22 15:00:53,989 INFO
[PriorityRpcServer.handler=7,queue=0,port=60020]
regionserver.HRegionServer: Open
image,it\x00\x00\x00\x006]x\x82,1401401129195.a440630ea65a477b027cf6dcfcaca548.
2014-08-22 15:00:53,989 INFO
[regionserver60020-smallCompactions-1408707426607] regionserver.HRegion:
Starting compaction on cf in region
classifier,\xCA\x90\xCD,1408632541341.c4b8745f4427e86ec1668eabe6adea1a.
2014-08-22 15:00:53,990 INFO
[regionserver60020-smallCompactions-1408707426607] regionserver.HStore:
Starting compaction of 3 file(s) in cf of
classifier,\xCA\x90\xCD,1408632541341.c4b8745f4427e86ec1668eabe6adea1a.
into
tmpdir=hdfs://visualmeta/hbase/data/default/classifier/c4b8745f4427e86ec1668eabe6adea1a/.tmp,
totalSize=997.9 K
2014-08-22 15:00:54,016 INFO
[PriorityRpcServer.handler=7,queue=0,port=60020]
regionserver.HRegionServer: Open
image,pl\x00\x00\x00\x00\x06\x0A\x0A\xB1,1404864856073.e8f628a9e05b5c425e1bceb90e407016.
2014-08-22 15:00:54,020 INFO
[PostOpenDeployTasks:c4b8745f4427e86ec1668eabe6adea1a] catalog.MetaEditor:
Updated row
classifier,\xCA\x90\xCD,1408632541341.c4b8745f4427e86ec1668eabe6adea1a.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:00:54,020 INFO
[PostOpenDeployTasks:c4b8745f4427e86ec1668eabe6adea1a]
regionserver.HRegionServer: Finished post open deploy task for
classifier,\xCA\x90\xCD,1408632541341.c4b8745f4427e86ec1668eabe6adea1a.
2014-08-22 15:00:54,020 INFO
[PostOpenDeployTasks:0606a108cfb00a6954eae8c84bc39669] catalog.MetaEditor:
Updated row
image,de\x00\x00\x00\x02Q\xA4\xA4\xCF,1400622870851.0606a108cfb00a6954eae8c84bc39669.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:00:54,021 INFO
[PostOpenDeployTasks:0606a108cfb00a6954eae8c84bc39669]
regionserver.HRegionServer: Finished post open deploy task for
image,de\x00\x00\x00\x02Q\xA4\xA4\xCF,1400622870851.0606a108cfb00a6954eae8c84bc39669.
2014-08-22 15:00:54,038 INFO
[PriorityRpcServer.handler=7,queue=0,port=60020]
regionserver.HRegionServer: Open
image,se\x00\x00\x00\x00-\x8C\xD2},1407897158298.deabfc61edb34a1c8ee5366a467f15cd.
2014-08-22 15:00:54,059 INFO [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for fd14404a2b524d99a7584a1847f6d453
2014-08-22 15:00:54,072 INFO [RS_OPEN_REGION-r3c7s4:60020-0]
regionserver.HRegion: Onlined 7d62f7f7139aea672d70e0e78f6edfda; next
sequenceid=821164840
2014-08-22 15:00:54,117 INFO
[PostOpenDeployTasks:7d62f7f7139aea672d70e0e78f6edfda]
regionserver.HRegionServer: Post open deploy tasks for
region=image,de\x00\x00\x00\x02{\x04>=,1406099001127.7d62f7f7139aea672d70e0e78f6edfda.
2014-08-22 15:00:54,151 INFO
[StoreOpener-02aa0832f54f795aad7f4016a3a87a99-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:00:54,166 INFO
[PostOpenDeployTasks:7d62f7f7139aea672d70e0e78f6edfda] catalog.MetaEditor:
Updated row
image,de\x00\x00\x00\x02{\x04>=,1406099001127.7d62f7f7139aea672d70e0e78f6edfda.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:00:54,166 INFO
[PostOpenDeployTasks:7d62f7f7139aea672d70e0e78f6edfda]
regionserver.HRegionServer: Finished post open deploy task for
image,de\x00\x00\x00\x02{\x04>=,1406099001127.7d62f7f7139aea672d70e0e78f6edfda.
2014-08-22 15:00:54,170 INFO [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 50040f72f62e49909e9049218b1ca092
2014-08-22 15:00:54,183 INFO
[StoreOpener-2b16fc52a132709296646ebc00b2f616-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:00:54,184 INFO [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 9941517d71e94f85ba721053ff58d8d1
2014-08-22 15:00:54,193 INFO [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 5c668fc88da548c4873300850a53837d
2014-08-22 15:00:54,195 INFO [RS_OPEN_REGION-r3c7s4:60020-2]
regionserver.HRegion: Onlined 2b16fc52a132709296646ebc00b2f616; next
sequenceid=824869877
2014-08-22 15:00:54,210 INFO
[PostOpenDeployTasks:2b16fc52a132709296646ebc00b2f616]
regionserver.HRegionServer: Post open deploy tasks for
region=image,fi\x00\x00\x00\x00\x0A~\xFA\x08,1400494510921.2b16fc52a132709296646ebc00b2f616.
2014-08-22 15:00:54,223 INFO [RS_OPEN_REGION-r3c7s4:60020-1]
regionserver.HRegion: Onlined 02aa0832f54f795aad7f4016a3a87a99; next
sequenceid=826595682
2014-08-22 15:00:54,232 INFO
[PostOpenDeployTasks:02aa0832f54f795aad7f4016a3a87a99]
regionserver.HRegionServer: Post open deploy tasks for
region=image,dk\x00\x00\x00\x00u\xB5\xB4\x1E,1405615660019.02aa0832f54f795aad7f4016a3a87a99.
2014-08-22 15:00:54,234 INFO
[regionserver60020-smallCompactions-1408707426607] regionserver.HStore:
Completed compaction of 3 file(s) in cf of
classifier,\xCA\x90\xCD,1408632541341.c4b8745f4427e86ec1668eabe6adea1a.
into 4ce2bb9edbe240b1b439e433e4fcd352(size=916.2 K), total size for store
is 433.3 M. This selection was in queue for 0sec, and took 0sec to execute.
2014-08-22 15:00:54,234 INFO
[regionserver60020-smallCompactions-1408707426607]
regionserver.CompactSplitThread: Completed compaction: Request =
regionName=classifier,\xCA\x90\xCD,1408632541341.c4b8745f4427e86ec1668eabe6adea1a.,
storeName=cf, fileCount=3, fileSize=997.9 K, priority=6,
time=2526287950921618; duration=0sec
2014-08-22 15:00:54,235 INFO
[PostOpenDeployTasks:2b16fc52a132709296646ebc00b2f616] catalog.MetaEditor:
Updated row
image,fi\x00\x00\x00\x00\x0A~\xFA\x08,1400494510921.2b16fc52a132709296646ebc00b2f616.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:00:54,236 INFO
[PostOpenDeployTasks:2b16fc52a132709296646ebc00b2f616]
regionserver.HRegionServer: Finished post open deploy task for
image,fi\x00\x00\x00\x00\x0A~\xFA\x08,1400494510921.2b16fc52a132709296646ebc00b2f616.
2014-08-22 15:00:54,269 INFO
[StoreOpener-f4de800d88fb4bd41eb746f9d4b747be-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:00:54,269 INFO
[PostOpenDeployTasks:02aa0832f54f795aad7f4016a3a87a99] catalog.MetaEditor:
Updated row
image,dk\x00\x00\x00\x00u\xB5\xB4\x1E,1405615660019.02aa0832f54f795aad7f4016a3a87a99.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:00:54,270 INFO
[PostOpenDeployTasks:02aa0832f54f795aad7f4016a3a87a99]
regionserver.HRegionServer: Finished post open deploy task for
image,dk\x00\x00\x00\x00u\xB5\xB4\x1E,1405615660019.02aa0832f54f795aad7f4016a3a87a99.
2014-08-22 15:00:54,278 INFO [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for fe60d458f1644cb893c28b265407a8b6
2014-08-22 15:00:54,280 INFO [RS_OPEN_REGION-r3c7s4:60020-0]
regionserver.HRegion: Onlined f4de800d88fb4bd41eb746f9d4b747be; next
sequenceid=827405090
2014-08-22 15:00:54,316 INFO
[PostOpenDeployTasks:f4de800d88fb4bd41eb746f9d4b747be]
regionserver.HRegionServer: Post open deploy tasks for
region=image,fr\x00\x00\x00\x00\xA4x6\x0B,1405342105031.f4de800d88fb4bd41eb746f9d4b747be.
2014-08-22 15:00:54,348 INFO
[PostOpenDeployTasks:f4de800d88fb4bd41eb746f9d4b747be] catalog.MetaEditor:
Updated row
image,fr\x00\x00\x00\x00\xA4x6\x0B,1405342105031.f4de800d88fb4bd41eb746f9d4b747be.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:00:54,348 INFO
[PostOpenDeployTasks:f4de800d88fb4bd41eb746f9d4b747be]
regionserver.HRegionServer: Finished post open deploy task for
image,fr\x00\x00\x00\x00\xA4x6\x0B,1405342105031.f4de800d88fb4bd41eb746f9d4b747be.
2014-08-22 15:00:54,349 INFO
[StoreOpener-a440630ea65a477b027cf6dcfcaca548-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:00:54,362 INFO
[StoreOpener-e8f628a9e05b5c425e1bceb90e407016-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:00:54,364 INFO [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 1d0bd533c1de4667a7b3d56c003ed461
2014-08-22 15:00:54,366 INFO [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 63d6d479b0a54aa98cbcaff47c3cce26
2014-08-22 15:00:54,368 INFO [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for d77fad667b56456c95c58dd5ed8012a6
2014-08-22 15:00:54,392 INFO [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for ae9264312faf4d7f97f8d6d801ac648e
2014-08-22 15:00:54,394 INFO [RS_OPEN_REGION-r3c7s4:60020-1]
regionserver.HRegion: Onlined e8f628a9e05b5c425e1bceb90e407016; next
sequenceid=826674750
2014-08-22 15:00:54,400 INFO [RS_OPEN_REGION-r3c7s4:60020-2]
regionserver.HRegion: Onlined a440630ea65a477b027cf6dcfcaca548; next
sequenceid=848079895
2014-08-22 15:00:54,408 INFO
[PostOpenDeployTasks:e8f628a9e05b5c425e1bceb90e407016]
regionserver.HRegionServer: Post open deploy tasks for
region=image,pl\x00\x00\x00\x00\x06\x0A\x0A\xB1,1404864856073.e8f628a9e05b5c425e1bceb90e407016.
2014-08-22 15:00:54,408 INFO
[PostOpenDeployTasks:a440630ea65a477b027cf6dcfcaca548]
regionserver.HRegionServer: Post open deploy tasks for
region=image,it\x00\x00\x00\x006]x\x82,1401401129195.a440630ea65a477b027cf6dcfcaca548.
2014-08-22 15:00:54,417 INFO
[StoreOpener-deabfc61edb34a1c8ee5366a467f15cd-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:00:54,425 INFO
[PostOpenDeployTasks:e8f628a9e05b5c425e1bceb90e407016] catalog.MetaEditor:
Updated row
image,pl\x00\x00\x00\x00\x06\x0A\x0A\xB1,1404864856073.e8f628a9e05b5c425e1bceb90e407016.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:00:54,425 INFO
[PostOpenDeployTasks:e8f628a9e05b5c425e1bceb90e407016]
regionserver.HRegionServer: Finished post open deploy task for
image,pl\x00\x00\x00\x00\x06\x0A\x0A\xB1,1404864856073.e8f628a9e05b5c425e1bceb90e407016.
2014-08-22 15:00:54,425 INFO
[PostOpenDeployTasks:a440630ea65a477b027cf6dcfcaca548] catalog.MetaEditor:
Updated row
image,it\x00\x00\x00\x006]x\x82,1401401129195.a440630ea65a477b027cf6dcfcaca548.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:00:54,425 INFO
[PostOpenDeployTasks:a440630ea65a477b027cf6dcfcaca548]
regionserver.HRegionServer: Finished post open deploy task for
image,it\x00\x00\x00\x006]x\x82,1401401129195.a440630ea65a477b027cf6dcfcaca548.
2014-08-22 15:00:54,441 INFO [RS_OPEN_REGION-r3c7s4:60020-0]
regionserver.HRegion: Onlined deabfc61edb34a1c8ee5366a467f15cd; next
sequenceid=833150761
2014-08-22 15:00:54,455 INFO
[PostOpenDeployTasks:deabfc61edb34a1c8ee5366a467f15cd]
regionserver.HRegionServer: Post open deploy tasks for
region=image,se\x00\x00\x00\x00-\x8C\xD2},1407897158298.deabfc61edb34a1c8ee5366a467f15cd.
2014-08-22 15:00:54,473 INFO
[PostOpenDeployTasks:deabfc61edb34a1c8ee5366a467f15cd] catalog.MetaEditor:
Updated row
image,se\x00\x00\x00\x00-\x8C\xD2},1407897158298.deabfc61edb34a1c8ee5366a467f15cd.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:00:54,473 INFO
[PostOpenDeployTasks:deabfc61edb34a1c8ee5366a467f15cd]
regionserver.HRegionServer: Finished post open deploy task for
image,se\x00\x00\x00\x00-\x8C\xD2},1407897158298.deabfc61edb34a1c8ee5366a467f15cd.
2014-08-22 15:00:55,627 INFO [ReplicationExecutor-0]
replication.ReplicationQueuesZKImpl: Moving
r1c14s3.colo.vm,60020,1408688739222's hlogs to my queue
2014-08-22 15:00:55,637 INFO [ReplicationExecutor-0]
replication.ReplicationQueuesZKImpl: Won't transfer the queue, another RS
took care of it because of: KeeperErrorCode = NoNode for
/hbase/replication/rs/r1c14s3.colo.vm,60020,1408688739222/lock
2014-08-22 15:02:58,116 INFO [regionserver60020-EventThread]
replication.ReplicationTrackerZKImpl:
/hbase/rs/r3c3s4.colo.vm,60020,1408711455393 znode expired, triggering
replicatorRemoved event
2014-08-22 15:02:58,674 INFO
[PriorityRpcServer.handler=3,queue=0,port=60020]
regionserver.HRegionServer: Open classifier,\xDD|
\x0B\xD6_X(,1404308250116.991ff9d5f42cba40ae71bcd17ea5000f.
2014-08-22 15:02:58,701 INFO
[PriorityRpcServer.handler=3,queue=0,port=60020]
regionserver.HRegionServer: Open
image,de\x00\x00\x00\x02H\xEF6\xBE,1400132008394.d4d5cf8c8d1c64a00a235df670036e01.
2014-08-22 15:02:58,731 INFO
[PriorityRpcServer.handler=3,queue=0,port=60020]
regionserver.HRegionServer: Open
image,de\x00\x00\x00\x02}\x1DfW,1406751533083.67dbe3fa51662bd6753df1e0c5314a92.
2014-08-22 15:02:58,746 INFO
[PriorityRpcServer.handler=3,queue=0,port=60020]
regionserver.HRegionServer: Open
image,es\x00\x00\x00\x00"Wfu,1402968397139.3de6d45475ae36aeb3cfa3b823ceaeba.
2014-08-22 15:02:58,749 INFO
[StoreOpener-991ff9d5f42cba40ae71bcd17ea5000f-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:02:58,760 INFO [RS_OPEN_REGION-r3c7s4:60020-1]
regionserver.HRegion: Onlined 991ff9d5f42cba40ae71bcd17ea5000f; next
sequenceid=826516917
2014-08-22 15:02:58,773 INFO
[PriorityRpcServer.handler=3,queue=0,port=60020]
regionserver.HRegionServer: Open
image,fr\x00\x00\x00\x00\x91\xA17\xFE,1398927754577.48c5d3d1a5b744f4d5d84bb2cd392986.
2014-08-22 15:02:58,773 INFO
[PostOpenDeployTasks:991ff9d5f42cba40ae71bcd17ea5000f]
regionserver.HRegionServer: Post open deploy tasks for
region=classifier,\xDD|
\x0B\xD6_X(,1404308250116.991ff9d5f42cba40ae71bcd17ea5000f.
2014-08-22 15:02:58,775 INFO
[StoreOpener-d4d5cf8c8d1c64a00a235df670036e01-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:02:58,780 INFO [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 02cf17c9c191440d8aac96205834569d
2014-08-22 15:02:58,791 INFO [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 9b2d695887454347b7c12a9edac977bc
2014-08-22 15:02:58,793 INFO [RS_OPEN_REGION-r3c7s4:60020-2]
regionserver.HRegion: Onlined d4d5cf8c8d1c64a00a235df670036e01; next
sequenceid=847122540
2014-08-22 15:02:58,806 INFO
[PriorityRpcServer.handler=3,queue=0,port=60020]
regionserver.HRegionServer: Open
image,fr\x00\x00\x00\x00\xAB\xE5\xD8\xA6,1408494911543.468e62d7a5e8eb41ed8fd2d25d5950b8.
2014-08-22 15:02:58,809 INFO
[StoreOpener-67dbe3fa51662bd6753df1e0c5314a92-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:02:58,810 INFO
[PostOpenDeployTasks:991ff9d5f42cba40ae71bcd17ea5000f] catalog.MetaEditor:
Updated row classifier,\xDD|
\x0B\xD6_X(,1404308250116.991ff9d5f42cba40ae71bcd17ea5000f. with
server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:02:58,810 INFO
[PostOpenDeployTasks:991ff9d5f42cba40ae71bcd17ea5000f]
regionserver.HRegionServer: Finished post open deploy task for
classifier,\xDD| \x0B\xD6_X(,1404308250116.991ff9d5f42cba40ae71bcd17ea5000f.
2014-08-22 15:02:58,812 INFO
[PostOpenDeployTasks:d4d5cf8c8d1c64a00a235df670036e01]
regionserver.HRegionServer: Post open deploy tasks for
region=image,de\x00\x00\x00\x02H\xEF6\xBE,1400132008394.d4d5cf8c8d1c64a00a235df670036e01.
2014-08-22 15:02:58,820 INFO [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 346cc92277bb4813a58d509b8c00ccd9
2014-08-22 15:02:58,821 INFO [RS_OPEN_REGION-r3c7s4:60020-0]
regionserver.HRegion: Onlined 67dbe3fa51662bd6753df1e0c5314a92; next
sequenceid=826459200
2014-08-22 15:02:58,833 INFO
[PriorityRpcServer.handler=3,queue=0,port=60020]
regionserver.HRegionServer: Open
image,it\x00\x00\x00\x008\xD6g\x8A,1402990061104.1bb84f733bb61e56f6fab8f0368d1f18.
2014-08-22 15:02:58,833 INFO
[PostOpenDeployTasks:67dbe3fa51662bd6753df1e0c5314a92]
regionserver.HRegionServer: Post open deploy tasks for
region=image,de\x00\x00\x00\x02}\x1DfW,1406751533083.67dbe3fa51662bd6753df1e0c5314a92.
2014-08-22 15:02:58,837 INFO
[PostOpenDeployTasks:d4d5cf8c8d1c64a00a235df670036e01] catalog.MetaEditor:
Updated row
image,de\x00\x00\x00\x02H\xEF6\xBE,1400132008394.d4d5cf8c8d1c64a00a235df670036e01.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:02:58,837 INFO
[PostOpenDeployTasks:d4d5cf8c8d1c64a00a235df670036e01]
regionserver.HRegionServer: Finished post open deploy task for
image,de\x00\x00\x00\x02H\xEF6\xBE,1400132008394.d4d5cf8c8d1c64a00a235df670036e01.
2014-08-22 15:02:58,863 INFO
[PriorityRpcServer.handler=3,queue=0,port=60020]
regionserver.HRegionServer: Open
image,nl\x00\x00\x00\x002\xD8\xA8$,1404928936490.21e6c12b729c76e144924f36f34cba4e.
2014-08-22 15:02:58,866 INFO
[PostOpenDeployTasks:67dbe3fa51662bd6753df1e0c5314a92] catalog.MetaEditor:
Updated row
image,de\x00\x00\x00\x02}\x1DfW,1406751533083.67dbe3fa51662bd6753df1e0c5314a92.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:02:58,866 INFO
[PostOpenDeployTasks:67dbe3fa51662bd6753df1e0c5314a92]
regionserver.HRegionServer: Finished post open deploy task for
image,de\x00\x00\x00\x02}\x1DfW,1406751533083.67dbe3fa51662bd6753df1e0c5314a92.
2014-08-22 15:02:58,872 INFO
[PriorityRpcServer.handler=3,queue=0,port=60020]
regionserver.HRegionServer: Open
image,ru\x00\x00\x00\x00\x0250\xBD,1404692062662.4af7a8fb3cd560d92c074bc1fce01ae6.
2014-08-22 15:02:58,896 INFO
[StoreOpener-3de6d45475ae36aeb3cfa3b823ceaeba-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:02:58,901 INFO [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 9fb48e6e462f49f290a0034c01661a11
2014-08-22 15:02:58,902 INFO
[PriorityRpcServer.handler=3,queue=0,port=60020]
regionserver.HRegionServer: Open
image,tr\x00\x00\x00\x00\x04\xB1f\x0C,1405919250032.60522dfa3cd1b4a2963c3a5aad56f06b.
2014-08-22 15:02:58,910 INFO [RS_OPEN_REGION-r3c7s4:60020-1]
regionserver.HRegion: Onlined 3de6d45475ae36aeb3cfa3b823ceaeba; next
sequenceid=826442691
2014-08-22 15:02:58,947 INFO
[PostOpenDeployTasks:3de6d45475ae36aeb3cfa3b823ceaeba]
regionserver.HRegionServer: Post open deploy tasks for
region=image,es\x00\x00\x00\x00"Wfu,1402968397139.3de6d45475ae36aeb3cfa3b823ceaeba.
2014-08-22 15:02:58,950 INFO
[StoreOpener-48c5d3d1a5b744f4d5d84bb2cd392986-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:02:58,958 INFO [RS_OPEN_REGION-r3c7s4:60020-2]
regionserver.HRegion: Onlined 48c5d3d1a5b744f4d5d84bb2cd392986; next
sequenceid=826605440
2014-08-22 15:02:58,980 INFO
[PostOpenDeployTasks:48c5d3d1a5b744f4d5d84bb2cd392986]
regionserver.HRegionServer: Post open deploy tasks for
region=image,fr\x00\x00\x00\x00\x91\xA17\xFE,1398927754577.48c5d3d1a5b744f4d5d84bb2cd392986.
2014-08-22 15:02:58,982 INFO
[PostOpenDeployTasks:3de6d45475ae36aeb3cfa3b823ceaeba] catalog.MetaEditor:
Updated row
image,es\x00\x00\x00\x00"Wfu,1402968397139.3de6d45475ae36aeb3cfa3b823ceaeba.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:02:58,982 INFO
[PostOpenDeployTasks:3de6d45475ae36aeb3cfa3b823ceaeba]
regionserver.HRegionServer: Finished post open deploy task for
image,es\x00\x00\x00\x00"Wfu,1402968397139.3de6d45475ae36aeb3cfa3b823ceaeba.
2014-08-22 15:02:58,983 INFO
[StoreOpener-468e62d7a5e8eb41ed8fd2d25d5950b8-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:02:59,016 INFO
[PostOpenDeployTasks:48c5d3d1a5b744f4d5d84bb2cd392986] catalog.MetaEditor:
Updated row
image,fr\x00\x00\x00\x00\x91\xA17\xFE,1398927754577.48c5d3d1a5b744f4d5d84bb2cd392986.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:02:59,016 INFO
[PostOpenDeployTasks:48c5d3d1a5b744f4d5d84bb2cd392986]
regionserver.HRegionServer: Finished post open deploy task for
image,fr\x00\x00\x00\x00\x91\xA17\xFE,1398927754577.48c5d3d1a5b744f4d5d84bb2cd392986.
2014-08-22 15:02:59,019 INFO [RS_OPEN_REGION-r3c7s4:60020-0]
regionserver.HRegion: Onlined 468e62d7a5e8eb41ed8fd2d25d5950b8; next
sequenceid=855595540
2014-08-22 15:02:59,046 INFO
[PostOpenDeployTasks:468e62d7a5e8eb41ed8fd2d25d5950b8]
regionserver.HRegionServer: Post open deploy tasks for
region=image,fr\x00\x00\x00\x00\xAB\xE5\xD8\xA6,1408494911543.468e62d7a5e8eb41ed8fd2d25d5950b8.
2014-08-22 15:02:59,076 INFO
[PostOpenDeployTasks:468e62d7a5e8eb41ed8fd2d25d5950b8] catalog.MetaEditor:
Updated row
image,fr\x00\x00\x00\x00\xAB\xE5\xD8\xA6,1408494911543.468e62d7a5e8eb41ed8fd2d25d5950b8.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:02:59,076 INFO
[PostOpenDeployTasks:468e62d7a5e8eb41ed8fd2d25d5950b8]
regionserver.HRegionServer: Finished post open deploy task for
image,fr\x00\x00\x00\x00\xAB\xE5\xD8\xA6,1408494911543.468e62d7a5e8eb41ed8fd2d25d5950b8.
2014-08-22 15:02:59,085 INFO
[StoreOpener-1bb84f733bb61e56f6fab8f0368d1f18-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:02:59,097 INFO [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for aabe8cf7bc0242fe9fc5a84cc6c9007f
2014-08-22 15:02:59,101 INFO [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for ce472d7873344b979a7228f40db343f9
2014-08-22 15:02:59,103 INFO [RS_OPEN_REGION-r3c7s4:60020-1]
regionserver.HRegion: Onlined 1bb84f733bb61e56f6fab8f0368d1f18; next
sequenceid=824267600
2014-08-22 15:02:59,113 INFO
[PostOpenDeployTasks:1bb84f733bb61e56f6fab8f0368d1f18]
regionserver.HRegionServer: Post open deploy tasks for
region=image,it\x00\x00\x00\x008\xD6g\x8A,1402990061104.1bb84f733bb61e56f6fab8f0368d1f18.
2014-08-22 15:02:59,115 INFO
[StoreOpener-21e6c12b729c76e144924f36f34cba4e-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:02:59,124 INFO [RS_OPEN_REGION-r3c7s4:60020-2]
regionserver.HRegion: Onlined 21e6c12b729c76e144924f36f34cba4e; next
sequenceid=824260645
2014-08-22 15:02:59,143 INFO
[PostOpenDeployTasks:21e6c12b729c76e144924f36f34cba4e]
regionserver.HRegionServer: Post open deploy tasks for
region=image,nl\x00\x00\x00\x002\xD8\xA8$,1404928936490.21e6c12b729c76e144924f36f34cba4e.
2014-08-22 15:02:59,145 INFO
[PostOpenDeployTasks:1bb84f733bb61e56f6fab8f0368d1f18] catalog.MetaEditor:
Updated row
image,it\x00\x00\x00\x008\xD6g\x8A,1402990061104.1bb84f733bb61e56f6fab8f0368d1f18.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:02:59,145 INFO
[PostOpenDeployTasks:1bb84f733bb61e56f6fab8f0368d1f18]
regionserver.HRegionServer: Finished post open deploy task for
image,it\x00\x00\x00\x008\xD6g\x8A,1402990061104.1bb84f733bb61e56f6fab8f0368d1f18.
2014-08-22 15:02:59,175 INFO
[StoreOpener-4af7a8fb3cd560d92c074bc1fce01ae6-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:02:59,175 INFO
[PostOpenDeployTasks:21e6c12b729c76e144924f36f34cba4e] catalog.MetaEditor:
Updated row
image,nl\x00\x00\x00\x002\xD8\xA8$,1404928936490.21e6c12b729c76e144924f36f34cba4e.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:02:59,175 INFO
[PostOpenDeployTasks:21e6c12b729c76e144924f36f34cba4e]
regionserver.HRegionServer: Finished post open deploy task for
image,nl\x00\x00\x00\x002\xD8\xA8$,1404928936490.21e6c12b729c76e144924f36f34cba4e.
2014-08-22 15:02:59,184 INFO [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 54b1f0ac77d04fb19b45c6d15f97e1e2
2014-08-22 15:02:59,189 INFO [RS_OPEN_REGION-r3c7s4:60020-0]
regionserver.HRegion: Onlined 4af7a8fb3cd560d92c074bc1fce01ae6; next
sequenceid=795162564
2014-08-22 15:02:59,209 INFO
[PostOpenDeployTasks:4af7a8fb3cd560d92c074bc1fce01ae6]
regionserver.HRegionServer: Post open deploy tasks for
region=image,ru\x00\x00\x00\x00\x0250\xBD,1404692062662.4af7a8fb3cd560d92c074bc1fce01ae6.
2014-08-22 15:02:59,223 INFO
[PostOpenDeployTasks:4af7a8fb3cd560d92c074bc1fce01ae6] catalog.MetaEditor:
Updated row
image,ru\x00\x00\x00\x00\x0250\xBD,1404692062662.4af7a8fb3cd560d92c074bc1fce01ae6.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:02:59,223 INFO
[PostOpenDeployTasks:4af7a8fb3cd560d92c074bc1fce01ae6]
regionserver.HRegionServer: Finished post open deploy task for
image,ru\x00\x00\x00\x00\x0250\xBD,1404692062662.4af7a8fb3cd560d92c074bc1fce01ae6.
2014-08-22 15:02:59,223 INFO
[StoreOpener-60522dfa3cd1b4a2963c3a5aad56f06b-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:02:59,241 INFO [RS_OPEN_REGION-r3c7s4:60020-1]
regionserver.HRegion: Onlined 60522dfa3cd1b4a2963c3a5aad56f06b; next
sequenceid=826442022
2014-08-22 15:02:59,251 INFO
[PostOpenDeployTasks:60522dfa3cd1b4a2963c3a5aad56f06b]
regionserver.HRegionServer: Post open deploy tasks for
region=image,tr\x00\x00\x00\x00\x04\xB1f\x0C,1405919250032.60522dfa3cd1b4a2963c3a5aad56f06b.
2014-08-22 15:02:59,265 INFO
[PostOpenDeployTasks:60522dfa3cd1b4a2963c3a5aad56f06b] catalog.MetaEditor:
Updated row
image,tr\x00\x00\x00\x00\x04\xB1f\x0C,1405919250032.60522dfa3cd1b4a2963c3a5aad56f06b.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:02:59,265 INFO
[PostOpenDeployTasks:60522dfa3cd1b4a2963c3a5aad56f06b]
regionserver.HRegionServer: Finished post open deploy task for
image,tr\x00\x00\x00\x00\x04\xB1f\x0C,1405919250032.60522dfa3cd1b4a2963c3a5aad56f06b.
2014-08-22 15:03:00,231 INFO [ReplicationExecutor-0]
replication.ReplicationQueuesZKImpl: Moving
r3c3s4.colo.vm,60020,1408711455393's hlogs to my queue
2014-08-22 15:03:00,245 INFO [ReplicationExecutor-0]
replication.ReplicationQueuesZKImpl: Won't transfer the queue, another RS
took care of it because of: KeeperErrorCode = NoNode for
/hbase/replication/rs/r3c3s4.colo.vm,60020,1408711455393/lock
2014-08-22 15:03:34,296 INFO [regionserver60020-EventThread]
replication.ReplicationTrackerZKImpl:
/hbase/rs/r3c5s3.colo.vm,60020,1408711481735 znode expired, triggering
replicatorRemoved event
2014-08-22 15:03:34,823 INFO
[PriorityRpcServer.handler=9,queue=0,port=60020]
regionserver.HRegionServer: Open
item,de\x00\x00\x00\x00\x0EY\x01(,1408563380665.ab9367ef6041f7a756117b5e22b7d566.
2014-08-22 15:03:34,851 INFO
[PriorityRpcServer.handler=9,queue=0,port=60020]
regionserver.HRegionServer: Open
image,de\x00\x00\x00\x021\xCD\xDBh,1399930900713.246d9c429f38cd6c87ed1c15541be1c3.
2014-08-22 15:03:34,881 INFO
[PriorityRpcServer.handler=9,queue=0,port=60020]
regionserver.HRegionServer: Open
image,de\x00\x00\x00\x02u\xF7\x92\x10,1405380638297.3b16a20aafe6c66168fdfb4b9e71c7da.
2014-08-22 15:03:34,911 INFO
[PriorityRpcServer.handler=9,queue=0,port=60020]
regionserver.HRegionServer: Open
image,dk\x00\x00\x00\x00v5\x0E;,1406781032737.bb89220cb394a0e6e5d126513144bb1d.
2014-08-22 15:03:34,926 INFO
[StoreOpener-ab9367ef6041f7a756117b5e22b7d566-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:03:34,926 INFO
[StoreOpener-246d9c429f38cd6c87ed1c15541be1c3-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:03:34,936 INFO [RS_OPEN_REGION-r3c7s4:60020-0]
regionserver.HRegion: Onlined 246d9c429f38cd6c87ed1c15541be1c3; next
sequenceid=797536166
2014-08-22 15:03:34,941 INFO
[PriorityRpcServer.handler=9,queue=0,port=60020]
regionserver.HRegionServer: Open
image,fi\x00\x00\x00\x00\x0C\xAF\xB4u,1405990240037.d7d06b01882d3965f42f81ccfd6709da.
2014-08-22 15:03:34,943 INFO [StoreFileOpenerThread-cf_feature_vector-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 9ed6c933753e462698acc816b6b66477
2014-08-22 15:03:34,951 INFO
[PostOpenDeployTasks:246d9c429f38cd6c87ed1c15541be1c3]
regionserver.HRegionServer: Post open deploy tasks for
region=image,de\x00\x00\x00\x021\xCD\xDBh,1399930900713.246d9c429f38cd6c87ed1c15541be1c3.
2014-08-22 15:03:34,953 INFO
[StoreOpener-3b16a20aafe6c66168fdfb4b9e71c7da-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:03:34,956 INFO [StoreFileOpenerThread-cf_feature_vector-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for dfe815de2af646539b8f51fdbbe4b929
2014-08-22 15:03:34,958 INFO [RS_OPEN_REGION-r3c7s4:60020-2]
regionserver.HRegion: Onlined ab9367ef6041f7a756117b5e22b7d566; next
sequenceid=857629192
2014-08-22 15:03:34,964 INFO [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for deb5fa28c5de4e859614b014ff8f52a4
2014-08-22 15:03:34,966 INFO [RS_OPEN_REGION-r3c7s4:60020-1]
regionserver.HRegion: Onlined 3b16a20aafe6c66168fdfb4b9e71c7da; next
sequenceid=821191177
2014-08-22 15:03:34,971 INFO
[PriorityRpcServer.handler=9,queue=0,port=60020]
regionserver.HRegionServer: Open
image,fr\x00\x00\x00\x00\x9B\x7F\xC95,1402431337599.969547bff4ba635df24f3597a829ec78.
2014-08-22 15:03:34,981 INFO
[PostOpenDeployTasks:ab9367ef6041f7a756117b5e22b7d566]
regionserver.HRegionServer: Post open deploy tasks for
region=item,de\x00\x00\x00\x00\x0EY\x01(,1408563380665.ab9367ef6041f7a756117b5e22b7d566.
2014-08-22 15:03:34,981 INFO
[PriorityRpcServer.handler=9,queue=0,port=60020]
regionserver.HRegionServer: Open
image,in\x00\x00\x00\x00\x01F\xE73,1400549232558.eb0c6870c459ffe5d21ff223b9169b56.
2014-08-22 15:03:34,981 INFO
[PostOpenDeployTasks:3b16a20aafe6c66168fdfb4b9e71c7da]
regionserver.HRegionServer: Post open deploy tasks for
region=image,de\x00\x00\x00\x02u\xF7\x92\x10,1405380638297.3b16a20aafe6c66168fdfb4b9e71c7da.
2014-08-22 15:03:34,983 INFO
[PostOpenDeployTasks:246d9c429f38cd6c87ed1c15541be1c3] catalog.MetaEditor:
Updated row
image,de\x00\x00\x00\x021\xCD\xDBh,1399930900713.246d9c429f38cd6c87ed1c15541be1c3.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:03:34,983 INFO
[PostOpenDeployTasks:246d9c429f38cd6c87ed1c15541be1c3]
regionserver.HRegionServer: Finished post open deploy task for
image,de\x00\x00\x00\x021\xCD\xDBh,1399930900713.246d9c429f38cd6c87ed1c15541be1c3.
2014-08-22 15:03:35,001 INFO
[PriorityRpcServer.handler=9,queue=0,port=60020]
regionserver.HRegionServer: Open
image,it\x00\x00\x00\x00<\xC9H\xE7,1406031094658.50f9d7e3b62898fcea5b306b59474db6.
2014-08-22 15:03:35,003 INFO
[PostOpenDeployTasks:ab9367ef6041f7a756117b5e22b7d566] catalog.MetaEditor:
Updated row
item,de\x00\x00\x00\x00\x0EY\x01(,1408563380665.ab9367ef6041f7a756117b5e22b7d566.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:03:35,003 INFO
[PostOpenDeployTasks:ab9367ef6041f7a756117b5e22b7d566]
regionserver.HRegionServer: Finished post open deploy task for
item,de\x00\x00\x00\x00\x0EY\x01(,1408563380665.ab9367ef6041f7a756117b5e22b7d566.
2014-08-22 15:03:35,006 INFO
[PostOpenDeployTasks:3b16a20aafe6c66168fdfb4b9e71c7da] catalog.MetaEditor:
Updated row
image,de\x00\x00\x00\x02u\xF7\x92\x10,1405380638297.3b16a20aafe6c66168fdfb4b9e71c7da.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:03:35,006 INFO
[PostOpenDeployTasks:3b16a20aafe6c66168fdfb4b9e71c7da]
regionserver.HRegionServer: Finished post open deploy task for
image,de\x00\x00\x00\x02u\xF7\x92\x10,1405380638297.3b16a20aafe6c66168fdfb4b9e71c7da.
2014-08-22 15:03:35,044 INFO
[PriorityRpcServer.handler=9,queue=0,port=60020]
regionserver.HRegionServer: Open
image,ru\x00\x00\x00\x00\x00\xF9\xB9\x9C,1400121006136.23ba7e5b9354d05a71c526d2fa0ecd47.
2014-08-22 15:03:35,085 INFO
[PriorityRpcServer.handler=9,queue=0,port=60020]
regionserver.HRegionServer: Open
image,se\x00\x00\x00\x00,\xAA\xCD,,1407172090506.efb6f85fa856f07db6284d6aecde2c22.
2014-08-22 15:03:35,088 INFO
[StoreOpener-bb89220cb394a0e6e5d126513144bb1d-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:03:35,119 INFO [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 86c9ef0fa2dc4ad488d1f0d47d3bbf1a
2014-08-22 15:03:35,121 INFO [RS_OPEN_REGION-r3c7s4:60020-0]
regionserver.HRegion: Onlined bb89220cb394a0e6e5d126513144bb1d; next
sequenceid=826521768
2014-08-22 15:03:35,127 INFO
[PostOpenDeployTasks:bb89220cb394a0e6e5d126513144bb1d]
regionserver.HRegionServer: Post open deploy tasks for
region=image,dk\x00\x00\x00\x00v5\x0E;,1406781032737.bb89220cb394a0e6e5d126513144bb1d.
2014-08-22 15:03:35,142 INFO
[StoreOpener-d7d06b01882d3965f42f81ccfd6709da-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:03:35,152 INFO [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for fc3fbfe5d21649568ec29aae7399bd4c
2014-08-22 15:03:35,154 INFO [RS_OPEN_REGION-r3c7s4:60020-2]
regionserver.HRegion: Onlined d7d06b01882d3965f42f81ccfd6709da; next
sequenceid=826462229
2014-08-22 15:03:35,170 INFO
[PostOpenDeployTasks:bb89220cb394a0e6e5d126513144bb1d] catalog.MetaEditor:
Updated row
image,dk\x00\x00\x00\x00v5\x0E;,1406781032737.bb89220cb394a0e6e5d126513144bb1d.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:03:35,170 INFO
[PostOpenDeployTasks:bb89220cb394a0e6e5d126513144bb1d]
regionserver.HRegionServer: Finished post open deploy task for
image,dk\x00\x00\x00\x00v5\x0E;,1406781032737.bb89220cb394a0e6e5d126513144bb1d.
2014-08-22 15:03:35,171 INFO
[StoreOpener-969547bff4ba635df24f3597a829ec78-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:03:35,177 INFO [RS_OPEN_REGION-r3c7s4:60020-1]
regionserver.HRegion: Onlined 969547bff4ba635df24f3597a829ec78; next
sequenceid=825161857
2014-08-22 15:03:35,180 INFO
[PostOpenDeployTasks:d7d06b01882d3965f42f81ccfd6709da]
regionserver.HRegionServer: Post open deploy tasks for
region=image,fi\x00\x00\x00\x00\x0C\xAF\xB4u,1405990240037.d7d06b01882d3965f42f81ccfd6709da.
2014-08-22 15:03:35,226 INFO
[PostOpenDeployTasks:969547bff4ba635df24f3597a829ec78]
regionserver.HRegionServer: Post open deploy tasks for
region=image,fr\x00\x00\x00\x00\x9B\x7F\xC95,1402431337599.969547bff4ba635df24f3597a829ec78.
2014-08-22 15:03:35,240 INFO
[PostOpenDeployTasks:d7d06b01882d3965f42f81ccfd6709da] catalog.MetaEditor:
Updated row
image,fi\x00\x00\x00\x00\x0C\xAF\xB4u,1405990240037.d7d06b01882d3965f42f81ccfd6709da.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:03:35,240 INFO
[PostOpenDeployTasks:d7d06b01882d3965f42f81ccfd6709da]
regionserver.HRegionServer: Finished post open deploy task for
image,fi\x00\x00\x00\x00\x0C\xAF\xB4u,1405990240037.d7d06b01882d3965f42f81ccfd6709da.
2014-08-22 15:03:35,319 INFO
[PostOpenDeployTasks:969547bff4ba635df24f3597a829ec78] catalog.MetaEditor:
Updated row
image,fr\x00\x00\x00\x00\x9B\x7F\xC95,1402431337599.969547bff4ba635df24f3597a829ec78.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:03:35,319 INFO
[PostOpenDeployTasks:969547bff4ba635df24f3597a829ec78]
regionserver.HRegionServer: Finished post open deploy task for
image,fr\x00\x00\x00\x00\x9B\x7F\xC95,1402431337599.969547bff4ba635df24f3597a829ec78.
2014-08-22 15:03:35,413 INFO
[StoreOpener-eb0c6870c459ffe5d21ff223b9169b56-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:03:35,421 INFO [RS_OPEN_REGION-r3c7s4:60020-0]
regionserver.HRegion: Onlined eb0c6870c459ffe5d21ff223b9169b56; next
sequenceid=793193777
2014-08-22 15:03:35,448 INFO
[PostOpenDeployTasks:eb0c6870c459ffe5d21ff223b9169b56]
regionserver.HRegionServer: Post open deploy tasks for
region=image,in\x00\x00\x00\x00\x01F\xE73,1400549232558.eb0c6870c459ffe5d21ff223b9169b56.
2014-08-22 15:03:35,451 INFO
[StoreOpener-50f9d7e3b62898fcea5b306b59474db6-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:03:35,461 INFO [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 60256fe2d4c248aebdbbbe176924cbdb
2014-08-22 15:03:35,462 INFO [RS_OPEN_REGION-r3c7s4:60020-2]
regionserver.HRegion: Onlined 50f9d7e3b62898fcea5b306b59474db6; next
sequenceid=848082246
2014-08-22 15:03:35,476 INFO
[StoreOpener-23ba7e5b9354d05a71c526d2fa0ecd47-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:03:35,478 INFO
[PostOpenDeployTasks:50f9d7e3b62898fcea5b306b59474db6]
regionserver.HRegionServer: Post open deploy tasks for
region=image,it\x00\x00\x00\x00<\xC9H\xE7,1406031094658.50f9d7e3b62898fcea5b306b59474db6.
2014-08-22 15:03:35,480 INFO
[PostOpenDeployTasks:eb0c6870c459ffe5d21ff223b9169b56] catalog.MetaEditor:
Updated row
image,in\x00\x00\x00\x00\x01F\xE73,1400549232558.eb0c6870c459ffe5d21ff223b9169b56.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:03:35,480 INFO
[PostOpenDeployTasks:eb0c6870c459ffe5d21ff223b9169b56]
regionserver.HRegionServer: Finished post open deploy task for
image,in\x00\x00\x00\x00\x01F\xE73,1400549232558.eb0c6870c459ffe5d21ff223b9169b56.
2014-08-22 15:03:35,489 INFO [RS_OPEN_REGION-r3c7s4:60020-1]
regionserver.HRegion: Onlined 23ba7e5b9354d05a71c526d2fa0ecd47; next
sequenceid=755730346
2014-08-22 15:03:35,510 INFO
[PostOpenDeployTasks:23ba7e5b9354d05a71c526d2fa0ecd47]
regionserver.HRegionServer: Post open deploy tasks for
region=image,ru\x00\x00\x00\x00\x00\xF9\xB9\x9C,1400121006136.23ba7e5b9354d05a71c526d2fa0ecd47.
2014-08-22 15:03:35,512 INFO
[PostOpenDeployTasks:50f9d7e3b62898fcea5b306b59474db6] catalog.MetaEditor:
Updated row
image,it\x00\x00\x00\x00<\xC9H\xE7,1406031094658.50f9d7e3b62898fcea5b306b59474db6.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:03:35,512 INFO
[PostOpenDeployTasks:50f9d7e3b62898fcea5b306b59474db6]
regionserver.HRegionServer: Finished post open deploy task for
image,it\x00\x00\x00\x00<\xC9H\xE7,1406031094658.50f9d7e3b62898fcea5b306b59474db6.
2014-08-22 15:03:35,524 INFO
[PostOpenDeployTasks:23ba7e5b9354d05a71c526d2fa0ecd47] catalog.MetaEditor:
Updated row
image,ru\x00\x00\x00\x00\x00\xF9\xB9\x9C,1400121006136.23ba7e5b9354d05a71c526d2fa0ecd47.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:03:35,524 INFO
[PostOpenDeployTasks:23ba7e5b9354d05a71c526d2fa0ecd47]
regionserver.HRegionServer: Finished post open deploy task for
image,ru\x00\x00\x00\x00\x00\xF9\xB9\x9C,1400121006136.23ba7e5b9354d05a71c526d2fa0ecd47.
2014-08-22 15:03:35,561 INFO
[StoreOpener-efb6f85fa856f07db6284d6aecde2c22-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:03:35,566 INFO [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 63293fdc236d480daf03ff70bc34c174
2014-08-22 15:03:35,571 INFO [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 71760500775d4061825984efa6b0ab2d
2014-08-22 15:03:35,579 INFO [RS_OPEN_REGION-r3c7s4:60020-0]
regionserver.HRegion: Onlined efb6f85fa856f07db6284d6aecde2c22; next
sequenceid=844862826
2014-08-22 15:03:35,594 INFO
[PostOpenDeployTasks:efb6f85fa856f07db6284d6aecde2c22]
regionserver.HRegionServer: Post open deploy tasks for
region=image,se\x00\x00\x00\x00,\xAA\xCD,,1407172090506.efb6f85fa856f07db6284d6aecde2c22.
2014-08-22 15:03:35,614 INFO
[PostOpenDeployTasks:efb6f85fa856f07db6284d6aecde2c22] catalog.MetaEditor:
Updated row
image,se\x00\x00\x00\x00,\xAA\xCD,,1407172090506.efb6f85fa856f07db6284d6aecde2c22.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:03:35,614 INFO
[PostOpenDeployTasks:efb6f85fa856f07db6284d6aecde2c22]
regionserver.HRegionServer: Finished post open deploy task for
image,se\x00\x00\x00\x00,\xAA\xCD,,1407172090506.efb6f85fa856f07db6284d6aecde2c22.
2014-08-22 15:03:36,631 INFO [ReplicationExecutor-0]
replication.ReplicationQueuesZKImpl: Moving
r3c5s3.colo.vm,60020,1408711481735's hlogs to my queue
2014-08-22 15:03:36,638 INFO [ReplicationExecutor-0]
replication.ReplicationQueuesZKImpl: Won't transfer the queue, another RS
took care of it because of: KeeperErrorCode = NoNode for
/hbase/replication/rs/r3c5s3.colo.vm,60020,1408711481735/lock
2014-08-22 15:08:45,292 INFO [regionserver60020.periodicFlusher]
regionserver.HRegionServer: regionserver60020.periodicFlusher requesting
flush for region
item,it\x00\x00\x00\x00\x05\xCC<X,1408619395176.19560c73dc6038a658d94ee627eabb90.
after a delay of 5511
2014-08-22 15:08:50,848 INFO [MemStoreFlusher.0]
regionserver.DefaultStoreFlusher: Flushed, sequenceid=827083295,
memsize=14.3 K, hasBloomFilter=true, into tmp file
hdfs://visualmeta/hbase/data/default/item/19560c73dc6038a658d94ee627eabb90/.tmp/65ae64d279cd43e7b8b05a597d5c9626
2014-08-22 15:08:50,866 INFO [MemStoreFlusher.0] regionserver.HStore:
Added
hdfs://visualmeta/hbase/data/default/item/19560c73dc6038a658d94ee627eabb90/cf_feature_vector/65ae64d279cd43e7b8b05a597d5c9626,
entries=78, sequenceid=827083295, filesize=5.5 K
2014-08-22 15:08:50,866 INFO [MemStoreFlusher.0] regionserver.HRegion:
Finished memstore flush of ~14.3 K/14664, currentsize=0/0 for region
item,it\x00\x00\x00\x00\x05\xCC<X,1408619395176.19560c73dc6038a658d94ee627eabb90.
in 62ms, sequenceid=827083295, compaction requested=true
2014-08-22 15:09:35,289 WARN
[defaultRpcServer.handler=23,queue=2,port=60020] hdfs.BlockReaderFactory:
I/O error constructing remote block reader.
java.io.IOException: Got error for OP_READ_BLOCK, self=/192.168.3.179:56326,
remote=/192.168.3.63:50010, for file
/hbase/data/default/image/1db3051c8d41943892c0230cb75bc1f2/cf/75ae96ea8ca2477e8f83291d7e1fe7cb,
for pool BP-1157637685-192.168.3.192-1382642140917 block
1293205473_253317737
at
org.apache.hadoop.hdfs.RemoteBlockReader2.checkSuccess(RemoteBlockReader2.java:432)
at
org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:397)
at
org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReader(BlockReaderFactory.java:786)
at
org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:665)
at
org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:325)
at
org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1025)
at
org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:968)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1301)
at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:90)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1223)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1430)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1312)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:387)
at
org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:253)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:524)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:572)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:257)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:173)
at
org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:55)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:313)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:269)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:695)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.seekAsDirection(StoreScanner.java:683)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:533)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:140)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3937)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:4017)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3885)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3867)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3854)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4827)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4801)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2849)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29583)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2026)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
at java.lang.Thread.run(Thread.java:744)
2014-08-22 15:09:35,289 WARN
[defaultRpcServer.handler=23,queue=2,port=60020] hdfs.DFSClient:
Connection failure: Failed to connect to /192.168.3.63:50010 for file
/hbase/data/default/image/1db3051c8d41943892c0230cb75bc1f2/cf/75ae96ea8ca2477e8f83291d7e1fe7cb
for block
BP-1157637685-192.168.3.192-1382642140917:blk_1293205473_253317737:java.io.IOException:
Got error for OP_READ_BLOCK, self=/192.168.3.179:56326, remote=/
192.168.3.63:50010, for file
/hbase/data/default/image/1db3051c8d41943892c0230cb75bc1f2/cf/75ae96ea8ca2477e8f83291d7e1fe7cb,
for pool BP-1157637685-192.168.3.192-1382642140917 block
1293205473_253317737
java.io.IOException: Got error for OP_READ_BLOCK, self=/192.168.3.179:56326,
remote=/192.168.3.63:50010, for file
/hbase/data/default/image/1db3051c8d41943892c0230cb75bc1f2/cf/75ae96ea8ca2477e8f83291d7e1fe7cb,
for pool BP-1157637685-192.168.3.192-1382642140917 block
1293205473_253317737
at
org.apache.hadoop.hdfs.RemoteBlockReader2.checkSuccess(RemoteBlockReader2.java:432)
at
org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:397)
at
org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReader(BlockReaderFactory.java:786)
at
org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:665)
at
org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:325)
at
org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1025)
at
org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:968)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1301)
at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:90)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1223)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1430)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1312)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:387)
at
org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:253)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:524)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:572)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:257)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:173)
at
org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:55)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:313)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:269)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:695)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.seekAsDirection(StoreScanner.java:683)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:533)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:140)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3937)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:4017)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3885)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3867)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3854)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4827)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4801)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2849)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29583)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2026)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
at java.lang.Thread.run(Thread.java:744)
2014-08-22 15:11:55,830 INFO [regionserver60020-EventThread]
replication.ReplicationTrackerZKImpl:
/hbase/rs/r3c3s3.colo.vm,60020,1408696671696 znode expired, triggering
replicatorRemoved event
2014-08-22 15:11:56,530 INFO
[PriorityRpcServer.handler=1,queue=0,port=60020]
regionserver.HRegionServer: Open item,de\x00\x00\x00\x00\x13\xD2G@
,1408558905273.aafeddc1719a1443b4106ff225b03f0c.
2014-08-22 15:11:56,555 INFO
[PriorityRpcServer.handler=1,queue=0,port=60020]
regionserver.HRegionServer: Open
image,de\x00\x00\x00\x00\xF7\x1F\xCF\x14,1395571953942.76d5202b21f739ed2cfd10f70bfb643f.
2014-08-22 15:11:56,585 INFO
[PriorityRpcServer.handler=1,queue=0,port=60020]
regionserver.HRegionServer: Open
image,de\x00\x00\x00\x02_\xC8\x94\x92,1402344949643.0be3775016616493c42e2b16cc9318dd.
2014-08-22 15:11:56,621 INFO
[PriorityRpcServer.handler=1,queue=0,port=60020]
regionserver.HRegionServer: Open
image,de\x00\x00\x00\x02\x82\x8Dr\x10,1408181214855.3e5f0743b69c4a1f5b0e45dbdb4b82ee.
2014-08-22 15:11:56,644 INFO
[StoreOpener-aafeddc1719a1443b4106ff225b03f0c-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:11:56,651 INFO
[PriorityRpcServer.handler=1,queue=0,port=60020]
regionserver.HRegionServer: Open
image,es\x00\x00\x00\x00%\xC1\xF7(,1407279563106.aacf1fbdad44db2c0438877a37ac78d5.
2014-08-22 15:11:56,655 INFO
[StoreOpener-76d5202b21f739ed2cfd10f70bfb643f-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:11:56,661 INFO [StoreFileOpenerThread-cf_feature_vector-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 501762388a0144ebb2a855fc635cdec3
2014-08-22 15:11:56,670 INFO [RS_OPEN_REGION-r3c7s4:60020-2]
regionserver.HRegion: Onlined aafeddc1719a1443b4106ff225b03f0c; next
sequenceid=827838722
2014-08-22 15:11:56,676 INFO [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 052722f3641e4fad84f8f8b99f8b29e1
2014-08-22 15:11:56,687 INFO
[PriorityRpcServer.handler=1,queue=0,port=60020]
regionserver.HRegionServer: Open
image,fr\x00\x00\x00\x00\x98\xC5\xD5\xD3,1401564947642.a8812708f97e06cb189c1a6df01473f5.
2014-08-22 15:11:56,688 INFO [RS_OPEN_REGION-r3c7s4:60020-1]
regionserver.HRegion: Onlined 76d5202b21f739ed2cfd10f70bfb643f; next
sequenceid=826511417
2014-08-22 15:11:56,691 INFO
[StoreOpener-0be3775016616493c42e2b16cc9318dd-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:11:56,694 INFO
[PostOpenDeployTasks:aafeddc1719a1443b4106ff225b03f0c]
regionserver.HRegionServer: Post open deploy tasks for
region=item,de\x00\x00\x00\x00\x13\xD2G@
,1408558905273.aafeddc1719a1443b4106ff225b03f0c.
2014-08-22 15:11:56,699 INFO [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 2d53061897524242a7db969444860ee9
2014-08-22 15:11:56,706 INFO [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for aa3ab7c419d940a48e5f221169a21298
2014-08-22 15:11:56,716 INFO
[PriorityRpcServer.handler=1,queue=0,port=60020]
regionserver.HRegionServer: Open
image,fr\x00\x00\x00\x00\xA8\xD8\xB7N,1407180602813.44f0f9f806475a2a68d20a4741b2d627.
2014-08-22 15:11:56,716 INFO
[PostOpenDeployTasks:76d5202b21f739ed2cfd10f70bfb643f]
regionserver.HRegionServer: Post open deploy tasks for
region=image,de\x00\x00\x00\x00\xF7\x1F\xCF\x14,1395571953942.76d5202b21f739ed2cfd10f70bfb643f.
2014-08-22 15:11:56,716 INFO [RS_OPEN_REGION-r3c7s4:60020-0]
regionserver.HRegion: Onlined 0be3775016616493c42e2b16cc9318dd; next
sequenceid=826424041
2014-08-22 15:11:56,726 INFO
[PostOpenDeployTasks:aafeddc1719a1443b4106ff225b03f0c] catalog.MetaEditor:
Updated row item,de\x00\x00\x00\x00\x13\xD2G@,1408558905273.aafeddc1719a1443b4106ff225b03f0c.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:11:56,727 INFO
[PostOpenDeployTasks:aafeddc1719a1443b4106ff225b03f0c]
regionserver.HRegionServer: Finished post open deploy task for
item,de\x00\x00\x00\x00\x13\xD2G@
,1408558905273.aafeddc1719a1443b4106ff225b03f0c.
2014-08-22 15:11:56,760 INFO
[PriorityRpcServer.handler=1,queue=0,port=60020]
regionserver.HRegionServer: Open
image,it\x00\x00\x00\x006\x04\xBD;,1401330620861.cbea316034f5a456e4f7866a988e56fb.
2014-08-22 15:11:56,760 INFO
[PostOpenDeployTasks:0be3775016616493c42e2b16cc9318dd]
regionserver.HRegionServer: Post open deploy tasks for
region=image,de\x00\x00\x00\x02_\xC8\x94\x92,1402344949643.0be3775016616493c42e2b16cc9318dd.
2014-08-22 15:11:56,762 INFO
[PostOpenDeployTasks:76d5202b21f739ed2cfd10f70bfb643f] catalog.MetaEditor:
Updated row
image,de\x00\x00\x00\x00\xF7\x1F\xCF\x14,1395571953942.76d5202b21f739ed2cfd10f70bfb643f.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:11:56,762 INFO
[PostOpenDeployTasks:76d5202b21f739ed2cfd10f70bfb643f]
regionserver.HRegionServer: Finished post open deploy task for
image,de\x00\x00\x00\x00\xF7\x1F\xCF\x14,1395571953942.76d5202b21f739ed2cfd10f70bfb643f.
2014-08-22 15:11:56,788 INFO
[PriorityRpcServer.handler=1,queue=0,port=60020]
regionserver.HRegionServer: Open
image,nl\x00\x00\x00\x002\xE1\xE6.,1405002295327.4bba527b4d574c7090e425e0f8f01969.
2014-08-22 15:11:56,790 INFO
[PostOpenDeployTasks:0be3775016616493c42e2b16cc9318dd] catalog.MetaEditor:
Updated row
image,de\x00\x00\x00\x02_\xC8\x94\x92,1402344949643.0be3775016616493c42e2b16cc9318dd.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:11:56,790 INFO
[PostOpenDeployTasks:0be3775016616493c42e2b16cc9318dd]
regionserver.HRegionServer: Finished post open deploy task for
image,de\x00\x00\x00\x02_\xC8\x94\x92,1402344949643.0be3775016616493c42e2b16cc9318dd.
2014-08-22 15:11:56,826 INFO
[PriorityRpcServer.handler=1,queue=0,port=60020]
regionserver.HRegionServer: Open
image,se\x00\x00\x00\x00)S~a,1401960422149.cdb390aa949c463b320d5835a5eb29da.
2014-08-22 15:11:56,860 INFO
[PriorityRpcServer.handler=1,queue=0,port=60020]
regionserver.HRegionServer: Open
image,tr\x00\x00\x00\x00\x04T\x84',1405349548090.61a54649329084e73b43c331caf1f403.
2014-08-22 15:11:56,871 INFO
[StoreOpener-3e5f0743b69c4a1f5b0e45dbdb4b82ee-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:11:56,882 INFO [RS_OPEN_REGION-r3c7s4:60020-2]
regionserver.HRegion: Onlined 3e5f0743b69c4a1f5b0e45dbdb4b82ee; next
sequenceid=858767012
2014-08-22 15:11:56,887 INFO
[StoreOpener-aacf1fbdad44db2c0438877a37ac78d5-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:11:56,892 INFO [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 327e010537294dd08fabf648116cdfee
2014-08-22 15:11:56,893 INFO
[PostOpenDeployTasks:3e5f0743b69c4a1f5b0e45dbdb4b82ee]
regionserver.HRegionServer: Post open deploy tasks for
region=image,de\x00\x00\x00\x02\x82\x8Dr\x10,1408181214855.3e5f0743b69c4a1f5b0e45dbdb4b82ee.
2014-08-22 15:11:56,897 INFO [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for b4339e4a561945eb845952109bf29e94
2014-08-22 15:11:56,904 INFO [RS_OPEN_REGION-r3c7s4:60020-1]
regionserver.HRegion: Onlined aacf1fbdad44db2c0438877a37ac78d5; next
sequenceid=800476243
2014-08-22 15:11:56,926 INFO
[PostOpenDeployTasks:aacf1fbdad44db2c0438877a37ac78d5]
regionserver.HRegionServer: Post open deploy tasks for
region=image,es\x00\x00\x00\x00%\xC1\xF7(,1407279563106.aacf1fbdad44db2c0438877a37ac78d5.
2014-08-22 15:11:56,928 INFO
[PostOpenDeployTasks:3e5f0743b69c4a1f5b0e45dbdb4b82ee] catalog.MetaEditor:
Updated row
image,de\x00\x00\x00\x02\x82\x8Dr\x10,1408181214855.3e5f0743b69c4a1f5b0e45dbdb4b82ee.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:11:56,928 INFO
[PostOpenDeployTasks:3e5f0743b69c4a1f5b0e45dbdb4b82ee]
regionserver.HRegionServer: Finished post open deploy task for
image,de\x00\x00\x00\x02\x82\x8Dr\x10,1408181214855.3e5f0743b69c4a1f5b0e45dbdb4b82ee.
2014-08-22 15:11:56,956 INFO
[StoreOpener-a8812708f97e06cb189c1a6df01473f5-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:11:56,963 INFO [RS_OPEN_REGION-r3c7s4:60020-0]
regionserver.HRegion: Onlined a8812708f97e06cb189c1a6df01473f5; next
sequenceid=828266015
2014-08-22 15:11:56,967 INFO
[PostOpenDeployTasks:aacf1fbdad44db2c0438877a37ac78d5] catalog.MetaEditor:
Updated row
image,es\x00\x00\x00\x00%\xC1\xF7(,1407279563106.aacf1fbdad44db2c0438877a37ac78d5.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:11:56,967 INFO
[PostOpenDeployTasks:aacf1fbdad44db2c0438877a37ac78d5]
regionserver.HRegionServer: Finished post open deploy task for
image,es\x00\x00\x00\x00%\xC1\xF7(,1407279563106.aacf1fbdad44db2c0438877a37ac78d5.
2014-08-22 15:11:56,976 INFO
[PostOpenDeployTasks:a8812708f97e06cb189c1a6df01473f5]
regionserver.HRegionServer: Post open deploy tasks for
region=image,fr\x00\x00\x00\x00\x98\xC5\xD5\xD3,1401564947642.a8812708f97e06cb189c1a6df01473f5.
2014-08-22 15:11:56,997 INFO
[PostOpenDeployTasks:a8812708f97e06cb189c1a6df01473f5] catalog.MetaEditor:
Updated row
image,fr\x00\x00\x00\x00\x98\xC5\xD5\xD3,1401564947642.a8812708f97e06cb189c1a6df01473f5.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:11:56,997 INFO
[PostOpenDeployTasks:a8812708f97e06cb189c1a6df01473f5]
regionserver.HRegionServer: Finished post open deploy task for
image,fr\x00\x00\x00\x00\x98\xC5\xD5\xD3,1401564947642.a8812708f97e06cb189c1a6df01473f5.
2014-08-22 15:11:57,028 INFO
[StoreOpener-44f0f9f806475a2a68d20a4741b2d627-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:11:57,033 INFO [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 269ec65f620947e1b65f16e5bbb7893a
2014-08-22 15:11:57,036 INFO [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for be8bdc230ac74f1c8dcd28c17882a0f9
2014-08-22 15:11:57,043 INFO [RS_OPEN_REGION-r3c7s4:60020-2]
regionserver.HRegion: Onlined 44f0f9f806475a2a68d20a4741b2d627; next
sequenceid=830213588
2014-08-22 15:11:57,058 INFO
[StoreOpener-cbea316034f5a456e4f7866a988e56fb-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:11:57,066 INFO
[PostOpenDeployTasks:44f0f9f806475a2a68d20a4741b2d627]
regionserver.HRegionServer: Post open deploy tasks for
region=image,fr\x00\x00\x00\x00\xA8\xD8\xB7N,1407180602813.44f0f9f806475a2a68d20a4741b2d627.
2014-08-22 15:11:57,067 INFO [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 6bde84cd3f1a420ab5c615e52adb7ec4
2014-08-22 15:11:57,075 INFO [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for a433503e47814feb95f63467cd112d37
2014-08-22 15:11:57,077 INFO [RS_OPEN_REGION-r3c7s4:60020-1]
regionserver.HRegion: Onlined cbea316034f5a456e4f7866a988e56fb; next
sequenceid=826598203
2014-08-22 15:11:57,089 INFO
[PostOpenDeployTasks:cbea316034f5a456e4f7866a988e56fb]
regionserver.HRegionServer: Post open deploy tasks for
region=image,it\x00\x00\x00\x006\x04\xBD;,1401330620861.cbea316034f5a456e4f7866a988e56fb.
2014-08-22 15:11:57,091 INFO
[PostOpenDeployTasks:44f0f9f806475a2a68d20a4741b2d627] catalog.MetaEditor:
Updated row
image,fr\x00\x00\x00\x00\xA8\xD8\xB7N,1407180602813.44f0f9f806475a2a68d20a4741b2d627.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:11:57,091 INFO
[PostOpenDeployTasks:44f0f9f806475a2a68d20a4741b2d627]
regionserver.HRegionServer: Finished post open deploy task for
image,fr\x00\x00\x00\x00\xA8\xD8\xB7N,1407180602813.44f0f9f806475a2a68d20a4741b2d627.
2014-08-22 15:11:57,092 INFO
[StoreOpener-4bba527b4d574c7090e425e0f8f01969-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:11:57,102 INFO [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 0f316ba775194f5f9235ed88cd2bc5d8
2014-08-22 15:11:57,105 INFO [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for a95360a0f3e348d99fe1b7a727fbe825
2014-08-22 15:11:57,106 INFO [RS_OPEN_REGION-r3c7s4:60020-0]
regionserver.HRegion: Onlined 4bba527b4d574c7090e425e0f8f01969; next
sequenceid=821172061
2014-08-22 15:11:57,116 INFO
[PostOpenDeployTasks:4bba527b4d574c7090e425e0f8f01969]
regionserver.HRegionServer: Post open deploy tasks for
region=image,nl\x00\x00\x00\x002\xE1\xE6.,1405002295327.4bba527b4d574c7090e425e0f8f01969.
2014-08-22 15:11:57,118 INFO
[PostOpenDeployTasks:cbea316034f5a456e4f7866a988e56fb] catalog.MetaEditor:
Updated row
image,it\x00\x00\x00\x006\x04\xBD;,1401330620861.cbea316034f5a456e4f7866a988e56fb.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:11:57,118 INFO
[PostOpenDeployTasks:cbea316034f5a456e4f7866a988e56fb]
regionserver.HRegionServer: Finished post open deploy task for
image,it\x00\x00\x00\x006\x04\xBD;,1401330620861.cbea316034f5a456e4f7866a988e56fb.
2014-08-22 15:11:57,134 INFO
[PostOpenDeployTasks:4bba527b4d574c7090e425e0f8f01969] catalog.MetaEditor:
Updated row
image,nl\x00\x00\x00\x002\xE1\xE6.,1405002295327.4bba527b4d574c7090e425e0f8f01969.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:11:57,134 INFO
[PostOpenDeployTasks:4bba527b4d574c7090e425e0f8f01969]
regionserver.HRegionServer: Finished post open deploy task for
image,nl\x00\x00\x00\x002\xE1\xE6.,1405002295327.4bba527b4d574c7090e425e0f8f01969.
2014-08-22 15:11:57,147 INFO
[StoreOpener-cdb390aa949c463b320d5835a5eb29da-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:11:57,168 INFO [RS_OPEN_REGION-r3c7s4:60020-2]
regionserver.HRegion: Onlined cdb390aa949c463b320d5835a5eb29da; next
sequenceid=826595433
2014-08-22 15:11:57,182 INFO
[StoreOpener-61a54649329084e73b43c331caf1f403-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:11:57,185 INFO
[PostOpenDeployTasks:cdb390aa949c463b320d5835a5eb29da]
regionserver.HRegionServer: Post open deploy tasks for
region=image,se\x00\x00\x00\x00)S~a,1401960422149.cdb390aa949c463b320d5835a5eb29da.
2014-08-22 15:11:57,197 INFO [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 91c192751cf346ae91678bee7b7d86d5
2014-08-22 15:11:57,198 INFO [RS_OPEN_REGION-r3c7s4:60020-1]
regionserver.HRegion: Onlined 61a54649329084e73b43c331caf1f403; next
sequenceid=795164401
2014-08-22 15:11:57,199 INFO
[PostOpenDeployTasks:cdb390aa949c463b320d5835a5eb29da] catalog.MetaEditor:
Updated row
image,se\x00\x00\x00\x00)S~a,1401960422149.cdb390aa949c463b320d5835a5eb29da.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:11:57,199 INFO
[PostOpenDeployTasks:cdb390aa949c463b320d5835a5eb29da]
regionserver.HRegionServer: Finished post open deploy task for
image,se\x00\x00\x00\x00)S~a,1401960422149.cdb390aa949c463b320d5835a5eb29da.
2014-08-22 15:11:57,218 INFO
[PostOpenDeployTasks:61a54649329084e73b43c331caf1f403]
regionserver.HRegionServer: Post open deploy tasks for
region=image,tr\x00\x00\x00\x00\x04T\x84',1405349548090.61a54649329084e73b43c331caf1f403.
2014-08-22 15:11:57,247 INFO
[PostOpenDeployTasks:61a54649329084e73b43c331caf1f403] catalog.MetaEditor:
Updated row
image,tr\x00\x00\x00\x00\x04T\x84',1405349548090.61a54649329084e73b43c331caf1f403.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:11:57,247 INFO
[PostOpenDeployTasks:61a54649329084e73b43c331caf1f403]
regionserver.HRegionServer: Finished post open deploy task for
image,tr\x00\x00\x00\x00\x04T\x84',1405349548090.61a54649329084e73b43c331caf1f403.
2014-08-22 15:11:57,950 INFO [ReplicationExecutor-0]
replication.ReplicationQueuesZKImpl: Moving
r3c3s3.colo.vm,60020,1408696671696's hlogs to my queue
2014-08-22 15:11:57,958 INFO [ReplicationExecutor-0]
replication.ReplicationQueuesZKImpl: Won't transfer the queue, another RS
took care of it because of: KeeperErrorCode = NoNode for
/hbase/replication/rs/r3c3s3.colo.vm,60020,1408696671696/lock
2014-08-22 15:12:39,348 INFO
[PriorityRpcServer.handler=8,queue=0,port=60020]
regionserver.HRegionServer: Close ab9367ef6041f7a756117b5e22b7d566, via
zk=yes, znode version=0, on r3c5s3.colo.vm,60020,1408712894162
2014-08-22 15:12:39,445 INFO [RS_CLOSE_REGION-r3c7s4:60020-1]
regionserver.DefaultStoreFlusher: Flushed, sequenceid=857629207,
memsize=5.1 K, hasBloomFilter=true, into tmp file
hdfs://visualmeta/hbase/data/default/item/ab9367ef6041f7a756117b5e22b7d566/.tmp/eb964aa729f94df5b3f55d7b9c56f5e2
2014-08-22 15:12:39,472 INFO [RS_CLOSE_REGION-r3c7s4:60020-1]
regionserver.HStore: Added
hdfs://visualmeta/hbase/data/default/item/ab9367ef6041f7a756117b5e22b7d566/cf_feature_vector/eb964aa729f94df5b3f55d7b9c56f5e2,
entries=28, sequenceid=857629207, filesize=2.6 K
2014-08-22 15:12:39,472 INFO [RS_CLOSE_REGION-r3c7s4:60020-1]
regionserver.HRegion: Finished memstore flush of ~5.1 K/5264,
currentsize=0/0 for region
item,de\x00\x00\x00\x00\x0EY\x01(,1408563380665.ab9367ef6041f7a756117b5e22b7d566.
in 111ms, sequenceid=857629207, compaction requested=true
2014-08-22 15:12:39,517 INFO
[StoreCloserThread-item,de\x00\x00\x00\x00\x0EY\x01(,1408563380665.ab9367ef6041f7a756117b5e22b7d566.-1]
regionserver.HStore: Closed cf_feature_vector
2014-08-22 15:12:39,518 INFO [RS_CLOSE_REGION-r3c7s4:60020-1]
regionserver.HRegion: Closed
item,de\x00\x00\x00\x00\x0EY\x01(,1408563380665.ab9367ef6041f7a756117b5e22b7d566.
2014-08-22 15:12:39,518 INFO [RS_CLOSE_REGION-r3c7s4:60020-1]
regionserver.HRegionServer: Adding moved region record:
ab9367ef6041f7a756117b5e22b7d566 to
r3c5s3.colo.vm,60020,1408712894162:60020 as of 857629192
2014-08-22 15:12:39,698 INFO
[PriorityRpcServer.handler=0,queue=0,port=60020]
regionserver.HRegionServer: Close 3129db940c9f94729b986b742f0161c2, via
zk=yes, znode version=0, on r3c5s3.colo.vm,60020,1408712894162
2014-08-22 15:12:39,710 INFO
[PriorityRpcServer.handler=2,queue=0,port=60020]
regionserver.HRegionServer: Close dc121bab8ed97c25f4f135c4f4f6fdcc, via
zk=yes, znode version=0, on r3c5s3.colo.vm,60020,1408712894162
2014-08-22 15:12:39,824 INFO [RS_CLOSE_REGION-r3c7s4:60020-2]
regionserver.DefaultStoreFlusher: Flushed, sequenceid=830369747,
memsize=111.6 K, hasBloomFilter=true, into tmp file
hdfs://visualmeta/hbase/data/default/classifier/3129db940c9f94729b986b742f0161c2/.tmp/1fb37a3ae1e346bea58b4ff1babc6b88
2014-08-22 15:12:39,896 INFO [RS_CLOSE_REGION-r3c7s4:60020-0]
regionserver.DefaultStoreFlusher: Flushed, sequenceid=827557339,
memsize=451.1 K, hasBloomFilter=true, into tmp file
hdfs://visualmeta/hbase/data/default/classifier/dc121bab8ed97c25f4f135c4f4f6fdcc/.tmp/377065ea713742e6b0dbc818499a787a
2014-08-22 15:12:39,899 INFO [RS_CLOSE_REGION-r3c7s4:60020-2]
regionserver.HStore: Added
hdfs://visualmeta/hbase/data/default/classifier/3129db940c9f94729b986b742f0161c2/cf/1fb37a3ae1e346bea58b4ff1babc6b88,
entries=5, sequenceid=830369747, filesize=112.0 K
2014-08-22 15:12:39,899 INFO [RS_CLOSE_REGION-r3c7s4:60020-2]
regionserver.HRegion: Finished memstore flush of ~111.6 K/114240,
currentsize=0/0 for region
classifier,\xE9\x02-\xF0\xA2\xDF\x17*,1408110936897.3129db940c9f94729b986b742f0161c2.
in 196ms, sequenceid=830369747, compaction requested=true
2014-08-22 15:12:39,923 INFO [RS_CLOSE_REGION-r3c7s4:60020-0]
regionserver.HStore: Added
hdfs://visualmeta/hbase/data/default/classifier/dc121bab8ed97c25f4f135c4f4f6fdcc/cf/377065ea713742e6b0dbc818499a787a,
entries=8, sequenceid=827557339, filesize=451.4 K
2014-08-22 15:12:39,923 INFO [RS_CLOSE_REGION-r3c7s4:60020-0]
regionserver.HRegion: Finished memstore flush of ~451.1 K/461928,
currentsize=0/0 for region
classifier,\xB3\x87H,1408395375533.dc121bab8ed97c25f4f135c4f4f6fdcc. in
213ms, sequenceid=827557339, compaction requested=true
2014-08-22 15:12:39,947 INFO
[StoreCloserThread-classifier,\xE9\x02-\xF0\xA2\xDF\x17*,1408110936897.3129db940c9f94729b986b742f0161c2.-1]
regionserver.HStore: Closed cf
2014-08-22 15:12:39,947 INFO [RS_CLOSE_REGION-r3c7s4:60020-2]
regionserver.HRegion: Closed
classifier,\xE9\x02-\xF0\xA2\xDF\x17*,1408110936897.3129db940c9f94729b986b742f0161c2.
2014-08-22 15:12:39,948 INFO [RS_CLOSE_REGION-r3c7s4:60020-2]
regionserver.HRegionServer: Adding moved region record:
3129db940c9f94729b986b742f0161c2 to
r3c5s3.colo.vm,60020,1408712894162:60020 as of 830369735
2014-08-22 15:12:39,965 INFO
[StoreCloserThread-classifier,\xB3\x87H,1408395375533.dc121bab8ed97c25f4f135c4f4f6fdcc.-1]
regionserver.HStore: Closed cf
2014-08-22 15:12:39,965 INFO [RS_CLOSE_REGION-r3c7s4:60020-0]
regionserver.HRegion: Closed
classifier,\xB3\x87H,1408395375533.dc121bab8ed97c25f4f135c4f4f6fdcc.
2014-08-22 15:12:39,966 INFO [RS_CLOSE_REGION-r3c7s4:60020-0]
regionserver.HRegionServer: Adding moved region record:
dc121bab8ed97c25f4f135c4f4f6fdcc to
r3c5s3.colo.vm,60020,1408712894162:60020 as of 827557330
2014-08-22 15:12:43,210 INFO
[PriorityRpcServer.handler=5,queue=0,port=60020]
regionserver.HRegionServer: Close de56dc1dc39aa245d466bb86158f0709, via
zk=yes, znode version=0, on r3c5s3.colo.vm,60020,1408712894162
2014-08-22 15:12:43,235 INFO
[PriorityRpcServer.handler=4,queue=0,port=60020]
regionserver.HRegionServer: Close d26e27c320268993613c8c0c4ef6eb57, via
zk=yes, znode version=0, on r3c5s3.colo.vm,60020,1408712894162
2014-08-22 15:12:43,245 INFO
[StoreCloserThread-image,fr\x00\x00\x00\x00\x98\xEAI\x8A,1401591381273.de56dc1dc39aa245d466bb86158f0709.-1]
regionserver.HStore: Closed cf
2014-08-22 15:12:43,245 INFO [RS_CLOSE_REGION-r3c7s4:60020-1]
regionserver.HRegion: Closed
image,fr\x00\x00\x00\x00\x98\xEAI\x8A,1401591381273.de56dc1dc39aa245d466bb86158f0709.
2014-08-22 15:12:43,245 INFO [RS_CLOSE_REGION-r3c7s4:60020-1]
regionserver.HRegionServer: Adding moved region record:
de56dc1dc39aa245d466bb86158f0709 to
r3c5s3.colo.vm,60020,1408712894162:60020 as of 855087042
2014-08-22 15:12:43,258 INFO
[PriorityRpcServer.handler=6,queue=0,port=60020]
regionserver.HRegionServer: Close 688b0564f44b3dbf080a1ef421374261, via
zk=yes, znode version=0, on r3c5s3.colo.vm,60020,1408712894162
2014-08-22 15:12:43,261 INFO
[StoreCloserThread-image,se\x00\x00\x00\x00)\xF5\xC3F,1402967065165.d26e27c320268993613c8c0c4ef6eb57.-1]
regionserver.HStore: Closed cf
2014-08-22 15:12:43,262 INFO [RS_CLOSE_REGION-r3c7s4:60020-2]
regionserver.HRegion: Closed
image,se\x00\x00\x00\x00)\xF5\xC3F,1402967065165.d26e27c320268993613c8c0c4ef6eb57.
2014-08-22 15:12:43,262 INFO [RS_CLOSE_REGION-r3c7s4:60020-2]
regionserver.HRegionServer: Adding moved region record:
d26e27c320268993613c8c0c4ef6eb57 to
r3c5s3.colo.vm,60020,1408712894162:60020 as of 855084513
2014-08-22 15:12:43,270 INFO
[PriorityRpcServer.handler=7,queue=0,port=60020]
regionserver.HRegionServer: Close f266cc9eb0de018f335dab7592b08db4, via
zk=yes, znode version=0, on r3c5s3.colo.vm,60020,1408712894162
2014-08-22 15:12:43,278 INFO [StoreCloserThread-image,es\x00\x00\x00\x00
\xA2tG,1400138220753.688b0564f44b3dbf080a1ef421374261.-1]
regionserver.HStore: Closed cf
2014-08-22 15:12:43,279 INFO [RS_CLOSE_REGION-r3c7s4:60020-0]
regionserver.HRegion: Closed image,es\x00\x00\x00\x00
\xA2tG,1400138220753.688b0564f44b3dbf080a1ef421374261.
2014-08-22 15:12:43,279 INFO [RS_CLOSE_REGION-r3c7s4:60020-0]
regionserver.HRegionServer: Adding moved region record:
688b0564f44b3dbf080a1ef421374261 to
r3c5s3.colo.vm,60020,1408712894162:60020 as of 756505811
2014-08-22 15:12:43,294 INFO
[PriorityRpcServer.handler=3,queue=0,port=60020]
regionserver.HRegionServer: Close 50f9d7e3b62898fcea5b306b59474db6, via
zk=yes, znode version=0, on r3c5s3.colo.vm,60020,1408712894162
2014-08-22 15:12:43,300 INFO
[StoreCloserThread-image,fi\x00\x00\x00\x00\x0D\x96\xF3\xD1,1408470461149.f266cc9eb0de018f335dab7592b08db4.-1]
regionserver.HStore: Closed cf
2014-08-22 15:12:43,300 INFO [RS_CLOSE_REGION-r3c7s4:60020-1]
regionserver.HRegion: Closed
image,fi\x00\x00\x00\x00\x0D\x96\xF3\xD1,1408470461149.f266cc9eb0de018f335dab7592b08db4.
2014-08-22 15:12:43,300 INFO [RS_CLOSE_REGION-r3c7s4:60020-1]
regionserver.HRegionServer: Adding moved region record:
f266cc9eb0de018f335dab7592b08db4 to
r3c5s3.colo.vm,60020,1408712894162:60020 as of 846307853
2014-08-22 15:12:43,329 INFO
[PriorityRpcServer.handler=9,queue=0,port=60020]
regionserver.HRegionServer: Close eb0c6870c459ffe5d21ff223b9169b56, via
zk=yes, znode version=0, on r3c5s3.colo.vm,60020,1408712894162
2014-08-22 15:12:43,337 INFO
[StoreCloserThread-image,it\x00\x00\x00\x00<\xC9H\xE7,1406031094658.50f9d7e3b62898fcea5b306b59474db6.-1]
regionserver.HStore: Closed cf
2014-08-22 15:12:43,337 INFO [RS_CLOSE_REGION-r3c7s4:60020-2]
regionserver.HRegion: Closed
image,it\x00\x00\x00\x00<\xC9H\xE7,1406031094658.50f9d7e3b62898fcea5b306b59474db6.
2014-08-22 15:12:43,337 INFO [RS_CLOSE_REGION-r3c7s4:60020-2]
regionserver.HRegionServer: Adding moved region record:
50f9d7e3b62898fcea5b306b59474db6 to
r3c5s3.colo.vm,60020,1408712894162:60020 as of 848082246
2014-08-22 15:12:43,345 INFO
[StoreCloserThread-image,in\x00\x00\x00\x00\x01F\xE73,1400549232558.eb0c6870c459ffe5d21ff223b9169b56.-1]
regionserver.HStore: Closed cf
2014-08-22 15:12:43,346 INFO [RS_CLOSE_REGION-r3c7s4:60020-0]
regionserver.HRegion: Closed
image,in\x00\x00\x00\x00\x01F\xE73,1400549232558.eb0c6870c459ffe5d21ff223b9169b56.
2014-08-22 15:12:43,346 INFO [RS_CLOSE_REGION-r3c7s4:60020-0]
regionserver.HRegionServer: Adding moved region record:
eb0c6870c459ffe5d21ff223b9169b56 to
r3c5s3.colo.vm,60020,1408712894162:60020 as of 793193777
2014-08-22 15:12:43,348 INFO
[PriorityRpcServer.handler=1,queue=0,port=60020]
regionserver.HRegionServer: Close d65a237e26153d29201b57dd4870c590, via
zk=yes, znode version=0, on r3c5s3.colo.vm,60020,1408712894162
2014-08-22 15:12:43,360 INFO
[PriorityRpcServer.handler=8,queue=0,port=60020]
regionserver.HRegionServer: Close 492dc247123bfea48058d447e5a3d7ab, via
zk=yes, znode version=0, on r3c5s3.colo.vm,60020,1408712894162
2014-08-22 15:12:43,372 INFO
[PriorityRpcServer.handler=0,queue=0,port=60020]
regionserver.HRegionServer: Close 23ba7e5b9354d05a71c526d2fa0ecd47, via
zk=yes, znode version=0, on r3c5s3.colo.vm,60020,1408712894162
2014-08-22 15:12:43,376 INFO
[StoreCloserThread-image,in\x00\x00\x00\x00\x01\x04WI,1399325596246.492dc247123bfea48058d447e5a3d7ab.-1]
regionserver.HStore: Closed cf
2014-08-22 15:12:43,377 INFO [RS_CLOSE_REGION-r3c7s4:60020-2]
regionserver.HRegion: Closed
image,in\x00\x00\x00\x00\x01\x04WI,1399325596246.492dc247123bfea48058d447e5a3d7ab.
2014-08-22 15:12:43,377 INFO [RS_CLOSE_REGION-r3c7s4:60020-2]
regionserver.HRegionServer: Adding moved region record:
492dc247123bfea48058d447e5a3d7ab to
r3c5s3.colo.vm,60020,1408712894162:60020 as of 691695328
2014-08-22 15:12:43,378 INFO
[PriorityRpcServer.handler=2,queue=0,port=60020]
regionserver.HRegionServer: Close 3b16a20aafe6c66168fdfb4b9e71c7da, via
zk=yes, znode version=0, on r3c5s3.colo.vm,60020,1408712894162
2014-08-22 15:12:43,381 INFO
[StoreCloserThread-image,ru\x00\x00\x00\x00\x00\xF9\xB9\x9C,1400121006136.23ba7e5b9354d05a71c526d2fa0ecd47.-1]
regionserver.HStore: Closed cf
2014-08-22 15:12:43,381 INFO [RS_CLOSE_REGION-r3c7s4:60020-0]
regionserver.HRegion: Closed
image,ru\x00\x00\x00\x00\x00\xF9\xB9\x9C,1400121006136.23ba7e5b9354d05a71c526d2fa0ecd47.
2014-08-22 15:12:43,381 INFO [RS_CLOSE_REGION-r3c7s4:60020-0]
regionserver.HRegionServer: Adding moved region record:
23ba7e5b9354d05a71c526d2fa0ecd47 to
r3c5s3.colo.vm,60020,1408712894162:60020 as of 755730346
2014-08-22 15:12:43,388 INFO
[StoreCloserThread-image,fr\x00\x00\x00\x00\x9B+\x9A\xA7,1402266525879.d65a237e26153d29201b57dd4870c590.-1]
regionserver.HStore: Closed cf
2014-08-22 15:12:43,388 INFO [RS_CLOSE_REGION-r3c7s4:60020-1]
regionserver.HRegion: Closed
image,fr\x00\x00\x00\x00\x9B+\x9A\xA7,1402266525879.d65a237e26153d29201b57dd4870c590.
2014-08-22 15:12:43,388 INFO [RS_CLOSE_REGION-r3c7s4:60020-1]
regionserver.HRegionServer: Adding moved region record:
d65a237e26153d29201b57dd4870c590 to
r3c5s3.colo.vm,60020,1408712894162:60020 as of 824275348
2014-08-22 15:12:43,421 INFO
[StoreCloserThread-image,de\x00\x00\x00\x02u\xF7\x92\x10,1405380638297.3b16a20aafe6c66168fdfb4b9e71c7da.-1]
regionserver.HStore: Closed cf
2014-08-22 15:12:43,422 INFO [RS_CLOSE_REGION-r3c7s4:60020-2]
regionserver.HRegion: Closed
image,de\x00\x00\x00\x02u\xF7\x92\x10,1405380638297.3b16a20aafe6c66168fdfb4b9e71c7da.
2014-08-22 15:12:43,422 INFO [RS_CLOSE_REGION-r3c7s4:60020-2]
regionserver.HRegionServer: Adding moved region record:
3b16a20aafe6c66168fdfb4b9e71c7da to
r3c5s3.colo.vm,60020,1408712894162:60020 as of 821191177
RS Queue:
===========================================================
Compaction/Split Queue summary: compaction_queue=(0:0), split_queue=0,
merge_queue=0
Compaction/Split Queue dump:
LargeCompation Queue:
SmallCompation Queue:
Split Queue:
Region Merge Queue:
Flush Queue summary: flush_queue=0
Flush Queue Queue dump:
Flush Queue:
On Fri, Aug 22, 2014 at 7:38 PM, Stack <st...@duboce.net> wrote:
> Do you have a few thread dumps from a 'deaf' instance?
> St.Ack
>
>
>
> On Fri, Aug 22, 2014 at 10:28 AM, Johannes Schaback <
> johannes.schaback@visual-meta.com> wrote:
>
> > Dear HBase-Pros,
> >
> > we face a serious issue with our HBase production cluster for two days
> now.
> > Every couple minutes, a random RegionServer gets stuck and does not
> process
> > any requests. In addition this causes the other RegionServers to
> > freeze within a minute which brings down the entire cluster. Stopping the
> > affected RegionServer unblocks the cluster and everything comes back to
> > normal.
> >
> > We run 27 RegionServers, each having 31 GB JVM memory. The HBase Version
> is
> > 0.98.5 on Hadoop 2.4.1. We basically have two tables, the first having
> > about 4,500 Regions and holding 8 TB with 1000 requests per second, the
> > second table is around 200 Regions with about 50,000 to 120,000 requests
> > per sec over all Regions, 800 GB worth of data and with IN_MEMORY
> enabled.
> >
> > While investigating the problem, I found out, that every healthy
> > RegionServer has the following thread:
> >
> > Thread 12 (RpcServer.listener,port=60020):
> > State: RUNNABLE
> > Blocked count: 35
> > Waited count: 0
> > Stack:
> > sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
> >
> org.apache.hadoop.hbase.ipc.RpcServer$Listener.run(RpcServer.java:684)
> >
> >
> > When suddenly becoming a blocked RegionServer, this particular thread
> then
> > looks like
> >
> > Thread 12 (RpcServer.listener,port=60020):
> > State: BLOCKED
> > Blocked count: 2889
> > Waited count: 0
> > Blocked on
> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader@38cba1a1
> > Blocked by 14 (RpcServer.reader=1,port=60020)
> > Stack:
> >
> >
> >
> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.registerChannel(RpcServer.java:619)
> >
> >
> org.apache.hadoop.hbase.ipc.RpcServer$Listener.doAccept(RpcServer.java:774)
> >
> org.apache.hadoop.hbase.ipc.RpcServer$Listener.run(RpcServer.java:692)
> >
> >
> > Also, JMX shows for an unhealthy RegionServer that
> >
> > - "queueSize" grows quickly and constantly to values greater than 60k,
> > and
> > - "numCallsInGeneralQueue" quickly reaches 300
> >
> > Both values are usually very small or 0 under normal circumstances, but
> in
> > case of a RS "getting stuck" they explode, which leads me to believe that
> > the IPC-queue does not get processed properly causing the RegionServer to
> > become "deaf".
> >
> > These two symptoms appear to bring down the entire cluster. When killign
> > that RS, everyhing goes back to normal.
> >
> > I could not find any correlation between this phenomenon and compactions,
> > load or other factors. hbck says it is all fine as well.
> >
> > The servers are all 3.2.0-4-amd64 Debian, 12 cores, 96 GB RAM. Besides
> the
> > RS and a DataNode, there isn't too much running on the boxes so the load
> > (top) is usually around 5 to 10 and bandwidth does not exceed 10 MB on
> > average.
> >
> > We currently survive by polling /jmx of all RegionServers constantly and
> > restarting those off that show the symptioms :(
> >
> > Do you have any idea what could be causing this?
> >
> > Thank you very much in advance!
> >
> > Johannes
> >
>
--
LadenZeile.de <http://www.ladenzeile.de/>
powered by Visual Meta GmbH - www.visual-meta.com
Tel.: +49 30 / 609 84 88 20
Fax: +49 30 / 609 84 88 21
E-Mail: johannes.schaback@visual-meta.com
Visual Meta GmbH, Schützenstraße 25, 10117 Berlin
Geschäftsführer: Robert M. Maier, Johannes Schaback
Handelsregister HRB 115795 B, Amtsgericht Charlottenburg
USt-IdNr.: DE263760203
Re: single RegionServer stuck, causing cluster to hang
Posted by Stack <st...@duboce.net>.
Do you have a few thread dumps from a 'deaf' instance?
St.Ack
On Fri, Aug 22, 2014 at 10:28 AM, Johannes Schaback <
johannes.schaback@visual-meta.com> wrote:
> Dear HBase-Pros,
>
> we face a serious issue with our HBase production cluster for two days now.
> Every couple minutes, a random RegionServer gets stuck and does not process
> any requests. In addition this causes the other RegionServers to
> freeze within a minute which brings down the entire cluster. Stopping the
> affected RegionServer unblocks the cluster and everything comes back to
> normal.
>
> We run 27 RegionServers, each having 31 GB JVM memory. The HBase Version is
> 0.98.5 on Hadoop 2.4.1. We basically have two tables, the first having
> about 4,500 Regions and holding 8 TB with 1000 requests per second, the
> second table is around 200 Regions with about 50,000 to 120,000 requests
> per sec over all Regions, 800 GB worth of data and with IN_MEMORY enabled.
>
> While investigating the problem, I found out, that every healthy
> RegionServer has the following thread:
>
> Thread 12 (RpcServer.listener,port=60020):
> State: RUNNABLE
> Blocked count: 35
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
> org.apache.hadoop.hbase.ipc.RpcServer$Listener.run(RpcServer.java:684)
>
>
> When suddenly becoming a blocked RegionServer, this particular thread then
> looks like
>
> Thread 12 (RpcServer.listener,port=60020):
> State: BLOCKED
> Blocked count: 2889
> Waited count: 0
> Blocked on org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader@38cba1a1
> Blocked by 14 (RpcServer.reader=1,port=60020)
> Stack:
>
>
> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.registerChannel(RpcServer.java:619)
>
> org.apache.hadoop.hbase.ipc.RpcServer$Listener.doAccept(RpcServer.java:774)
> org.apache.hadoop.hbase.ipc.RpcServer$Listener.run(RpcServer.java:692)
>
>
> Also, JMX shows for an unhealthy RegionServer that
>
> - "queueSize" grows quickly and constantly to values greater than 60k,
> and
> - "numCallsInGeneralQueue" quickly reaches 300
>
> Both values are usually very small or 0 under normal circumstances, but in
> case of a RS "getting stuck" they explode, which leads me to believe that
> the IPC-queue does not get processed properly causing the RegionServer to
> become "deaf".
>
> These two symptoms appear to bring down the entire cluster. When killign
> that RS, everyhing goes back to normal.
>
> I could not find any correlation between this phenomenon and compactions,
> load or other factors. hbck says it is all fine as well.
>
> The servers are all 3.2.0-4-amd64 Debian, 12 cores, 96 GB RAM. Besides the
> RS and a DataNode, there isn't too much running on the boxes so the load
> (top) is usually around 5 to 10 and bandwidth does not exceed 10 MB on
> average.
>
> We currently survive by polling /jmx of all RegionServers constantly and
> restarting those off that show the symptioms :(
>
> Do you have any idea what could be causing this?
>
> Thank you very much in advance!
>
> Johannes
>
Re: single RegionServer stuck, causing cluster to hang
Posted by Johannes Schaback <jo...@visual-meta.com>.
Hi Qiang,
no, we dont use coprocessors.
Thanks, Johannes
On Sun, Aug 24, 2014 at 6:04 AM, Qiang Tian <ti...@gmail.com> wrote:
> Hi Johannes,
> Do you use endpoint / coprocessor stuff?
> thanks.
>
>
> On Sun, Aug 24, 2014 at 11:51 AM, Qiang Tian <ti...@gmail.com> wrote:
>
> > Hi Stack,
> > I think you are right. the multiple queue change was introduced by
> > HBASE-11355(0.98.4). if there is only 1 queue, the stuck will not happen.
> > some handlers are gone but still some left to service the request(all
> > handlers gone looks a rare case)...
> > so the problem might have been there for some time, the change just
> > exposes it.
> >
> >
> >
> > On Sun, Aug 24, 2014 at 7:51 AM, Stack <st...@duboce.net> wrote:
> >
> >> On Sat, Aug 23, 2014 at 4:06 PM, Stack <st...@duboce.net> wrote:
> >>
> >> > ...
> >> > If you were looking for something to try, set
> >> hbase.ipc.server.callqueue.handler.factor
> >> > to 0. Multiple queues is what is new here. It should not make a
> >> difference
> >> > but...
> >> >
> >> >
> >> Hmm. Ignore above I'd say. I can't see how it would trigger this issue.
> >> Let me make a patch to give us more info and will have the handler keep
> >> going....
> >>
> >> St.Ack
> >>
> >
> >
>
--
LadenZeile.de <http://www.ladenzeile.de/>
powered by Visual Meta GmbH - www.visual-meta.com
Tel.: +49 30 / 609 84 88 20
Fax: +49 30 / 609 84 88 21
E-Mail: johannes.schaback@visual-meta.com
Visual Meta GmbH, Schützenstraße 25, 10117 Berlin
Geschäftsführer: Robert M. Maier, Johannes Schaback
Handelsregister HRB 115795 B, Amtsgericht Charlottenburg
USt-IdNr.: DE263760203
Re: single RegionServer stuck, causing cluster to hang
Posted by Qiang Tian <ti...@gmail.com>.
Hi Johannes,
Do you use endpoint / coprocessor stuff?
thanks.
On Sun, Aug 24, 2014 at 11:51 AM, Qiang Tian <ti...@gmail.com> wrote:
> Hi Stack,
> I think you are right. the multiple queue change was introduced by
> HBASE-11355(0.98.4). if there is only 1 queue, the stuck will not happen.
> some handlers are gone but still some left to service the request(all
> handlers gone looks a rare case)...
> so the problem might have been there for some time, the change just
> exposes it.
>
>
>
> On Sun, Aug 24, 2014 at 7:51 AM, Stack <st...@duboce.net> wrote:
>
>> On Sat, Aug 23, 2014 at 4:06 PM, Stack <st...@duboce.net> wrote:
>>
>> > ...
>> > If you were looking for something to try, set
>> hbase.ipc.server.callqueue.handler.factor
>> > to 0. Multiple queues is what is new here. It should not make a
>> difference
>> > but...
>> >
>> >
>> Hmm. Ignore above I'd say. I can't see how it would trigger this issue.
>> Let me make a patch to give us more info and will have the handler keep
>> going....
>>
>> St.Ack
>>
>
>
Re: single RegionServer stuck, causing cluster to hang
Posted by Qiang Tian <ti...@gmail.com>.
Hi Stack,
I think you are right. the multiple queue change was introduced by
HBASE-11355(0.98.4). if there is only 1 queue, the stuck will not happen.
some handlers are gone but still some left to service the request(all
handlers gone looks a rare case)...
so the problem might have been there for some time, the change just exposes
it.
On Sun, Aug 24, 2014 at 7:51 AM, Stack <st...@duboce.net> wrote:
> On Sat, Aug 23, 2014 at 4:06 PM, Stack <st...@duboce.net> wrote:
>
> > ...
> > If you were looking for something to try, set
> hbase.ipc.server.callqueue.handler.factor
> > to 0. Multiple queues is what is new here. It should not make a
> difference
> > but...
> >
> >
> Hmm. Ignore above I'd say. I can't see how it would trigger this issue.
> Let me make a patch to give us more info and will have the handler keep
> going....
>
> St.Ack
>
Re: single RegionServer stuck, causing cluster to hang
Posted by Stack <st...@duboce.net>.
On Sat, Aug 23, 2014 at 4:06 PM, Stack <st...@duboce.net> wrote:
> ...
> If you were looking for something to try, set hbase.ipc.server.callqueue.handler.factor
> to 0. Multiple queues is what is new here. It should not make a difference
> but...
>
>
Hmm. Ignore above I'd say. I can't see how it would trigger this issue.
Let me make a patch to give us more info and will have the handler keep
going....
St.Ack
Re: single RegionServer stuck, causing cluster to hang
Posted by Johannes Schaback <jo...@visual-meta.com>.
Great. Thank you. Checking out now. I hope I get everything assembled. I
keep you posted.
On Sun, Aug 24, 2014 at 9:57 AM, Stack <st...@duboce.net> wrote:
> I put up patches in the issue Johannes. Hopefully the reproduced
> stackoverflow is same as yours. See HBASE-11813.
> St.Ack
>
>
> On Sat, Aug 23, 2014 at 9:25 PM, Johannes Schaback <
> johannes.schaback@visual-meta.com> wrote:
>
> > We us all plain gets and puts (sometimes batched).
> >
> > We have hbase.client.keyvalue.maxsize increased to 536870912 bytes on the
> > client. That is the only thing I can see.
> >
> > I am about to send you a zip file with the respective classes to your
> email
> > address directly. I probably better dont post the code publicly.
> >
> > We will also attempt to set hbase.ipc.server.callqueue.handler.factor to
> 0
> > now. I keep you posted.
> >
> > Johannes
> >
> >
> >
> >
> > On Sun, Aug 24, 2014 at 1:06 AM, Stack <st...@duboce.net> wrote:
> >
> > > I am having trouble reproducing the stack overflow. Some particular
> > > response is triggering it (the code here has been around a while). Any
> > > particulars on how your client is accessing hbase? Anything unusual?
> > >
> > > If you were looking for something to try, set
> > > hbase.ipc.server.callqueue.handler.factor
> > > to 0. Multiple queues is what is new here. It should not make a
> > difference
> > > but...
> > >
> > > St.Ack
> > >
> > >
> > >
> > >
> > >
> > > On Sat, Aug 23, 2014 at 1:23 PM, Johannes Schaback <
> > > johannes.schaback@visual-meta.com> wrote:
> > >
> > > > Thank you.
> > > >
> > > > From the proposed resolution I imagine that the RS would then die in
> > case
> > > > of a handler error. So the question remains what error originally
> > occured
> > > > in the handler in the first place. The log of the entire lifecycle of
> > the
> > > > RS (http://schabby.de/wp-content/uploads/2014/08/filtered.txt) does
> > not
> > > > reveal much to me unfortunately. Do you find anything in there that
> > hints
> > > > to something that may cause the handler to end up in the
> > soon-to-be-fixed
> > > > recursion?
> > > >
> > > > @Ted, the line "at
> > > > org.apache.hadoop.hbase.CellUtil$1.advance(CellUtil.java:210)" is
> all I
> > > can
> > > > see unfortunately :(
> > > >
> > > >
> > > >
> > > > On Sat, Aug 23, 2014 at 9:43 PM, Andrew Purtell <apurtell@apache.org
> >
> > > > wrote:
> > > >
> > > > > On Sat, Aug 23, 2014 at 12:11 PM, Johannes Schaback <
> > > > > johannes.schaback@visual-meta.com> wrote:
> > > > >
> > > > > > Exception in thread
> "defaultRpcServer.handler=5,queue=2,port=60020"
> > > > > > java.lang.StackOverflowError
> > > > > > at
> > > > org.apache.hadoop.hbase.CellUtil$1.advance(CellUtil.java:210)
> > > > > > at
> > > > org.apache.hadoop.hbase.CellUtil$1.advance(CellUtil.java:210)
> > > > > > at
> > > > org.apache.hadoop.hbase.CellUtil$1.advance(CellUtil.java:210)
> > > > > > at
> > > > org.apache.hadoop.hbase.CellUtil$1.advance(CellUtil.java:210)
> > > > > > (and so on...)
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > That is the anonymous CellScanner instance we create from
> > > > > CellUtil#createCellScanner. See
> > > > > https://issues.apache.org/jira/browse/HBASE-11813
> > > > >
> > > > >
> > > > > > Filtering the .out file for "Exception" shows that several
> > handlers
> > > > > > crashed
> > > > > >
> > > > > > like that:
> > > > > >
> > > > > > Exception in thread
> "defaultRpcServer.handler=5,queue=2,port=60020"
> > > > > > java.lang.StackOverflowError
> > > > > > Exception in thread
> > "defaultRpcServer.handler=18,queue=0,port=60020"
> > > > > > java.lang.StackOverflowError
> > > > > > Exception in thread
> > "defaultRpcServer.handler=23,queue=2,port=60020"
> > > > > > java.lang.StackOverflowError
> > > > > > Exception in thread
> > "defaultRpcServer.handler=24,queue=0,port=60020"
> > > > > > java.lang.StackOverflowError
> > > > > > Exception in thread
> "defaultRpcServer.handler=2,queue=2,port=60020"
> > > > > > java.lang.StackOverflowError
> > > > > > Exception in thread
> > "defaultRpcServer.handler=11,queue=2,port=60020"
> > > > > > java.lang.StackOverflowError
> > > > > > Exception in thread
> > "defaultRpcServer.handler=25,queue=1,port=60020"
> > > > > > java.lang.StackOverflowError
> > > > > > Exception in thread
> > "defaultRpcServer.handler=20,queue=2,port=60020"
> > > > > > java.lang.StackOverflowError
> > > > > > Exception in thread
> > "defaultRpcServer.handler=19,queue=1,port=60020"
> > > > > > java.lang.StackOverflowError
> > > > > > Exception in thread
> > "defaultRpcServer.handler=15,queue=0,port=60020"
> > > > > > java.lang.StackOverflowError
> > > > > > Exception in thread
> "defaultRpcServer.handler=1,queue=1,port=60020"
> > > > > > java.lang.StackOverflowError
> > > > > > Exception in thread
> "defaultRpcServer.handler=7,queue=1,port=60020"
> > > > > > java.lang.StackOverflowError
> > > > > > Exception in thread
> "defaultRpcServer.handler=4,queue=1,port=60020"
> > > > > > java.lang.StackOverflowError
> > > > > >
> > > > >
> > > > >
> > > > > We should fix this so the RegionServer aborts if it loses a handler
> > to
> > > an
> > > > > Error.
> > > > >
> > > > >
> > > > > --
> > > > > Best regards,
> > > > >
> > > > > - Andy
> > > > >
> > > > > Problems worthy of attack prove their worth by hitting back. - Piet
> > > Hein
> > > > > (via Tom White)
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > LadenZeile.de <http://www.ladenzeile.de/>
> > > > powered by Visual Meta GmbH - www.visual-meta.com
> > > >
> > > > Tel.: +49 30 / 609 84 88 20
> > > > Fax: +49 30 / 609 84 88 21
> > > > E-Mail: johannes.schaback@visual-meta.com
> > > >
> > > > Visual Meta GmbH, Schützenstraße 25, 10117 Berlin
> > > > Geschäftsführer: Robert M. Maier, Johannes Schaback
> > > > Handelsregister HRB 115795 B, Amtsgericht Charlottenburg
> > > > USt-IdNr.: DE263760203
> > > >
> > >
> >
> >
> >
> > --
> > LadenZeile.de <http://www.ladenzeile.de/>
> > powered by Visual Meta GmbH - www.visual-meta.com
> >
> > Tel.: +49 30 / 609 84 88 20
> > Fax: +49 30 / 609 84 88 21
> > E-Mail: johannes.schaback@visual-meta.com
> >
> > Visual Meta GmbH, Schützenstraße 25, 10117 Berlin
> > Geschäftsführer: Robert M. Maier, Johannes Schaback
> > Handelsregister HRB 115795 B, Amtsgericht Charlottenburg
> > USt-IdNr.: DE263760203
> >
>
--
LadenZeile.de <http://www.ladenzeile.de/>
powered by Visual Meta GmbH - www.visual-meta.com
Tel.: +49 30 / 609 84 88 20
Fax: +49 30 / 609 84 88 21
E-Mail: johannes.schaback@visual-meta.com
Visual Meta GmbH, Schützenstraße 25, 10117 Berlin
Geschäftsführer: Robert M. Maier, Johannes Schaback
Handelsregister HRB 115795 B, Amtsgericht Charlottenburg
USt-IdNr.: DE263760203
Re: single RegionServer stuck, causing cluster to hang
Posted by Stack <st...@duboce.net>.
I put up patches in the issue Johannes. Hopefully the reproduced
stackoverflow is same as yours. See HBASE-11813.
St.Ack
On Sat, Aug 23, 2014 at 9:25 PM, Johannes Schaback <
johannes.schaback@visual-meta.com> wrote:
> We us all plain gets and puts (sometimes batched).
>
> We have hbase.client.keyvalue.maxsize increased to 536870912 bytes on the
> client. That is the only thing I can see.
>
> I am about to send you a zip file with the respective classes to your email
> address directly. I probably better dont post the code publicly.
>
> We will also attempt to set hbase.ipc.server.callqueue.handler.factor to 0
> now. I keep you posted.
>
> Johannes
>
>
>
>
> On Sun, Aug 24, 2014 at 1:06 AM, Stack <st...@duboce.net> wrote:
>
> > I am having trouble reproducing the stack overflow. Some particular
> > response is triggering it (the code here has been around a while). Any
> > particulars on how your client is accessing hbase? Anything unusual?
> >
> > If you were looking for something to try, set
> > hbase.ipc.server.callqueue.handler.factor
> > to 0. Multiple queues is what is new here. It should not make a
> difference
> > but...
> >
> > St.Ack
> >
> >
> >
> >
> >
> > On Sat, Aug 23, 2014 at 1:23 PM, Johannes Schaback <
> > johannes.schaback@visual-meta.com> wrote:
> >
> > > Thank you.
> > >
> > > From the proposed resolution I imagine that the RS would then die in
> case
> > > of a handler error. So the question remains what error originally
> occured
> > > in the handler in the first place. The log of the entire lifecycle of
> the
> > > RS (http://schabby.de/wp-content/uploads/2014/08/filtered.txt) does
> not
> > > reveal much to me unfortunately. Do you find anything in there that
> hints
> > > to something that may cause the handler to end up in the
> soon-to-be-fixed
> > > recursion?
> > >
> > > @Ted, the line "at
> > > org.apache.hadoop.hbase.CellUtil$1.advance(CellUtil.java:210)" is all I
> > can
> > > see unfortunately :(
> > >
> > >
> > >
> > > On Sat, Aug 23, 2014 at 9:43 PM, Andrew Purtell <ap...@apache.org>
> > > wrote:
> > >
> > > > On Sat, Aug 23, 2014 at 12:11 PM, Johannes Schaback <
> > > > johannes.schaback@visual-meta.com> wrote:
> > > >
> > > > > Exception in thread "defaultRpcServer.handler=5,queue=2,port=60020"
> > > > > java.lang.StackOverflowError
> > > > > at
> > > org.apache.hadoop.hbase.CellUtil$1.advance(CellUtil.java:210)
> > > > > at
> > > org.apache.hadoop.hbase.CellUtil$1.advance(CellUtil.java:210)
> > > > > at
> > > org.apache.hadoop.hbase.CellUtil$1.advance(CellUtil.java:210)
> > > > > at
> > > org.apache.hadoop.hbase.CellUtil$1.advance(CellUtil.java:210)
> > > > > (and so on...)
> > > > >
> > > > >
> > > >
> > > >
> > > > That is the anonymous CellScanner instance we create from
> > > > CellUtil#createCellScanner. See
> > > > https://issues.apache.org/jira/browse/HBASE-11813
> > > >
> > > >
> > > > > Filtering the .out file for "Exception" shows that several
> handlers
> > > > > crashed
> > > > >
> > > > > like that:
> > > > >
> > > > > Exception in thread "defaultRpcServer.handler=5,queue=2,port=60020"
> > > > > java.lang.StackOverflowError
> > > > > Exception in thread
> "defaultRpcServer.handler=18,queue=0,port=60020"
> > > > > java.lang.StackOverflowError
> > > > > Exception in thread
> "defaultRpcServer.handler=23,queue=2,port=60020"
> > > > > java.lang.StackOverflowError
> > > > > Exception in thread
> "defaultRpcServer.handler=24,queue=0,port=60020"
> > > > > java.lang.StackOverflowError
> > > > > Exception in thread "defaultRpcServer.handler=2,queue=2,port=60020"
> > > > > java.lang.StackOverflowError
> > > > > Exception in thread
> "defaultRpcServer.handler=11,queue=2,port=60020"
> > > > > java.lang.StackOverflowError
> > > > > Exception in thread
> "defaultRpcServer.handler=25,queue=1,port=60020"
> > > > > java.lang.StackOverflowError
> > > > > Exception in thread
> "defaultRpcServer.handler=20,queue=2,port=60020"
> > > > > java.lang.StackOverflowError
> > > > > Exception in thread
> "defaultRpcServer.handler=19,queue=1,port=60020"
> > > > > java.lang.StackOverflowError
> > > > > Exception in thread
> "defaultRpcServer.handler=15,queue=0,port=60020"
> > > > > java.lang.StackOverflowError
> > > > > Exception in thread "defaultRpcServer.handler=1,queue=1,port=60020"
> > > > > java.lang.StackOverflowError
> > > > > Exception in thread "defaultRpcServer.handler=7,queue=1,port=60020"
> > > > > java.lang.StackOverflowError
> > > > > Exception in thread "defaultRpcServer.handler=4,queue=1,port=60020"
> > > > > java.lang.StackOverflowError
> > > > >
> > > >
> > > >
> > > > We should fix this so the RegionServer aborts if it loses a handler
> to
> > an
> > > > Error.
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > >
> > > > - Andy
> > > >
> > > > Problems worthy of attack prove their worth by hitting back. - Piet
> > Hein
> > > > (via Tom White)
> > > >
> > >
> > >
> > >
> > > --
> > > LadenZeile.de <http://www.ladenzeile.de/>
> > > powered by Visual Meta GmbH - www.visual-meta.com
> > >
> > > Tel.: +49 30 / 609 84 88 20
> > > Fax: +49 30 / 609 84 88 21
> > > E-Mail: johannes.schaback@visual-meta.com
> > >
> > > Visual Meta GmbH, Schützenstraße 25, 10117 Berlin
> > > Geschäftsführer: Robert M. Maier, Johannes Schaback
> > > Handelsregister HRB 115795 B, Amtsgericht Charlottenburg
> > > USt-IdNr.: DE263760203
> > >
> >
>
>
>
> --
> LadenZeile.de <http://www.ladenzeile.de/>
> powered by Visual Meta GmbH - www.visual-meta.com
>
> Tel.: +49 30 / 609 84 88 20
> Fax: +49 30 / 609 84 88 21
> E-Mail: johannes.schaback@visual-meta.com
>
> Visual Meta GmbH, Schützenstraße 25, 10117 Berlin
> Geschäftsführer: Robert M. Maier, Johannes Schaback
> Handelsregister HRB 115795 B, Amtsgericht Charlottenburg
> USt-IdNr.: DE263760203
>
Re: single RegionServer stuck, causing cluster to hang
Posted by Johannes Schaback <jo...@visual-meta.com>.
We us all plain gets and puts (sometimes batched).
We have hbase.client.keyvalue.maxsize increased to 536870912 bytes on the
client. That is the only thing I can see.
I am about to send you a zip file with the respective classes to your email
address directly. I probably better dont post the code publicly.
We will also attempt to set hbase.ipc.server.callqueue.handler.factor to 0
now. I keep you posted.
Johannes
On Sun, Aug 24, 2014 at 1:06 AM, Stack <st...@duboce.net> wrote:
> I am having trouble reproducing the stack overflow. Some particular
> response is triggering it (the code here has been around a while). Any
> particulars on how your client is accessing hbase? Anything unusual?
>
> If you were looking for something to try, set
> hbase.ipc.server.callqueue.handler.factor
> to 0. Multiple queues is what is new here. It should not make a difference
> but...
>
> St.Ack
>
>
>
>
>
> On Sat, Aug 23, 2014 at 1:23 PM, Johannes Schaback <
> johannes.schaback@visual-meta.com> wrote:
>
> > Thank you.
> >
> > From the proposed resolution I imagine that the RS would then die in case
> > of a handler error. So the question remains what error originally occured
> > in the handler in the first place. The log of the entire lifecycle of the
> > RS (http://schabby.de/wp-content/uploads/2014/08/filtered.txt) does not
> > reveal much to me unfortunately. Do you find anything in there that hints
> > to something that may cause the handler to end up in the soon-to-be-fixed
> > recursion?
> >
> > @Ted, the line "at
> > org.apache.hadoop.hbase.CellUtil$1.advance(CellUtil.java:210)" is all I
> can
> > see unfortunately :(
> >
> >
> >
> > On Sat, Aug 23, 2014 at 9:43 PM, Andrew Purtell <ap...@apache.org>
> > wrote:
> >
> > > On Sat, Aug 23, 2014 at 12:11 PM, Johannes Schaback <
> > > johannes.schaback@visual-meta.com> wrote:
> > >
> > > > Exception in thread "defaultRpcServer.handler=5,queue=2,port=60020"
> > > > java.lang.StackOverflowError
> > > > at
> > org.apache.hadoop.hbase.CellUtil$1.advance(CellUtil.java:210)
> > > > at
> > org.apache.hadoop.hbase.CellUtil$1.advance(CellUtil.java:210)
> > > > at
> > org.apache.hadoop.hbase.CellUtil$1.advance(CellUtil.java:210)
> > > > at
> > org.apache.hadoop.hbase.CellUtil$1.advance(CellUtil.java:210)
> > > > (and so on...)
> > > >
> > > >
> > >
> > >
> > > That is the anonymous CellScanner instance we create from
> > > CellUtil#createCellScanner. See
> > > https://issues.apache.org/jira/browse/HBASE-11813
> > >
> > >
> > > > Filtering the .out file for "Exception" shows that several handlers
> > > > crashed
> > > >
> > > > like that:
> > > >
> > > > Exception in thread "defaultRpcServer.handler=5,queue=2,port=60020"
> > > > java.lang.StackOverflowError
> > > > Exception in thread "defaultRpcServer.handler=18,queue=0,port=60020"
> > > > java.lang.StackOverflowError
> > > > Exception in thread "defaultRpcServer.handler=23,queue=2,port=60020"
> > > > java.lang.StackOverflowError
> > > > Exception in thread "defaultRpcServer.handler=24,queue=0,port=60020"
> > > > java.lang.StackOverflowError
> > > > Exception in thread "defaultRpcServer.handler=2,queue=2,port=60020"
> > > > java.lang.StackOverflowError
> > > > Exception in thread "defaultRpcServer.handler=11,queue=2,port=60020"
> > > > java.lang.StackOverflowError
> > > > Exception in thread "defaultRpcServer.handler=25,queue=1,port=60020"
> > > > java.lang.StackOverflowError
> > > > Exception in thread "defaultRpcServer.handler=20,queue=2,port=60020"
> > > > java.lang.StackOverflowError
> > > > Exception in thread "defaultRpcServer.handler=19,queue=1,port=60020"
> > > > java.lang.StackOverflowError
> > > > Exception in thread "defaultRpcServer.handler=15,queue=0,port=60020"
> > > > java.lang.StackOverflowError
> > > > Exception in thread "defaultRpcServer.handler=1,queue=1,port=60020"
> > > > java.lang.StackOverflowError
> > > > Exception in thread "defaultRpcServer.handler=7,queue=1,port=60020"
> > > > java.lang.StackOverflowError
> > > > Exception in thread "defaultRpcServer.handler=4,queue=1,port=60020"
> > > > java.lang.StackOverflowError
> > > >
> > >
> > >
> > > We should fix this so the RegionServer aborts if it loses a handler to
> an
> > > Error.
> > >
> > >
> > > --
> > > Best regards,
> > >
> > > - Andy
> > >
> > > Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> > > (via Tom White)
> > >
> >
> >
> >
> > --
> > LadenZeile.de <http://www.ladenzeile.de/>
> > powered by Visual Meta GmbH - www.visual-meta.com
> >
> > Tel.: +49 30 / 609 84 88 20
> > Fax: +49 30 / 609 84 88 21
> > E-Mail: johannes.schaback@visual-meta.com
> >
> > Visual Meta GmbH, Schützenstraße 25, 10117 Berlin
> > Geschäftsführer: Robert M. Maier, Johannes Schaback
> > Handelsregister HRB 115795 B, Amtsgericht Charlottenburg
> > USt-IdNr.: DE263760203
> >
>
--
LadenZeile.de <http://www.ladenzeile.de/>
powered by Visual Meta GmbH - www.visual-meta.com
Tel.: +49 30 / 609 84 88 20
Fax: +49 30 / 609 84 88 21
E-Mail: johannes.schaback@visual-meta.com
Visual Meta GmbH, Schützenstraße 25, 10117 Berlin
Geschäftsführer: Robert M. Maier, Johannes Schaback
Handelsregister HRB 115795 B, Amtsgericht Charlottenburg
USt-IdNr.: DE263760203
Re: single RegionServer stuck, causing cluster to hang
Posted by Stack <st...@duboce.net>.
I am having trouble reproducing the stack overflow. Some particular
response is triggering it (the code here has been around a while). Any
particulars on how your client is accessing hbase? Anything unusual?
If you were looking for something to try, set
hbase.ipc.server.callqueue.handler.factor
to 0. Multiple queues is what is new here. It should not make a difference
but...
St.Ack
On Sat, Aug 23, 2014 at 1:23 PM, Johannes Schaback <
johannes.schaback@visual-meta.com> wrote:
> Thank you.
>
> From the proposed resolution I imagine that the RS would then die in case
> of a handler error. So the question remains what error originally occured
> in the handler in the first place. The log of the entire lifecycle of the
> RS (http://schabby.de/wp-content/uploads/2014/08/filtered.txt) does not
> reveal much to me unfortunately. Do you find anything in there that hints
> to something that may cause the handler to end up in the soon-to-be-fixed
> recursion?
>
> @Ted, the line "at
> org.apache.hadoop.hbase.CellUtil$1.advance(CellUtil.java:210)" is all I can
> see unfortunately :(
>
>
>
> On Sat, Aug 23, 2014 at 9:43 PM, Andrew Purtell <ap...@apache.org>
> wrote:
>
> > On Sat, Aug 23, 2014 at 12:11 PM, Johannes Schaback <
> > johannes.schaback@visual-meta.com> wrote:
> >
> > > Exception in thread "defaultRpcServer.handler=5,queue=2,port=60020"
> > > java.lang.StackOverflowError
> > > at
> org.apache.hadoop.hbase.CellUtil$1.advance(CellUtil.java:210)
> > > at
> org.apache.hadoop.hbase.CellUtil$1.advance(CellUtil.java:210)
> > > at
> org.apache.hadoop.hbase.CellUtil$1.advance(CellUtil.java:210)
> > > at
> org.apache.hadoop.hbase.CellUtil$1.advance(CellUtil.java:210)
> > > (and so on...)
> > >
> > >
> >
> >
> > That is the anonymous CellScanner instance we create from
> > CellUtil#createCellScanner. See
> > https://issues.apache.org/jira/browse/HBASE-11813
> >
> >
> > > Filtering the .out file for "Exception" shows that several handlers
> > > crashed
> > >
> > > like that:
> > >
> > > Exception in thread "defaultRpcServer.handler=5,queue=2,port=60020"
> > > java.lang.StackOverflowError
> > > Exception in thread "defaultRpcServer.handler=18,queue=0,port=60020"
> > > java.lang.StackOverflowError
> > > Exception in thread "defaultRpcServer.handler=23,queue=2,port=60020"
> > > java.lang.StackOverflowError
> > > Exception in thread "defaultRpcServer.handler=24,queue=0,port=60020"
> > > java.lang.StackOverflowError
> > > Exception in thread "defaultRpcServer.handler=2,queue=2,port=60020"
> > > java.lang.StackOverflowError
> > > Exception in thread "defaultRpcServer.handler=11,queue=2,port=60020"
> > > java.lang.StackOverflowError
> > > Exception in thread "defaultRpcServer.handler=25,queue=1,port=60020"
> > > java.lang.StackOverflowError
> > > Exception in thread "defaultRpcServer.handler=20,queue=2,port=60020"
> > > java.lang.StackOverflowError
> > > Exception in thread "defaultRpcServer.handler=19,queue=1,port=60020"
> > > java.lang.StackOverflowError
> > > Exception in thread "defaultRpcServer.handler=15,queue=0,port=60020"
> > > java.lang.StackOverflowError
> > > Exception in thread "defaultRpcServer.handler=1,queue=1,port=60020"
> > > java.lang.StackOverflowError
> > > Exception in thread "defaultRpcServer.handler=7,queue=1,port=60020"
> > > java.lang.StackOverflowError
> > > Exception in thread "defaultRpcServer.handler=4,queue=1,port=60020"
> > > java.lang.StackOverflowError
> > >
> >
> >
> > We should fix this so the RegionServer aborts if it loses a handler to an
> > Error.
> >
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>
>
>
> --
> LadenZeile.de <http://www.ladenzeile.de/>
> powered by Visual Meta GmbH - www.visual-meta.com
>
> Tel.: +49 30 / 609 84 88 20
> Fax: +49 30 / 609 84 88 21
> E-Mail: johannes.schaback@visual-meta.com
>
> Visual Meta GmbH, Schützenstraße 25, 10117 Berlin
> Geschäftsführer: Robert M. Maier, Johannes Schaback
> Handelsregister HRB 115795 B, Amtsgericht Charlottenburg
> USt-IdNr.: DE263760203
>
Re: single RegionServer stuck, causing cluster to hang
Posted by Johannes Schaback <jo...@visual-meta.com>.
Thank you.
>From the proposed resolution I imagine that the RS would then die in case
of a handler error. So the question remains what error originally occured
in the handler in the first place. The log of the entire lifecycle of the
RS (http://schabby.de/wp-content/uploads/2014/08/filtered.txt) does not
reveal much to me unfortunately. Do you find anything in there that hints
to something that may cause the handler to end up in the soon-to-be-fixed
recursion?
@Ted, the line "at
org.apache.hadoop.hbase.CellUtil$1.advance(CellUtil.java:210)" is all I can
see unfortunately :(
On Sat, Aug 23, 2014 at 9:43 PM, Andrew Purtell <ap...@apache.org> wrote:
> On Sat, Aug 23, 2014 at 12:11 PM, Johannes Schaback <
> johannes.schaback@visual-meta.com> wrote:
>
> > Exception in thread "defaultRpcServer.handler=5,queue=2,port=60020"
> > java.lang.StackOverflowError
> > at org.apache.hadoop.hbase.CellUtil$1.advance(CellUtil.java:210)
> > at org.apache.hadoop.hbase.CellUtil$1.advance(CellUtil.java:210)
> > at org.apache.hadoop.hbase.CellUtil$1.advance(CellUtil.java:210)
> > at org.apache.hadoop.hbase.CellUtil$1.advance(CellUtil.java:210)
> > (and so on...)
> >
> >
>
>
> That is the anonymous CellScanner instance we create from
> CellUtil#createCellScanner. See
> https://issues.apache.org/jira/browse/HBASE-11813
>
>
> > Filtering the .out file for "Exception" shows that several handlers
> > crashed
> >
> > like that:
> >
> > Exception in thread "defaultRpcServer.handler=5,queue=2,port=60020"
> > java.lang.StackOverflowError
> > Exception in thread "defaultRpcServer.handler=18,queue=0,port=60020"
> > java.lang.StackOverflowError
> > Exception in thread "defaultRpcServer.handler=23,queue=2,port=60020"
> > java.lang.StackOverflowError
> > Exception in thread "defaultRpcServer.handler=24,queue=0,port=60020"
> > java.lang.StackOverflowError
> > Exception in thread "defaultRpcServer.handler=2,queue=2,port=60020"
> > java.lang.StackOverflowError
> > Exception in thread "defaultRpcServer.handler=11,queue=2,port=60020"
> > java.lang.StackOverflowError
> > Exception in thread "defaultRpcServer.handler=25,queue=1,port=60020"
> > java.lang.StackOverflowError
> > Exception in thread "defaultRpcServer.handler=20,queue=2,port=60020"
> > java.lang.StackOverflowError
> > Exception in thread "defaultRpcServer.handler=19,queue=1,port=60020"
> > java.lang.StackOverflowError
> > Exception in thread "defaultRpcServer.handler=15,queue=0,port=60020"
> > java.lang.StackOverflowError
> > Exception in thread "defaultRpcServer.handler=1,queue=1,port=60020"
> > java.lang.StackOverflowError
> > Exception in thread "defaultRpcServer.handler=7,queue=1,port=60020"
> > java.lang.StackOverflowError
> > Exception in thread "defaultRpcServer.handler=4,queue=1,port=60020"
> > java.lang.StackOverflowError
> >
>
>
> We should fix this so the RegionServer aborts if it loses a handler to an
> Error.
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
--
LadenZeile.de <http://www.ladenzeile.de/>
powered by Visual Meta GmbH - www.visual-meta.com
Tel.: +49 30 / 609 84 88 20
Fax: +49 30 / 609 84 88 21
E-Mail: johannes.schaback@visual-meta.com
Visual Meta GmbH, Schützenstraße 25, 10117 Berlin
Geschäftsführer: Robert M. Maier, Johannes Schaback
Handelsregister HRB 115795 B, Amtsgericht Charlottenburg
USt-IdNr.: DE263760203
Re: single RegionServer stuck, causing cluster to hang
Posted by Andrew Purtell <ap...@apache.org>.
On Sat, Aug 23, 2014 at 12:11 PM, Johannes Schaback <
johannes.schaback@visual-meta.com> wrote:
> Exception in thread "defaultRpcServer.handler=5,queue=2,port=60020"
> java.lang.StackOverflowError
> at org.apache.hadoop.hbase.CellUtil$1.advance(CellUtil.java:210)
> at org.apache.hadoop.hbase.CellUtil$1.advance(CellUtil.java:210)
> at org.apache.hadoop.hbase.CellUtil$1.advance(CellUtil.java:210)
> at org.apache.hadoop.hbase.CellUtil$1.advance(CellUtil.java:210)
> (and so on...)
>
>
That is the anonymous CellScanner instance we create from
CellUtil#createCellScanner. See
https://issues.apache.org/jira/browse/HBASE-11813
> Filtering the .out file for "Exception" shows that several handlers
> crashed
>
> like that:
>
> Exception in thread "defaultRpcServer.handler=5,queue=2,port=60020"
> java.lang.StackOverflowError
> Exception in thread "defaultRpcServer.handler=18,queue=0,port=60020"
> java.lang.StackOverflowError
> Exception in thread "defaultRpcServer.handler=23,queue=2,port=60020"
> java.lang.StackOverflowError
> Exception in thread "defaultRpcServer.handler=24,queue=0,port=60020"
> java.lang.StackOverflowError
> Exception in thread "defaultRpcServer.handler=2,queue=2,port=60020"
> java.lang.StackOverflowError
> Exception in thread "defaultRpcServer.handler=11,queue=2,port=60020"
> java.lang.StackOverflowError
> Exception in thread "defaultRpcServer.handler=25,queue=1,port=60020"
> java.lang.StackOverflowError
> Exception in thread "defaultRpcServer.handler=20,queue=2,port=60020"
> java.lang.StackOverflowError
> Exception in thread "defaultRpcServer.handler=19,queue=1,port=60020"
> java.lang.StackOverflowError
> Exception in thread "defaultRpcServer.handler=15,queue=0,port=60020"
> java.lang.StackOverflowError
> Exception in thread "defaultRpcServer.handler=1,queue=1,port=60020"
> java.lang.StackOverflowError
> Exception in thread "defaultRpcServer.handler=7,queue=1,port=60020"
> java.lang.StackOverflowError
> Exception in thread "defaultRpcServer.handler=4,queue=1,port=60020"
> java.lang.StackOverflowError
>
We should fix this so the RegionServer aborts if it loses a handler to an
Error.
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: single RegionServer stuck, causing cluster to hang
Posted by Ted Yu <yu...@gmail.com>.
Can you show the complete stack trace for StackOverflowException (using pastebin) ?
Thanks
On Aug 23, 2014, at 12:11 PM, Johannes Schaback <jo...@visual-meta.com> wrote:
> Hi,
>
> we had to reduce load on the cluster yesterday night which reduced the
> frequency of the phenomenon. That is why I could not get a jstack dump yet
> because it did not occur since a couple hours. We will now get the load
> back up hoping to trigger it again.
>
> Yes, I cut out the properties from the /debug dump because they are all
> standard. We have hbase.ipc.server.callqueue.handler.factor to the standard
>
> <property>
> <name>hbase.ipc.server.callqueue.handler.factor</name>
> <value>0.1</value>
> <source>hbase-default.xml</source>
> </property>
>
> You find the complete config of the RS here: http://pastebin.com/iF1ibFb1
>
> The hint about the .out files was a great one (I never really looked at
> them actually). The .out files are flooded with StackOverflowExceptions:
>
> Exception in thread "defaultRpcServer.handler=5,queue=2,port=60020"
> java.lang.StackOverflowError
> at org.apache.hadoop.hbase.CellUtil$1.advance(CellUtil.java:210)
> at org.apache.hadoop.hbase.CellUtil$1.advance(CellUtil.java:210)
> at org.apache.hadoop.hbase.CellUtil$1.advance(CellUtil.java:210)
> at org.apache.hadoop.hbase.CellUtil$1.advance(CellUtil.java:210)
> (and so on...)
>
> Filtering the .out file for "Exception" shows that several handlers crashed
> like that:
>
> Exception in thread "defaultRpcServer.handler=5,queue=2,port=60020"
> java.lang.StackOverflowError
> Exception in thread "defaultRpcServer.handler=18,queue=0,port=60020"
> java.lang.StackOverflowError
> Exception in thread "defaultRpcServer.handler=23,queue=2,port=60020"
> java.lang.StackOverflowError
> Exception in thread "defaultRpcServer.handler=24,queue=0,port=60020"
> java.lang.StackOverflowError
> Exception in thread "defaultRpcServer.handler=2,queue=2,port=60020"
> java.lang.StackOverflowError
> Exception in thread "defaultRpcServer.handler=11,queue=2,port=60020"
> java.lang.StackOverflowError
> Exception in thread "defaultRpcServer.handler=25,queue=1,port=60020"
> java.lang.StackOverflowError
> Exception in thread "defaultRpcServer.handler=20,queue=2,port=60020"
> java.lang.StackOverflowError
> Exception in thread "defaultRpcServer.handler=19,queue=1,port=60020"
> java.lang.StackOverflowError
> Exception in thread "defaultRpcServer.handler=15,queue=0,port=60020"
> java.lang.StackOverflowError
> Exception in thread "defaultRpcServer.handler=1,queue=1,port=60020"
> java.lang.StackOverflowError
> Exception in thread "defaultRpcServer.handler=7,queue=1,port=60020"
> java.lang.StackOverflowError
> Exception in thread "defaultRpcServer.handler=4,queue=1,port=60020"
> java.lang.StackOverflowError
>
> Unfortunately, the exceptions are not timestamped so that I can not
> correlate their occurrence with the exact time when the RS starts filling
> up the queue.
>
>
> On Sat, Aug 23, 2014 at 8:28 PM, Stack <st...@duboce.net> wrote:
>
>> Anything in your .out that could help explain our losing handlers if you
>> can't find anything in the logs?
>>
>> You did the 'snipp' in the below, right Johannes?
>>
>> RS Configuration:
>> ===========================================================
>> <?xml version="1.0" encoding="UTF-8" standalone="no"?><configuration>
>> [snipp] no fancy stuff, all default, except absolute necessary settings
>> [snipp]
>> </configuration>
>>
>> As per Qian, if hbase.ipc.server.callqueue.handler.factor, that could help
>> explain why we have handlers but they are not 'taking' from the call queue,
>> they are stuck taking on those queues that do not have calls queued.
>>
>> St.Ack
>>
>>
>>
>>
>>
>>
>>
>> On Sat, Aug 23, 2014 at 2:56 AM, Qiang Tian <ti...@gmail.com> wrote:
>>
>>> Did you set hbase.ipc.server.callqueue.handler.factor?
>>> it looks there are 3 queues, handlers on queue 1 are all gone as Stack
>>> mentioned. jstack and pastebin regions server log would help.
>>>
>>>
>>>
>>>
>>>
>>> On Sat, Aug 23, 2014 at 7:02 AM, Stack <st...@duboce.net> wrote:
>>>
>>>> On Fri, Aug 22, 2014 at 3:24 PM, Johannes Schaback <
>>>> johannes.schaback@visual-meta.com> wrote:
>>>>
>>>>> ...
>>>>> I grep'ed "defaultRpcServer.handler=" on the log from that particular
>>> RS.
>>>>> The
>>>>> RS started at 15:35. After that, the handlers
>>>>>
>>>>> 6, 24, 0, 15, 28, 26, 7, 19, 21, 3, 5 and 23
>>>>>
>>>>> make an appearance in error messages of HDFS related exceptions:
>>>>>
>>>>> 2014-08-22 13:54:57,470 WARN
>>>>> [defaultRpcServer.handler=6,queue=0,port=60020]
>>> hdfs.BlockReaderFactory:
>>>>> I/O error constructing remote block reader.
>>>>> 2014-08-22 13:54:57,472 WARN
>>>>> [defaultRpcServer.handler=6,queue=0,port=60020] hdfs.DFSClient:
>>>> Connection
>>>>> failure: Failed to connect to /192.168.3.233:50010 for file
>>>>> /hbase/data/default/image/dae1b8e3bfdb608571d09916bf0f
>>>>> a156/cf/866a773857654b0d83275dc4e4558be6 for block
>> BP-1157637685-192.168.3.192-1382642140917:blk_1294949058_255264636:java.io.IOException:
>>>>> Got error for OP_READ_BLOCK, self=/192.168.3.179:37835, remote=/
>>>>> 192.168.3.233:50 010, for file
>> /hbase/data/default/image/dae1b8e3bfdb608571d09916bf0fa156/cf/866a773857654b0d83275dc4e4558be6,
>>>>> for pool BP-1157637685-192.168.3.192-1382642140917 block
>>>>> 1294949058_255264636
>>>>> 2014-08-22 13:56:58,525 WARN
>>>>> [defaultRpcServer.handler=24,queue=0,port=60020]
>>>> hdfs.BlockReaderFactory:
>>>>> I/O error constructing remote block reader.
>>>>> 2014-08-22 13:56:58,799 WARN
>>>>> [defaultRpcServer.handler=24,queue=0,port=60020] hdfs.DFSClient:
>>>>> Connection failure: Failed to connect to /192.168.3.142:50010 for
>> file
>>>>> /hbase/data/default/image/cf493cbd9921ae6ca5e5281cc07
>>>>> 18ca2/cf/7f9618dcdeae40ddbff21165d08e0a83 for block
>> BP-1157637685-192.168.3.192-1382642140917:blk_1294072754_254292863:java.io.IOException:
>>>>> Got error for OP_READ_BLOCK, self=/192.168.3.179:59268, remote=/
>>>>> 192.168.3.142:5 0010, for file
>> /hbase/data/default/image/cf493cbd9921ae6ca5e5281cc0718ca2/cf/7f9618dcdeae40ddbff21165d08e0a83,
>>>>> for pool BP-1157637685-192.168.3.192-1382642140917 block
>>>>> 1294072754_254292863
>>>>> 2014-08-22 14:08:01,632 WARN
>>>>> [defaultRpcServer.handler=0,queue=0,port=60020]
>>> hdfs.BlockReaderFactory:
>>>>> I/O error constructing remote block reader.
>>>>> 2014-08-22 14:08:01,633 WARN
>>>>> [defaultRpcServer.handler=0,queue=0,port=60020] hdfs.DFSClient:
>>>> Connection
>>>>> failure: Failed to connect to /192.168.3.53:50010 for file
>>>>> /hbase/data/default/image/af02bc7fb404f4c054dcd64b44b0e
>>>>> 2a9/cf/a8881b6e2d5d41b3b56fd34fd4ca8ffd for block
>> BP-1157637685-192.168.3.192-1382642140917:blk_1294876123_255182439:java.io.IOException:
>>>>> Got error for OP_READ_BLOCK, self=/192.168.3.179:60737, remote=/
>>>>> 192.168.3.53:5001 0, for file
>> /hbase/data/default/image/af02bc7fb404f4c054dcd64b44b0e2a9/cf/a8881b6e2d5d41b3b56fd34fd4ca8ffd,
>>>>> for pool BP-1157637685-192.168.3.192-1382642140917 block
>>>>> 1294876123_255182439
>>>>> 2014-08-22 14:11:34,192 WARN
>>>>> [defaultRpcServer.handler=15,queue=0,port=60020] ipc.RpcServer:
>>>>> (responseTooSlow):
>> {"processingtimems":15432,"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiReque
>>>>> st)","client":"192.168.3.54:52838
>> ","starttimems":1408709478713,"queuetimems":0,"class":"HRegionServer","responsesize":99638,"method":"Multi"}
>>>>> 2014-08-22 14:22:21,847 WARN
>>>>> [defaultRpcServer.handler=28,queue=1,port=60020]
>>>> hdfs.BlockReaderFactory:
>>>>> I/O error constructing remote block reader.
>>>>> 2014-08-22 14:22:21,848 WARN
>>>>> [defaultRpcServer.handler=28,queue=1,port=60020] hdfs.DFSClient:
>>>>> Connection failure: Failed to connect to /192.168.3.233:50010 for
>> file
>>>>> /hbase/data/default/image/1d33a251862055cd999078bbd10
>>>>> aa44c/cf/a7470351725f4cc192b6210bac9b7c44 for block
>> BP-1157637685-192.168.3.192-1382642140917:blk_1294949316_255264902:java.io.IOException:
>>>>> Got error for OP_READ_BLOCK, self=/192.168.3.179:44787, remote=/
>>>>> 192.168.3.233:5 0010, for file
>> /hbase/data/default/image/1d33a251862055cd999078bbd10aa44c/cf/a7470351725f4cc192b6210bac9b7c44,
>>>>> for pool BP-1157637685-192.168.3.192-1382642140917 block
>>>>> 1294949316_255264902
>>>>> 2014-08-22 14:23:22,628 WARN
>>>>> [defaultRpcServer.handler=26,queue=2,port=60020]
>>>> hdfs.BlockReaderFactory:
>>>>> I/O error constructing remote block reader.
>>>>> 2014-08-22 14:23:22,628 WARN
>>>>> [defaultRpcServer.handler=26,queue=2,port=60020] hdfs.DFSClient:
>>>>> Connection failure: Failed to connect to /192.168.3.142:50010 for
>> file
>>>>> /hbase/data/default/image/4a9c830a4fe006f0a6af7418164
>>>>> dd86d/cf/47d6f2c3f7054d40aed66cae9787c464 for block
>> BP-1157637685-192.168.3.192-1382642140917:blk_1293459594_253612452:java.io.IOException:
>>>>> Got error for OP_READ_BLOCK, self=/192.168.3.179:37660, remote=/
>>>>> 192.168.3.142:5 0010, for file
>> /hbase/data/default/image/4a9c830a4fe006f0a6af7418164dd86d/cf/47d6f2c3f7054d40aed66cae9787c464,
>>>>> for pool BP-1157637685-192.168.3.192-1382642140917 block
>>>>> 1293459594_253612452
>>>>> 2014-08-22 14:25:35,003 WARN
>>>>> [defaultRpcServer.handler=7,queue=1,port=60020]
>>> hdfs.BlockReaderFactory:
>>>>> I/O error constructing remote block reader.
>>>>> 2014-08-22 14:25:35,004 WARN
>>>>> [defaultRpcServer.handler=7,queue=1,port=60020] hdfs.DFSClient:
>>>> Connection
>>>>> failure: Failed to connect to /192.168.3.51:50010 for file
>>>>> /hbase/data/default/run_automaton_cache/da03f8123004be3
>>>>> 2659e1c8a51afbbf8/cf/1daf234e740f4b00889bb60e574dc79b for block
>> BP-1157637685-192.168.3.192-1382642140917:blk_1294614349_254896345:java.io.IOException:
>>>>> Got error for OP_READ_BLOCK, self=/192.168.3.179:53627, remote=/192
>>>>> .168.3.51:50010, for file
>> /hbase/data/default/run_automaton_cache/da03f8123004be32659e1c8a51afbbf8/cf/1daf234e740f4b00889bb60e574dc79b,
>>>>> for pool BP-1157637685-192.168.3.192-1382642140917 block
>>>>> 1294614349_254896345
>>>>> 2014-08-22 14:25:46,831 WARN
>>>>> [defaultRpcServer.handler=19,queue=1,port=60020]
>>>> hdfs.BlockReaderFactory:
>>>>> I/O error constructing remote block reader.
>>>>> 2014-08-22 14:25:46,832 WARN
>>>>> [defaultRpcServer.handler=19,queue=1,port=60020] hdfs.DFSClient:
>>>>> Connection failure: Failed to connect to /192.168.3.30:50010 for
>> file
>>>>> /hbase/data/default/image/1c3bab43e260ddb46a06cf04e293
>>>>> e386/cf/740f8240ac9a4abc9e5fcf6ec7df18bc for block
>> BP-1157637685-192.168.3.192-1382642140917:blk_1294151564_254380545:java.io.IOException:
>>>>> Got error for OP_READ_BLOCK, self=/192.168.3.179:55660, remote=/
>>>>> 192.168.3.30:500 10, for file
>> /hbase/data/default/image/1c3bab43e260ddb46a06cf04e293e386/cf/740f8240ac9a4abc9e5fcf6ec7df18bc,
>>>>> for pool BP-1157637685-192.168.3.192-1382642140917 block
>>>>> 1294151564_254380545
>>>>> 2014-08-22 14:28:22,395 WARN
>>>>> [defaultRpcServer.handler=26,queue=2,port=60020]
>>>> hdfs.BlockReaderFactory:
>>>>> I/O error constructing remote block reader.
>>>>> 2014-08-22 14:28:22,397 WARN
>>>>> [defaultRpcServer.handler=26,queue=2,port=60020] hdfs.DFSClient:
>>>>> Connection failure: Failed to connect to /192.168.3.143:50010 for
>> file
>>>>> /hbase/data/default/image/b4b48e20a606c431e393b674f92
>>>>> 79daf/cf/76ff3628306a48f187a285b2a21d9ac9 for block
>> BP-1157637685-192.168.3.192-1382642140917:blk_1294308080_254554506:java.io.IOException:
>>>>> Got error for OP_READ_BLOCK, self=/192.168.3.179:51759, remote=/
>>>>> 192.168.3.143:5 0010, for file
>> /hbase/data/default/image/b4b48e20a606c431e393b674f9279daf/cf/76ff3628306a48f187a285b2a21d9ac9,
>>>>> for pool BP-1157637685-192.168.3.192-1382642140917 block
>>>>> 1294308080_254554506
>>>>> 2014-08-22 14:30:29,395 WARN
>>>>> [defaultRpcServer.handler=21,queue=0,port=60020]
>>>> hdfs.BlockReaderFactory:
>>>>> I/O error constructing remote block reader.
>>>>> 2014-08-22 14:30:29,396 WARN
>>>>> [defaultRpcServer.handler=21,queue=0,port=60020] hdfs.DFSClient:
>>>>> Connection failure: Failed to connect to /192.168.3.30:50010 for
>> file
>>>>> /hbase/data/default/image/77ceb80d73c065f1a4ba2ad3b7cf
>>>>> 04a2/cf/151c244177ac42b5996af0c9052660cc for block
>> BP-1157637685-192.168.3.192-1382642140917:blk_1294365296_254618150:java.io.IOException:
>>>>> Got error for OP_READ_BLOCK, self=/192.168.3.179:57238, remote=/
>>>>> 192.168.3.30:500 10, for file
>> /hbase/data/default/image/77ceb80d73c065f1a4ba2ad3b7cf04a2/cf/151c244177ac42b5996af0c9052660cc,
>>>>> for pool BP-1157637685-192.168.3.192-1382642140917 block
>>>>> 1294365296_254618150
>>>>> 2014-08-22 14:31:34,016 WARN
>>>>> [defaultRpcServer.handler=6,queue=0,port=60020]
>>> hdfs.BlockReaderFactory:
>>>>> I/O error constructing remote block reader.
>>>>> 2014-08-22 14:31:34,018 WARN
>>>>> [defaultRpcServer.handler=6,queue=0,port=60020] hdfs.DFSClient:
>>>> Connection
>>>>> failure: Failed to connect to /192.168.3.212:50010 for file
>>>>> /hbase/data/default/image/45578853a5c807919578043c7715
>>>>> 1efa/cf/40c00d49a7494929a319467d82b383da for block
>> BP-1157637685-192.168.3.192-1382642140917:blk_1294930505_255244337:java.io.IOException:
>>>>> Got error for OP_READ_BLOCK, self=/192.168.3.179:33973, remote=/
>>>>> 192.168.3.212:50 010, for file
>> /hbase/data/default/image/45578853a5c807919578043c77151efa/cf/40c00d49a7494929a319467d82b383da,
>>>>> for pool BP-1157637685-192.168.3.192-1382642140917 block
>>>>> 1294930505_255244337
>>>>> 2014-08-22 14:33:55,994 WARN
>>>>> [defaultRpcServer.handler=3,queue=0,port=60020]
>>> hdfs.BlockReaderFactory:
>>>>> I/O error constructing remote block reader.
>>>>> 2014-08-22 14:33:55,995 WARN
>>>>> [defaultRpcServer.handler=3,queue=0,port=60020] hdfs.DFSClient:
>>>> Connection
>>>>> failure: Failed to connect to /192.168.3.52:50010 for file
>>>>> /hbase/data/default/image/1f07e0160fbba29a2c26104dd5966
>>>>> 39f/cf/7e36136497fc4a5a9889797a1dfb5d3b for block
>> BP-1157637685-192.168.3.192-1382642140917:blk_1294915607_255226575:java.io.IOException:
>>>>> Got error for OP_READ_BLOCK, self=/192.168.3.179:41017, remote=/
>>>>> 192.168.3.52:5001 0, for file
>> /hbase/data/default/image/1f07e0160fbba29a2c26104dd596639f/cf/7e36136497fc4a5a9889797a1dfb5d3b,
>>>>> for pool BP-1157637685-192.168.3.192-1382642140917 block
>>>>> 1294915607_255226575
>>>>> 2014-08-22 14:44:14,301 WARN
>>>>> [defaultRpcServer.handler=3,queue=0,port=60020]
>>> hdfs.BlockReaderFactory:
>>>>> I/O error constructing remote block reader.
>>>>> 2014-08-22 14:44:14,302 WARN
>>>>> [defaultRpcServer.handler=3,queue=0,port=60020] hdfs.DFSClient:
>>>> Connection
>>>>> failure: Failed to connec
Re: single RegionServer stuck, causing cluster to hang
Posted by Johannes Schaback <jo...@visual-meta.com>.
Hi,
we had to reduce load on the cluster yesterday night which reduced the
frequency of the phenomenon. That is why I could not get a jstack dump yet
because it did not occur since a couple hours. We will now get the load
back up hoping to trigger it again.
Yes, I cut out the properties from the /debug dump because they are all
standard. We have hbase.ipc.server.callqueue.handler.factor to the standard
<property>
<name>hbase.ipc.server.callqueue.handler.factor</name>
<value>0.1</value>
<source>hbase-default.xml</source>
</property>
You find the complete config of the RS here: http://pastebin.com/iF1ibFb1
The hint about the .out files was a great one (I never really looked at
them actually). The .out files are flooded with StackOverflowExceptions:
Exception in thread "defaultRpcServer.handler=5,queue=2,port=60020"
java.lang.StackOverflowError
at org.apache.hadoop.hbase.CellUtil$1.advance(CellUtil.java:210)
at org.apache.hadoop.hbase.CellUtil$1.advance(CellUtil.java:210)
at org.apache.hadoop.hbase.CellUtil$1.advance(CellUtil.java:210)
at org.apache.hadoop.hbase.CellUtil$1.advance(CellUtil.java:210)
(and so on...)
Filtering the .out file for "Exception" shows that several handlers crashed
like that:
Exception in thread "defaultRpcServer.handler=5,queue=2,port=60020"
java.lang.StackOverflowError
Exception in thread "defaultRpcServer.handler=18,queue=0,port=60020"
java.lang.StackOverflowError
Exception in thread "defaultRpcServer.handler=23,queue=2,port=60020"
java.lang.StackOverflowError
Exception in thread "defaultRpcServer.handler=24,queue=0,port=60020"
java.lang.StackOverflowError
Exception in thread "defaultRpcServer.handler=2,queue=2,port=60020"
java.lang.StackOverflowError
Exception in thread "defaultRpcServer.handler=11,queue=2,port=60020"
java.lang.StackOverflowError
Exception in thread "defaultRpcServer.handler=25,queue=1,port=60020"
java.lang.StackOverflowError
Exception in thread "defaultRpcServer.handler=20,queue=2,port=60020"
java.lang.StackOverflowError
Exception in thread "defaultRpcServer.handler=19,queue=1,port=60020"
java.lang.StackOverflowError
Exception in thread "defaultRpcServer.handler=15,queue=0,port=60020"
java.lang.StackOverflowError
Exception in thread "defaultRpcServer.handler=1,queue=1,port=60020"
java.lang.StackOverflowError
Exception in thread "defaultRpcServer.handler=7,queue=1,port=60020"
java.lang.StackOverflowError
Exception in thread "defaultRpcServer.handler=4,queue=1,port=60020"
java.lang.StackOverflowError
Unfortunately, the exceptions are not timestamped so that I can not
correlate their occurrence with the exact time when the RS starts filling
up the queue.
On Sat, Aug 23, 2014 at 8:28 PM, Stack <st...@duboce.net> wrote:
> Anything in your .out that could help explain our losing handlers if you
> can't find anything in the logs?
>
> You did the 'snipp' in the below, right Johannes?
>
> RS Configuration:
> ===========================================================
> <?xml version="1.0" encoding="UTF-8" standalone="no"?><configuration>
> [snipp] no fancy stuff, all default, except absolute necessary settings
> [snipp]
> </configuration>
>
> As per Qian, if hbase.ipc.server.callqueue.handler.factor, that could help
> explain why we have handlers but they are not 'taking' from the call queue,
> they are stuck taking on those queues that do not have calls queued.
>
> St.Ack
>
>
>
>
>
>
>
> On Sat, Aug 23, 2014 at 2:56 AM, Qiang Tian <ti...@gmail.com> wrote:
>
> > Did you set hbase.ipc.server.callqueue.handler.factor?
> > it looks there are 3 queues, handlers on queue 1 are all gone as Stack
> > mentioned. jstack and pastebin regions server log would help.
> >
> >
> >
> >
> >
> > On Sat, Aug 23, 2014 at 7:02 AM, Stack <st...@duboce.net> wrote:
> >
> > > On Fri, Aug 22, 2014 at 3:24 PM, Johannes Schaback <
> > > johannes.schaback@visual-meta.com> wrote:
> > >
> > > > ...
> > > > I grep'ed "defaultRpcServer.handler=" on the log from that particular
> > RS.
> > > > The
> > > > RS started at 15:35. After that, the handlers
> > > >
> > > > 6, 24, 0, 15, 28, 26, 7, 19, 21, 3, 5 and 23
> > > >
> > > > make an appearance in error messages of HDFS related exceptions:
> > > >
> > > > 2014-08-22 13:54:57,470 WARN
> > > > [defaultRpcServer.handler=6,queue=0,port=60020]
> > hdfs.BlockReaderFactory:
> > > > I/O error constructing remote block reader.
> > > > 2014-08-22 13:54:57,472 WARN
> > > > [defaultRpcServer.handler=6,queue=0,port=60020] hdfs.DFSClient:
> > > Connection
> > > > failure: Failed to connect to /192.168.3.233:50010 for file
> > > > /hbase/data/default/image/dae1b8e3bfdb608571d09916bf0f
> > > > a156/cf/866a773857654b0d83275dc4e4558be6 for block
> > > >
> > > >
> > >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1294949058_255264636:java.io.IOException:
> > > > Got error for OP_READ_BLOCK, self=/192.168.3.179:37835, remote=/
> > > > 192.168.3.233:50 010, for file
> > > >
> > > >
> > >
> >
> /hbase/data/default/image/dae1b8e3bfdb608571d09916bf0fa156/cf/866a773857654b0d83275dc4e4558be6,
> > > > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > > > 1294949058_255264636
> > > > 2014-08-22 13:56:58,525 WARN
> > > > [defaultRpcServer.handler=24,queue=0,port=60020]
> > > hdfs.BlockReaderFactory:
> > > > I/O error constructing remote block reader.
> > > > 2014-08-22 13:56:58,799 WARN
> > > > [defaultRpcServer.handler=24,queue=0,port=60020] hdfs.DFSClient:
> > > > Connection failure: Failed to connect to /192.168.3.142:50010 for
> file
> > > > /hbase/data/default/image/cf493cbd9921ae6ca5e5281cc07
> > > > 18ca2/cf/7f9618dcdeae40ddbff21165d08e0a83 for block
> > > >
> > > >
> > >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1294072754_254292863:java.io.IOException:
> > > > Got error for OP_READ_BLOCK, self=/192.168.3.179:59268, remote=/
> > > > 192.168.3.142:5 0010, for file
> > > >
> > > >
> > >
> >
> /hbase/data/default/image/cf493cbd9921ae6ca5e5281cc0718ca2/cf/7f9618dcdeae40ddbff21165d08e0a83,
> > > > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > > > 1294072754_254292863
> > > > 2014-08-22 14:08:01,632 WARN
> > > > [defaultRpcServer.handler=0,queue=0,port=60020]
> > hdfs.BlockReaderFactory:
> > > > I/O error constructing remote block reader.
> > > > 2014-08-22 14:08:01,633 WARN
> > > > [defaultRpcServer.handler=0,queue=0,port=60020] hdfs.DFSClient:
> > > Connection
> > > > failure: Failed to connect to /192.168.3.53:50010 for file
> > > > /hbase/data/default/image/af02bc7fb404f4c054dcd64b44b0e
> > > > 2a9/cf/a8881b6e2d5d41b3b56fd34fd4ca8ffd for block
> > > >
> > > >
> > >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1294876123_255182439:java.io.IOException:
> > > > Got error for OP_READ_BLOCK, self=/192.168.3.179:60737, remote=/
> > > > 192.168.3.53:5001 0, for file
> > > >
> > > >
> > >
> >
> /hbase/data/default/image/af02bc7fb404f4c054dcd64b44b0e2a9/cf/a8881b6e2d5d41b3b56fd34fd4ca8ffd,
> > > > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > > > 1294876123_255182439
> > > > 2014-08-22 14:11:34,192 WARN
> > > > [defaultRpcServer.handler=15,queue=0,port=60020] ipc.RpcServer:
> > > > (responseTooSlow):
> > > >
> > > >
> > >
> >
> {"processingtimems":15432,"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiReque
> > > > st)","client":"192.168.3.54:52838
> > > >
> > > >
> > >
> >
> ","starttimems":1408709478713,"queuetimems":0,"class":"HRegionServer","responsesize":99638,"method":"Multi"}
> > > > 2014-08-22 14:22:21,847 WARN
> > > > [defaultRpcServer.handler=28,queue=1,port=60020]
> > > hdfs.BlockReaderFactory:
> > > > I/O error constructing remote block reader.
> > > > 2014-08-22 14:22:21,848 WARN
> > > > [defaultRpcServer.handler=28,queue=1,port=60020] hdfs.DFSClient:
> > > > Connection failure: Failed to connect to /192.168.3.233:50010 for
> file
> > > > /hbase/data/default/image/1d33a251862055cd999078bbd10
> > > > aa44c/cf/a7470351725f4cc192b6210bac9b7c44 for block
> > > >
> > > >
> > >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1294949316_255264902:java.io.IOException:
> > > > Got error for OP_READ_BLOCK, self=/192.168.3.179:44787, remote=/
> > > > 192.168.3.233:5 0010, for file
> > > >
> > > >
> > >
> >
> /hbase/data/default/image/1d33a251862055cd999078bbd10aa44c/cf/a7470351725f4cc192b6210bac9b7c44,
> > > > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > > > 1294949316_255264902
> > > > 2014-08-22 14:23:22,628 WARN
> > > > [defaultRpcServer.handler=26,queue=2,port=60020]
> > > hdfs.BlockReaderFactory:
> > > > I/O error constructing remote block reader.
> > > > 2014-08-22 14:23:22,628 WARN
> > > > [defaultRpcServer.handler=26,queue=2,port=60020] hdfs.DFSClient:
> > > > Connection failure: Failed to connect to /192.168.3.142:50010 for
> file
> > > > /hbase/data/default/image/4a9c830a4fe006f0a6af7418164
> > > > dd86d/cf/47d6f2c3f7054d40aed66cae9787c464 for block
> > > >
> > > >
> > >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1293459594_253612452:java.io.IOException:
> > > > Got error for OP_READ_BLOCK, self=/192.168.3.179:37660, remote=/
> > > > 192.168.3.142:5 0010, for file
> > > >
> > > >
> > >
> >
> /hbase/data/default/image/4a9c830a4fe006f0a6af7418164dd86d/cf/47d6f2c3f7054d40aed66cae9787c464,
> > > > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > > > 1293459594_253612452
> > > > 2014-08-22 14:25:35,003 WARN
> > > > [defaultRpcServer.handler=7,queue=1,port=60020]
> > hdfs.BlockReaderFactory:
> > > > I/O error constructing remote block reader.
> > > > 2014-08-22 14:25:35,004 WARN
> > > > [defaultRpcServer.handler=7,queue=1,port=60020] hdfs.DFSClient:
> > > Connection
> > > > failure: Failed to connect to /192.168.3.51:50010 for file
> > > > /hbase/data/default/run_automaton_cache/da03f8123004be3
> > > > 2659e1c8a51afbbf8/cf/1daf234e740f4b00889bb60e574dc79b for block
> > > >
> > > >
> > >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1294614349_254896345:java.io.IOException:
> > > > Got error for OP_READ_BLOCK, self=/192.168.3.179:53627, remote=/192
> > > > .168.3.51:50010, for file
> > > >
> > > >
> > >
> >
> /hbase/data/default/run_automaton_cache/da03f8123004be32659e1c8a51afbbf8/cf/1daf234e740f4b00889bb60e574dc79b,
> > > > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > > > 1294614349_254896345
> > > > 2014-08-22 14:25:46,831 WARN
> > > > [defaultRpcServer.handler=19,queue=1,port=60020]
> > > hdfs.BlockReaderFactory:
> > > > I/O error constructing remote block reader.
> > > > 2014-08-22 14:25:46,832 WARN
> > > > [defaultRpcServer.handler=19,queue=1,port=60020] hdfs.DFSClient:
> > > > Connection failure: Failed to connect to /192.168.3.30:50010 for
> file
> > > > /hbase/data/default/image/1c3bab43e260ddb46a06cf04e293
> > > > e386/cf/740f8240ac9a4abc9e5fcf6ec7df18bc for block
> > > >
> > > >
> > >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1294151564_254380545:java.io.IOException:
> > > > Got error for OP_READ_BLOCK, self=/192.168.3.179:55660, remote=/
> > > > 192.168.3.30:500 10, for file
> > > >
> > > >
> > >
> >
> /hbase/data/default/image/1c3bab43e260ddb46a06cf04e293e386/cf/740f8240ac9a4abc9e5fcf6ec7df18bc,
> > > > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > > > 1294151564_254380545
> > > > 2014-08-22 14:28:22,395 WARN
> > > > [defaultRpcServer.handler=26,queue=2,port=60020]
> > > hdfs.BlockReaderFactory:
> > > > I/O error constructing remote block reader.
> > > > 2014-08-22 14:28:22,397 WARN
> > > > [defaultRpcServer.handler=26,queue=2,port=60020] hdfs.DFSClient:
> > > > Connection failure: Failed to connect to /192.168.3.143:50010 for
> file
> > > > /hbase/data/default/image/b4b48e20a606c431e393b674f92
> > > > 79daf/cf/76ff3628306a48f187a285b2a21d9ac9 for block
> > > >
> > > >
> > >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1294308080_254554506:java.io.IOException:
> > > > Got error for OP_READ_BLOCK, self=/192.168.3.179:51759, remote=/
> > > > 192.168.3.143:5 0010, for file
> > > >
> > > >
> > >
> >
> /hbase/data/default/image/b4b48e20a606c431e393b674f9279daf/cf/76ff3628306a48f187a285b2a21d9ac9,
> > > > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > > > 1294308080_254554506
> > > > 2014-08-22 14:30:29,395 WARN
> > > > [defaultRpcServer.handler=21,queue=0,port=60020]
> > > hdfs.BlockReaderFactory:
> > > > I/O error constructing remote block reader.
> > > > 2014-08-22 14:30:29,396 WARN
> > > > [defaultRpcServer.handler=21,queue=0,port=60020] hdfs.DFSClient:
> > > > Connection failure: Failed to connect to /192.168.3.30:50010 for
> file
> > > > /hbase/data/default/image/77ceb80d73c065f1a4ba2ad3b7cf
> > > > 04a2/cf/151c244177ac42b5996af0c9052660cc for block
> > > >
> > > >
> > >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1294365296_254618150:java.io.IOException:
> > > > Got error for OP_READ_BLOCK, self=/192.168.3.179:57238, remote=/
> > > > 192.168.3.30:500 10, for file
> > > >
> > > >
> > >
> >
> /hbase/data/default/image/77ceb80d73c065f1a4ba2ad3b7cf04a2/cf/151c244177ac42b5996af0c9052660cc,
> > > > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > > > 1294365296_254618150
> > > > 2014-08-22 14:31:34,016 WARN
> > > > [defaultRpcServer.handler=6,queue=0,port=60020]
> > hdfs.BlockReaderFactory:
> > > > I/O error constructing remote block reader.
> > > > 2014-08-22 14:31:34,018 WARN
> > > > [defaultRpcServer.handler=6,queue=0,port=60020] hdfs.DFSClient:
> > > Connection
> > > > failure: Failed to connect to /192.168.3.212:50010 for file
> > > > /hbase/data/default/image/45578853a5c807919578043c7715
> > > > 1efa/cf/40c00d49a7494929a319467d82b383da for block
> > > >
> > > >
> > >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1294930505_255244337:java.io.IOException:
> > > > Got error for OP_READ_BLOCK, self=/192.168.3.179:33973, remote=/
> > > > 192.168.3.212:50 010, for file
> > > >
> > > >
> > >
> >
> /hbase/data/default/image/45578853a5c807919578043c77151efa/cf/40c00d49a7494929a319467d82b383da,
> > > > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > > > 1294930505_255244337
> > > > 2014-08-22 14:33:55,994 WARN
> > > > [defaultRpcServer.handler=3,queue=0,port=60020]
> > hdfs.BlockReaderFactory:
> > > > I/O error constructing remote block reader.
> > > > 2014-08-22 14:33:55,995 WARN
> > > > [defaultRpcServer.handler=3,queue=0,port=60020] hdfs.DFSClient:
> > > Connection
> > > > failure: Failed to connect to /192.168.3.52:50010 for file
> > > > /hbase/data/default/image/1f07e0160fbba29a2c26104dd5966
> > > > 39f/cf/7e36136497fc4a5a9889797a1dfb5d3b for block
> > > >
> > > >
> > >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1294915607_255226575:java.io.IOException:
> > > > Got error for OP_READ_BLOCK, self=/192.168.3.179:41017, remote=/
> > > > 192.168.3.52:5001 0, for file
> > > >
> > > >
> > >
> >
> /hbase/data/default/image/1f07e0160fbba29a2c26104dd596639f/cf/7e36136497fc4a5a9889797a1dfb5d3b,
> > > > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > > > 1294915607_255226575
> > > > 2014-08-22 14:44:14,301 WARN
> > > > [defaultRpcServer.handler=3,queue=0,port=60020]
> > hdfs.BlockReaderFactory:
> > > > I/O error constructing remote block reader.
> > > > 2014-08-22 14:44:14,302 WARN
> > > > [defaultRpcServer.handler=3,queue=0,port=60020] hdfs.DFSClient:
> > > Connection
> > > > failure: Failed to connect to /192.168.3.154:50010 for file
> > > > /hbase/data/default/image/bb082a5f098d1d2a95365545349c
> > > > 77a4/cf/1af06c6fd36045bab888b4ec18e45f0f for block
> > > >
> > > >
> > >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1294911152_255221397:java.io.IOException:
> > > > Got error for OP_READ_BLOCK, self=/192.168.3.179:44711, remote=/
> > > > 192.168.3.154:50 010, for file
> > > >
> > > >
> > >
> >
> /hbase/data/default/image/bb082a5f098d1d2a95365545349c77a4/cf/1af06c6fd36045bab888b4ec18e45f0f,
> > > > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > > > 1294911152_255221397
> > > > 77a4/cf/1af06c6fd36045bab888b4ec18e45f0f for block
> > > >
> > > >
> > >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1294911152_255221397:java.io.IOException:
> > > > Got error for OP_READ_BLOCK, self=/192.168.3.179:44711, remote=/
> > > > 192.168.3.154:50 010, for file
> > > >
> > > >
> > >
> >
> /hbase/data/default/image/bb082a5f098d1d2a95365545349c77a4/cf/1af06c6fd36045bab888b4ec18e45f0f,
> > > > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > > > 1294911152_255221397
> > > > 2014-08-22 14:49:14,780 WARN
> > > > [defaultRpcServer.handler=5,queue=2,port=60020] ipc.RpcServer:
> > > > (responseTooSlow):
> > > >
> > > >
> > >
> >
> {"processingtimems":15494,"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiReques
> > > > t)","client":"192.168.3.11:53884
> > > >
> > > >
> > >
> >
> ","starttimems":1408711739283,"queuetimems":0,"class":"HRegionServer","responsesize":483492,"method":"Multi"}
> > > > 2014-08-22 14:50:37,900 WARN
> > > > [defaultRpcServer.handler=7,queue=1,port=60020]
> > hdfs.BlockReaderFactory:
> > > > I/O error constructing remote block reader.
> > > > 2014-08-22 14:50:37,901 WARN
> > > > [defaultRpcServer.handler=7,queue=1,port=60020] hdfs.DFSClient:
> > > Connection
> > > > failure: Failed to connect to /192.168.3.64:50010 for file
> > > > /hbase/data/default/image/a50d67b81c44c864265f5030c7c39
> > > > 959/cf/840ff51e591946c487413273d5341a24 for block
> > > >
> > > >
> > >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1294309355_254555905:java.io.IOException:
> > > > Got error for OP_READ_BLOCK, self=/192.168.3.179:45043, remote=/
> > > > 192.168.3.64:5001 0, for file
> > > >
> > > >
> > >
> >
> /hbase/data/default/image/a50d67b81c44c864265f5030c7c39959/cf/840ff51e591946c487413273d5341a24,
> > > > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > > > 1294309355_254555905
> > > > 2014-08-22 15:09:35,289 WARN
> > > > [defaultRpcServer.handler=23,queue=2,port=60020]
> > > hdfs.BlockReaderFactory:
> > > > I/O error constructing remote block reader.
> > > > 2014-08-22 15:09:35,289 WARN
> > > > [defaultRpcServer.handler=23,queue=2,port=60020] hdfs.DFSClient:
> > > > Connection failure: Failed to connect to /192.168.3.63:50010 for
> file
> > > > /hbase/data/default/image/1db3051c8d41943892c0230cb75b
> > > > c1f2/cf/75ae96ea8ca2477e8f83291d7e1fe7cb for block
> > > >
> > > >
> > >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1293205473_253317737:java.io.IOException:
> > > > Got error for OP_READ_BLOCK, self=/192.168.3.179:56326, remote=/
> > > > 192.168.3.63:500 10, for file
> > > >
> > > >
> > >
> >
> /hbase/data/default/image/1db3051c8d41943892c0230cb75bc1f2/cf/75ae96ea8ca2477e8f83291d7e1fe7cb,
> > > > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > > > 1293205473_253317737
> > > > 2014-08-22 15:18:38,891 WARN
> > > > [defaultRpcServer.handler=5,queue=2,port=60020] ipc.RpcServer:
> > > > (responseTooSlow):
> > > >
> > > >
> > >
> >
> {"processingtimems":26092,"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiReques
> > > > t)","client":"192.168.3.91:40189
> > > >
> > > >
> > >
> >
> ","starttimems":1408713492722,"queuetimems":0,"class":"HRegionServer","responsesize":483492,"method":"Multi"}
> > > >
> > > > What's interesting is that not all crashed handlers are later missing
> > > after
> > > > the freeze.
> > > >
> > > > Does that substantiate your conjecture?
> > > >
> > > >
> > > I saw those in your log. They are WARN from DFSClient which should be
> > ok.
> > > Any other exceptions going on in your logs that could explain our
> > handler
> > > loss? Or, can you see any thing particular about say the last mention
> of
> > > defaultRpcServer.handler=6 say?
> > >
> > > St.Ack
> > >
> >
>
--
LadenZeile.de <http://www.ladenzeile.de/>
powered by Visual Meta GmbH - www.visual-meta.com
Tel.: +49 30 / 609 84 88 20
Fax: +49 30 / 609 84 88 21
E-Mail: johannes.schaback@visual-meta.com
Visual Meta GmbH, Schützenstraße 25, 10117 Berlin
Geschäftsführer: Robert M. Maier, Johannes Schaback
Handelsregister HRB 115795 B, Amtsgericht Charlottenburg
USt-IdNr.: DE263760203
Re: single RegionServer stuck, causing cluster to hang
Posted by Stack <st...@duboce.net>.
Anything in your .out that could help explain our losing handlers if you
can't find anything in the logs?
You did the 'snipp' in the below, right Johannes?
RS Configuration:
===========================================================
<?xml version="1.0" encoding="UTF-8" standalone="no"?><configuration>
[snipp] no fancy stuff, all default, except absolute necessary settings
[snipp]
</configuration>
As per Qian, if hbase.ipc.server.callqueue.handler.factor, that could help
explain why we have handlers but they are not 'taking' from the call queue,
they are stuck taking on those queues that do not have calls queued.
St.Ack
On Sat, Aug 23, 2014 at 2:56 AM, Qiang Tian <ti...@gmail.com> wrote:
> Did you set hbase.ipc.server.callqueue.handler.factor?
> it looks there are 3 queues, handlers on queue 1 are all gone as Stack
> mentioned. jstack and pastebin regions server log would help.
>
>
>
>
>
> On Sat, Aug 23, 2014 at 7:02 AM, Stack <st...@duboce.net> wrote:
>
> > On Fri, Aug 22, 2014 at 3:24 PM, Johannes Schaback <
> > johannes.schaback@visual-meta.com> wrote:
> >
> > > ...
> > > I grep'ed "defaultRpcServer.handler=" on the log from that particular
> RS.
> > > The
> > > RS started at 15:35. After that, the handlers
> > >
> > > 6, 24, 0, 15, 28, 26, 7, 19, 21, 3, 5 and 23
> > >
> > > make an appearance in error messages of HDFS related exceptions:
> > >
> > > 2014-08-22 13:54:57,470 WARN
> > > [defaultRpcServer.handler=6,queue=0,port=60020]
> hdfs.BlockReaderFactory:
> > > I/O error constructing remote block reader.
> > > 2014-08-22 13:54:57,472 WARN
> > > [defaultRpcServer.handler=6,queue=0,port=60020] hdfs.DFSClient:
> > Connection
> > > failure: Failed to connect to /192.168.3.233:50010 for file
> > > /hbase/data/default/image/dae1b8e3bfdb608571d09916bf0f
> > > a156/cf/866a773857654b0d83275dc4e4558be6 for block
> > >
> > >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1294949058_255264636:java.io.IOException:
> > > Got error for OP_READ_BLOCK, self=/192.168.3.179:37835, remote=/
> > > 192.168.3.233:50 010, for file
> > >
> > >
> >
> /hbase/data/default/image/dae1b8e3bfdb608571d09916bf0fa156/cf/866a773857654b0d83275dc4e4558be6,
> > > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > > 1294949058_255264636
> > > 2014-08-22 13:56:58,525 WARN
> > > [defaultRpcServer.handler=24,queue=0,port=60020]
> > hdfs.BlockReaderFactory:
> > > I/O error constructing remote block reader.
> > > 2014-08-22 13:56:58,799 WARN
> > > [defaultRpcServer.handler=24,queue=0,port=60020] hdfs.DFSClient:
> > > Connection failure: Failed to connect to /192.168.3.142:50010 for file
> > > /hbase/data/default/image/cf493cbd9921ae6ca5e5281cc07
> > > 18ca2/cf/7f9618dcdeae40ddbff21165d08e0a83 for block
> > >
> > >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1294072754_254292863:java.io.IOException:
> > > Got error for OP_READ_BLOCK, self=/192.168.3.179:59268, remote=/
> > > 192.168.3.142:5 0010, for file
> > >
> > >
> >
> /hbase/data/default/image/cf493cbd9921ae6ca5e5281cc0718ca2/cf/7f9618dcdeae40ddbff21165d08e0a83,
> > > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > > 1294072754_254292863
> > > 2014-08-22 14:08:01,632 WARN
> > > [defaultRpcServer.handler=0,queue=0,port=60020]
> hdfs.BlockReaderFactory:
> > > I/O error constructing remote block reader.
> > > 2014-08-22 14:08:01,633 WARN
> > > [defaultRpcServer.handler=0,queue=0,port=60020] hdfs.DFSClient:
> > Connection
> > > failure: Failed to connect to /192.168.3.53:50010 for file
> > > /hbase/data/default/image/af02bc7fb404f4c054dcd64b44b0e
> > > 2a9/cf/a8881b6e2d5d41b3b56fd34fd4ca8ffd for block
> > >
> > >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1294876123_255182439:java.io.IOException:
> > > Got error for OP_READ_BLOCK, self=/192.168.3.179:60737, remote=/
> > > 192.168.3.53:5001 0, for file
> > >
> > >
> >
> /hbase/data/default/image/af02bc7fb404f4c054dcd64b44b0e2a9/cf/a8881b6e2d5d41b3b56fd34fd4ca8ffd,
> > > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > > 1294876123_255182439
> > > 2014-08-22 14:11:34,192 WARN
> > > [defaultRpcServer.handler=15,queue=0,port=60020] ipc.RpcServer:
> > > (responseTooSlow):
> > >
> > >
> >
> {"processingtimems":15432,"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiReque
> > > st)","client":"192.168.3.54:52838
> > >
> > >
> >
> ","starttimems":1408709478713,"queuetimems":0,"class":"HRegionServer","responsesize":99638,"method":"Multi"}
> > > 2014-08-22 14:22:21,847 WARN
> > > [defaultRpcServer.handler=28,queue=1,port=60020]
> > hdfs.BlockReaderFactory:
> > > I/O error constructing remote block reader.
> > > 2014-08-22 14:22:21,848 WARN
> > > [defaultRpcServer.handler=28,queue=1,port=60020] hdfs.DFSClient:
> > > Connection failure: Failed to connect to /192.168.3.233:50010 for file
> > > /hbase/data/default/image/1d33a251862055cd999078bbd10
> > > aa44c/cf/a7470351725f4cc192b6210bac9b7c44 for block
> > >
> > >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1294949316_255264902:java.io.IOException:
> > > Got error for OP_READ_BLOCK, self=/192.168.3.179:44787, remote=/
> > > 192.168.3.233:5 0010, for file
> > >
> > >
> >
> /hbase/data/default/image/1d33a251862055cd999078bbd10aa44c/cf/a7470351725f4cc192b6210bac9b7c44,
> > > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > > 1294949316_255264902
> > > 2014-08-22 14:23:22,628 WARN
> > > [defaultRpcServer.handler=26,queue=2,port=60020]
> > hdfs.BlockReaderFactory:
> > > I/O error constructing remote block reader.
> > > 2014-08-22 14:23:22,628 WARN
> > > [defaultRpcServer.handler=26,queue=2,port=60020] hdfs.DFSClient:
> > > Connection failure: Failed to connect to /192.168.3.142:50010 for file
> > > /hbase/data/default/image/4a9c830a4fe006f0a6af7418164
> > > dd86d/cf/47d6f2c3f7054d40aed66cae9787c464 for block
> > >
> > >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1293459594_253612452:java.io.IOException:
> > > Got error for OP_READ_BLOCK, self=/192.168.3.179:37660, remote=/
> > > 192.168.3.142:5 0010, for file
> > >
> > >
> >
> /hbase/data/default/image/4a9c830a4fe006f0a6af7418164dd86d/cf/47d6f2c3f7054d40aed66cae9787c464,
> > > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > > 1293459594_253612452
> > > 2014-08-22 14:25:35,003 WARN
> > > [defaultRpcServer.handler=7,queue=1,port=60020]
> hdfs.BlockReaderFactory:
> > > I/O error constructing remote block reader.
> > > 2014-08-22 14:25:35,004 WARN
> > > [defaultRpcServer.handler=7,queue=1,port=60020] hdfs.DFSClient:
> > Connection
> > > failure: Failed to connect to /192.168.3.51:50010 for file
> > > /hbase/data/default/run_automaton_cache/da03f8123004be3
> > > 2659e1c8a51afbbf8/cf/1daf234e740f4b00889bb60e574dc79b for block
> > >
> > >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1294614349_254896345:java.io.IOException:
> > > Got error for OP_READ_BLOCK, self=/192.168.3.179:53627, remote=/192
> > > .168.3.51:50010, for file
> > >
> > >
> >
> /hbase/data/default/run_automaton_cache/da03f8123004be32659e1c8a51afbbf8/cf/1daf234e740f4b00889bb60e574dc79b,
> > > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > > 1294614349_254896345
> > > 2014-08-22 14:25:46,831 WARN
> > > [defaultRpcServer.handler=19,queue=1,port=60020]
> > hdfs.BlockReaderFactory:
> > > I/O error constructing remote block reader.
> > > 2014-08-22 14:25:46,832 WARN
> > > [defaultRpcServer.handler=19,queue=1,port=60020] hdfs.DFSClient:
> > > Connection failure: Failed to connect to /192.168.3.30:50010 for file
> > > /hbase/data/default/image/1c3bab43e260ddb46a06cf04e293
> > > e386/cf/740f8240ac9a4abc9e5fcf6ec7df18bc for block
> > >
> > >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1294151564_254380545:java.io.IOException:
> > > Got error for OP_READ_BLOCK, self=/192.168.3.179:55660, remote=/
> > > 192.168.3.30:500 10, for file
> > >
> > >
> >
> /hbase/data/default/image/1c3bab43e260ddb46a06cf04e293e386/cf/740f8240ac9a4abc9e5fcf6ec7df18bc,
> > > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > > 1294151564_254380545
> > > 2014-08-22 14:28:22,395 WARN
> > > [defaultRpcServer.handler=26,queue=2,port=60020]
> > hdfs.BlockReaderFactory:
> > > I/O error constructing remote block reader.
> > > 2014-08-22 14:28:22,397 WARN
> > > [defaultRpcServer.handler=26,queue=2,port=60020] hdfs.DFSClient:
> > > Connection failure: Failed to connect to /192.168.3.143:50010 for file
> > > /hbase/data/default/image/b4b48e20a606c431e393b674f92
> > > 79daf/cf/76ff3628306a48f187a285b2a21d9ac9 for block
> > >
> > >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1294308080_254554506:java.io.IOException:
> > > Got error for OP_READ_BLOCK, self=/192.168.3.179:51759, remote=/
> > > 192.168.3.143:5 0010, for file
> > >
> > >
> >
> /hbase/data/default/image/b4b48e20a606c431e393b674f9279daf/cf/76ff3628306a48f187a285b2a21d9ac9,
> > > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > > 1294308080_254554506
> > > 2014-08-22 14:30:29,395 WARN
> > > [defaultRpcServer.handler=21,queue=0,port=60020]
> > hdfs.BlockReaderFactory:
> > > I/O error constructing remote block reader.
> > > 2014-08-22 14:30:29,396 WARN
> > > [defaultRpcServer.handler=21,queue=0,port=60020] hdfs.DFSClient:
> > > Connection failure: Failed to connect to /192.168.3.30:50010 for file
> > > /hbase/data/default/image/77ceb80d73c065f1a4ba2ad3b7cf
> > > 04a2/cf/151c244177ac42b5996af0c9052660cc for block
> > >
> > >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1294365296_254618150:java.io.IOException:
> > > Got error for OP_READ_BLOCK, self=/192.168.3.179:57238, remote=/
> > > 192.168.3.30:500 10, for file
> > >
> > >
> >
> /hbase/data/default/image/77ceb80d73c065f1a4ba2ad3b7cf04a2/cf/151c244177ac42b5996af0c9052660cc,
> > > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > > 1294365296_254618150
> > > 2014-08-22 14:31:34,016 WARN
> > > [defaultRpcServer.handler=6,queue=0,port=60020]
> hdfs.BlockReaderFactory:
> > > I/O error constructing remote block reader.
> > > 2014-08-22 14:31:34,018 WARN
> > > [defaultRpcServer.handler=6,queue=0,port=60020] hdfs.DFSClient:
> > Connection
> > > failure: Failed to connect to /192.168.3.212:50010 for file
> > > /hbase/data/default/image/45578853a5c807919578043c7715
> > > 1efa/cf/40c00d49a7494929a319467d82b383da for block
> > >
> > >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1294930505_255244337:java.io.IOException:
> > > Got error for OP_READ_BLOCK, self=/192.168.3.179:33973, remote=/
> > > 192.168.3.212:50 010, for file
> > >
> > >
> >
> /hbase/data/default/image/45578853a5c807919578043c77151efa/cf/40c00d49a7494929a319467d82b383da,
> > > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > > 1294930505_255244337
> > > 2014-08-22 14:33:55,994 WARN
> > > [defaultRpcServer.handler=3,queue=0,port=60020]
> hdfs.BlockReaderFactory:
> > > I/O error constructing remote block reader.
> > > 2014-08-22 14:33:55,995 WARN
> > > [defaultRpcServer.handler=3,queue=0,port=60020] hdfs.DFSClient:
> > Connection
> > > failure: Failed to connect to /192.168.3.52:50010 for file
> > > /hbase/data/default/image/1f07e0160fbba29a2c26104dd5966
> > > 39f/cf/7e36136497fc4a5a9889797a1dfb5d3b for block
> > >
> > >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1294915607_255226575:java.io.IOException:
> > > Got error for OP_READ_BLOCK, self=/192.168.3.179:41017, remote=/
> > > 192.168.3.52:5001 0, for file
> > >
> > >
> >
> /hbase/data/default/image/1f07e0160fbba29a2c26104dd596639f/cf/7e36136497fc4a5a9889797a1dfb5d3b,
> > > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > > 1294915607_255226575
> > > 2014-08-22 14:44:14,301 WARN
> > > [defaultRpcServer.handler=3,queue=0,port=60020]
> hdfs.BlockReaderFactory:
> > > I/O error constructing remote block reader.
> > > 2014-08-22 14:44:14,302 WARN
> > > [defaultRpcServer.handler=3,queue=0,port=60020] hdfs.DFSClient:
> > Connection
> > > failure: Failed to connect to /192.168.3.154:50010 for file
> > > /hbase/data/default/image/bb082a5f098d1d2a95365545349c
> > > 77a4/cf/1af06c6fd36045bab888b4ec18e45f0f for block
> > >
> > >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1294911152_255221397:java.io.IOException:
> > > Got error for OP_READ_BLOCK, self=/192.168.3.179:44711, remote=/
> > > 192.168.3.154:50 010, for file
> > >
> > >
> >
> /hbase/data/default/image/bb082a5f098d1d2a95365545349c77a4/cf/1af06c6fd36045bab888b4ec18e45f0f,
> > > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > > 1294911152_255221397
> > > 77a4/cf/1af06c6fd36045bab888b4ec18e45f0f for block
> > >
> > >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1294911152_255221397:java.io.IOException:
> > > Got error for OP_READ_BLOCK, self=/192.168.3.179:44711, remote=/
> > > 192.168.3.154:50 010, for file
> > >
> > >
> >
> /hbase/data/default/image/bb082a5f098d1d2a95365545349c77a4/cf/1af06c6fd36045bab888b4ec18e45f0f,
> > > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > > 1294911152_255221397
> > > 2014-08-22 14:49:14,780 WARN
> > > [defaultRpcServer.handler=5,queue=2,port=60020] ipc.RpcServer:
> > > (responseTooSlow):
> > >
> > >
> >
> {"processingtimems":15494,"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiReques
> > > t)","client":"192.168.3.11:53884
> > >
> > >
> >
> ","starttimems":1408711739283,"queuetimems":0,"class":"HRegionServer","responsesize":483492,"method":"Multi"}
> > > 2014-08-22 14:50:37,900 WARN
> > > [defaultRpcServer.handler=7,queue=1,port=60020]
> hdfs.BlockReaderFactory:
> > > I/O error constructing remote block reader.
> > > 2014-08-22 14:50:37,901 WARN
> > > [defaultRpcServer.handler=7,queue=1,port=60020] hdfs.DFSClient:
> > Connection
> > > failure: Failed to connect to /192.168.3.64:50010 for file
> > > /hbase/data/default/image/a50d67b81c44c864265f5030c7c39
> > > 959/cf/840ff51e591946c487413273d5341a24 for block
> > >
> > >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1294309355_254555905:java.io.IOException:
> > > Got error for OP_READ_BLOCK, self=/192.168.3.179:45043, remote=/
> > > 192.168.3.64:5001 0, for file
> > >
> > >
> >
> /hbase/data/default/image/a50d67b81c44c864265f5030c7c39959/cf/840ff51e591946c487413273d5341a24,
> > > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > > 1294309355_254555905
> > > 2014-08-22 15:09:35,289 WARN
> > > [defaultRpcServer.handler=23,queue=2,port=60020]
> > hdfs.BlockReaderFactory:
> > > I/O error constructing remote block reader.
> > > 2014-08-22 15:09:35,289 WARN
> > > [defaultRpcServer.handler=23,queue=2,port=60020] hdfs.DFSClient:
> > > Connection failure: Failed to connect to /192.168.3.63:50010 for file
> > > /hbase/data/default/image/1db3051c8d41943892c0230cb75b
> > > c1f2/cf/75ae96ea8ca2477e8f83291d7e1fe7cb for block
> > >
> > >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1293205473_253317737:java.io.IOException:
> > > Got error for OP_READ_BLOCK, self=/192.168.3.179:56326, remote=/
> > > 192.168.3.63:500 10, for file
> > >
> > >
> >
> /hbase/data/default/image/1db3051c8d41943892c0230cb75bc1f2/cf/75ae96ea8ca2477e8f83291d7e1fe7cb,
> > > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > > 1293205473_253317737
> > > 2014-08-22 15:18:38,891 WARN
> > > [defaultRpcServer.handler=5,queue=2,port=60020] ipc.RpcServer:
> > > (responseTooSlow):
> > >
> > >
> >
> {"processingtimems":26092,"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiReques
> > > t)","client":"192.168.3.91:40189
> > >
> > >
> >
> ","starttimems":1408713492722,"queuetimems":0,"class":"HRegionServer","responsesize":483492,"method":"Multi"}
> > >
> > > What's interesting is that not all crashed handlers are later missing
> > after
> > > the freeze.
> > >
> > > Does that substantiate your conjecture?
> > >
> > >
> > I saw those in your log. They are WARN from DFSClient which should be
> ok.
> > Any other exceptions going on in your logs that could explain our
> handler
> > loss? Or, can you see any thing particular about say the last mention of
> > defaultRpcServer.handler=6 say?
> >
> > St.Ack
> >
>
Re: single RegionServer stuck, causing cluster to hang
Posted by Qiang Tian <ti...@gmail.com>.
Did you set hbase.ipc.server.callqueue.handler.factor?
it looks there are 3 queues, handlers on queue 1 are all gone as Stack
mentioned. jstack and pastebin regions server log would help.
On Sat, Aug 23, 2014 at 7:02 AM, Stack <st...@duboce.net> wrote:
> On Fri, Aug 22, 2014 at 3:24 PM, Johannes Schaback <
> johannes.schaback@visual-meta.com> wrote:
>
> > ...
> > I grep'ed "defaultRpcServer.handler=" on the log from that particular RS.
> > The
> > RS started at 15:35. After that, the handlers
> >
> > 6, 24, 0, 15, 28, 26, 7, 19, 21, 3, 5 and 23
> >
> > make an appearance in error messages of HDFS related exceptions:
> >
> > 2014-08-22 13:54:57,470 WARN
> > [defaultRpcServer.handler=6,queue=0,port=60020] hdfs.BlockReaderFactory:
> > I/O error constructing remote block reader.
> > 2014-08-22 13:54:57,472 WARN
> > [defaultRpcServer.handler=6,queue=0,port=60020] hdfs.DFSClient:
> Connection
> > failure: Failed to connect to /192.168.3.233:50010 for file
> > /hbase/data/default/image/dae1b8e3bfdb608571d09916bf0f
> > a156/cf/866a773857654b0d83275dc4e4558be6 for block
> >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1294949058_255264636:java.io.IOException:
> > Got error for OP_READ_BLOCK, self=/192.168.3.179:37835, remote=/
> > 192.168.3.233:50 010, for file
> >
> >
> /hbase/data/default/image/dae1b8e3bfdb608571d09916bf0fa156/cf/866a773857654b0d83275dc4e4558be6,
> > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > 1294949058_255264636
> > 2014-08-22 13:56:58,525 WARN
> > [defaultRpcServer.handler=24,queue=0,port=60020]
> hdfs.BlockReaderFactory:
> > I/O error constructing remote block reader.
> > 2014-08-22 13:56:58,799 WARN
> > [defaultRpcServer.handler=24,queue=0,port=60020] hdfs.DFSClient:
> > Connection failure: Failed to connect to /192.168.3.142:50010 for file
> > /hbase/data/default/image/cf493cbd9921ae6ca5e5281cc07
> > 18ca2/cf/7f9618dcdeae40ddbff21165d08e0a83 for block
> >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1294072754_254292863:java.io.IOException:
> > Got error for OP_READ_BLOCK, self=/192.168.3.179:59268, remote=/
> > 192.168.3.142:5 0010, for file
> >
> >
> /hbase/data/default/image/cf493cbd9921ae6ca5e5281cc0718ca2/cf/7f9618dcdeae40ddbff21165d08e0a83,
> > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > 1294072754_254292863
> > 2014-08-22 14:08:01,632 WARN
> > [defaultRpcServer.handler=0,queue=0,port=60020] hdfs.BlockReaderFactory:
> > I/O error constructing remote block reader.
> > 2014-08-22 14:08:01,633 WARN
> > [defaultRpcServer.handler=0,queue=0,port=60020] hdfs.DFSClient:
> Connection
> > failure: Failed to connect to /192.168.3.53:50010 for file
> > /hbase/data/default/image/af02bc7fb404f4c054dcd64b44b0e
> > 2a9/cf/a8881b6e2d5d41b3b56fd34fd4ca8ffd for block
> >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1294876123_255182439:java.io.IOException:
> > Got error for OP_READ_BLOCK, self=/192.168.3.179:60737, remote=/
> > 192.168.3.53:5001 0, for file
> >
> >
> /hbase/data/default/image/af02bc7fb404f4c054dcd64b44b0e2a9/cf/a8881b6e2d5d41b3b56fd34fd4ca8ffd,
> > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > 1294876123_255182439
> > 2014-08-22 14:11:34,192 WARN
> > [defaultRpcServer.handler=15,queue=0,port=60020] ipc.RpcServer:
> > (responseTooSlow):
> >
> >
> {"processingtimems":15432,"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiReque
> > st)","client":"192.168.3.54:52838
> >
> >
> ","starttimems":1408709478713,"queuetimems":0,"class":"HRegionServer","responsesize":99638,"method":"Multi"}
> > 2014-08-22 14:22:21,847 WARN
> > [defaultRpcServer.handler=28,queue=1,port=60020]
> hdfs.BlockReaderFactory:
> > I/O error constructing remote block reader.
> > 2014-08-22 14:22:21,848 WARN
> > [defaultRpcServer.handler=28,queue=1,port=60020] hdfs.DFSClient:
> > Connection failure: Failed to connect to /192.168.3.233:50010 for file
> > /hbase/data/default/image/1d33a251862055cd999078bbd10
> > aa44c/cf/a7470351725f4cc192b6210bac9b7c44 for block
> >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1294949316_255264902:java.io.IOException:
> > Got error for OP_READ_BLOCK, self=/192.168.3.179:44787, remote=/
> > 192.168.3.233:5 0010, for file
> >
> >
> /hbase/data/default/image/1d33a251862055cd999078bbd10aa44c/cf/a7470351725f4cc192b6210bac9b7c44,
> > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > 1294949316_255264902
> > 2014-08-22 14:23:22,628 WARN
> > [defaultRpcServer.handler=26,queue=2,port=60020]
> hdfs.BlockReaderFactory:
> > I/O error constructing remote block reader.
> > 2014-08-22 14:23:22,628 WARN
> > [defaultRpcServer.handler=26,queue=2,port=60020] hdfs.DFSClient:
> > Connection failure: Failed to connect to /192.168.3.142:50010 for file
> > /hbase/data/default/image/4a9c830a4fe006f0a6af7418164
> > dd86d/cf/47d6f2c3f7054d40aed66cae9787c464 for block
> >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1293459594_253612452:java.io.IOException:
> > Got error for OP_READ_BLOCK, self=/192.168.3.179:37660, remote=/
> > 192.168.3.142:5 0010, for file
> >
> >
> /hbase/data/default/image/4a9c830a4fe006f0a6af7418164dd86d/cf/47d6f2c3f7054d40aed66cae9787c464,
> > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > 1293459594_253612452
> > 2014-08-22 14:25:35,003 WARN
> > [defaultRpcServer.handler=7,queue=1,port=60020] hdfs.BlockReaderFactory:
> > I/O error constructing remote block reader.
> > 2014-08-22 14:25:35,004 WARN
> > [defaultRpcServer.handler=7,queue=1,port=60020] hdfs.DFSClient:
> Connection
> > failure: Failed to connect to /192.168.3.51:50010 for file
> > /hbase/data/default/run_automaton_cache/da03f8123004be3
> > 2659e1c8a51afbbf8/cf/1daf234e740f4b00889bb60e574dc79b for block
> >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1294614349_254896345:java.io.IOException:
> > Got error for OP_READ_BLOCK, self=/192.168.3.179:53627, remote=/192
> > .168.3.51:50010, for file
> >
> >
> /hbase/data/default/run_automaton_cache/da03f8123004be32659e1c8a51afbbf8/cf/1daf234e740f4b00889bb60e574dc79b,
> > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > 1294614349_254896345
> > 2014-08-22 14:25:46,831 WARN
> > [defaultRpcServer.handler=19,queue=1,port=60020]
> hdfs.BlockReaderFactory:
> > I/O error constructing remote block reader.
> > 2014-08-22 14:25:46,832 WARN
> > [defaultRpcServer.handler=19,queue=1,port=60020] hdfs.DFSClient:
> > Connection failure: Failed to connect to /192.168.3.30:50010 for file
> > /hbase/data/default/image/1c3bab43e260ddb46a06cf04e293
> > e386/cf/740f8240ac9a4abc9e5fcf6ec7df18bc for block
> >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1294151564_254380545:java.io.IOException:
> > Got error for OP_READ_BLOCK, self=/192.168.3.179:55660, remote=/
> > 192.168.3.30:500 10, for file
> >
> >
> /hbase/data/default/image/1c3bab43e260ddb46a06cf04e293e386/cf/740f8240ac9a4abc9e5fcf6ec7df18bc,
> > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > 1294151564_254380545
> > 2014-08-22 14:28:22,395 WARN
> > [defaultRpcServer.handler=26,queue=2,port=60020]
> hdfs.BlockReaderFactory:
> > I/O error constructing remote block reader.
> > 2014-08-22 14:28:22,397 WARN
> > [defaultRpcServer.handler=26,queue=2,port=60020] hdfs.DFSClient:
> > Connection failure: Failed to connect to /192.168.3.143:50010 for file
> > /hbase/data/default/image/b4b48e20a606c431e393b674f92
> > 79daf/cf/76ff3628306a48f187a285b2a21d9ac9 for block
> >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1294308080_254554506:java.io.IOException:
> > Got error for OP_READ_BLOCK, self=/192.168.3.179:51759, remote=/
> > 192.168.3.143:5 0010, for file
> >
> >
> /hbase/data/default/image/b4b48e20a606c431e393b674f9279daf/cf/76ff3628306a48f187a285b2a21d9ac9,
> > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > 1294308080_254554506
> > 2014-08-22 14:30:29,395 WARN
> > [defaultRpcServer.handler=21,queue=0,port=60020]
> hdfs.BlockReaderFactory:
> > I/O error constructing remote block reader.
> > 2014-08-22 14:30:29,396 WARN
> > [defaultRpcServer.handler=21,queue=0,port=60020] hdfs.DFSClient:
> > Connection failure: Failed to connect to /192.168.3.30:50010 for file
> > /hbase/data/default/image/77ceb80d73c065f1a4ba2ad3b7cf
> > 04a2/cf/151c244177ac42b5996af0c9052660cc for block
> >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1294365296_254618150:java.io.IOException:
> > Got error for OP_READ_BLOCK, self=/192.168.3.179:57238, remote=/
> > 192.168.3.30:500 10, for file
> >
> >
> /hbase/data/default/image/77ceb80d73c065f1a4ba2ad3b7cf04a2/cf/151c244177ac42b5996af0c9052660cc,
> > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > 1294365296_254618150
> > 2014-08-22 14:31:34,016 WARN
> > [defaultRpcServer.handler=6,queue=0,port=60020] hdfs.BlockReaderFactory:
> > I/O error constructing remote block reader.
> > 2014-08-22 14:31:34,018 WARN
> > [defaultRpcServer.handler=6,queue=0,port=60020] hdfs.DFSClient:
> Connection
> > failure: Failed to connect to /192.168.3.212:50010 for file
> > /hbase/data/default/image/45578853a5c807919578043c7715
> > 1efa/cf/40c00d49a7494929a319467d82b383da for block
> >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1294930505_255244337:java.io.IOException:
> > Got error for OP_READ_BLOCK, self=/192.168.3.179:33973, remote=/
> > 192.168.3.212:50 010, for file
> >
> >
> /hbase/data/default/image/45578853a5c807919578043c77151efa/cf/40c00d49a7494929a319467d82b383da,
> > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > 1294930505_255244337
> > 2014-08-22 14:33:55,994 WARN
> > [defaultRpcServer.handler=3,queue=0,port=60020] hdfs.BlockReaderFactory:
> > I/O error constructing remote block reader.
> > 2014-08-22 14:33:55,995 WARN
> > [defaultRpcServer.handler=3,queue=0,port=60020] hdfs.DFSClient:
> Connection
> > failure: Failed to connect to /192.168.3.52:50010 for file
> > /hbase/data/default/image/1f07e0160fbba29a2c26104dd5966
> > 39f/cf/7e36136497fc4a5a9889797a1dfb5d3b for block
> >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1294915607_255226575:java.io.IOException:
> > Got error for OP_READ_BLOCK, self=/192.168.3.179:41017, remote=/
> > 192.168.3.52:5001 0, for file
> >
> >
> /hbase/data/default/image/1f07e0160fbba29a2c26104dd596639f/cf/7e36136497fc4a5a9889797a1dfb5d3b,
> > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > 1294915607_255226575
> > 2014-08-22 14:44:14,301 WARN
> > [defaultRpcServer.handler=3,queue=0,port=60020] hdfs.BlockReaderFactory:
> > I/O error constructing remote block reader.
> > 2014-08-22 14:44:14,302 WARN
> > [defaultRpcServer.handler=3,queue=0,port=60020] hdfs.DFSClient:
> Connection
> > failure: Failed to connect to /192.168.3.154:50010 for file
> > /hbase/data/default/image/bb082a5f098d1d2a95365545349c
> > 77a4/cf/1af06c6fd36045bab888b4ec18e45f0f for block
> >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1294911152_255221397:java.io.IOException:
> > Got error for OP_READ_BLOCK, self=/192.168.3.179:44711, remote=/
> > 192.168.3.154:50 010, for file
> >
> >
> /hbase/data/default/image/bb082a5f098d1d2a95365545349c77a4/cf/1af06c6fd36045bab888b4ec18e45f0f,
> > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > 1294911152_255221397
> > 77a4/cf/1af06c6fd36045bab888b4ec18e45f0f for block
> >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1294911152_255221397:java.io.IOException:
> > Got error for OP_READ_BLOCK, self=/192.168.3.179:44711, remote=/
> > 192.168.3.154:50 010, for file
> >
> >
> /hbase/data/default/image/bb082a5f098d1d2a95365545349c77a4/cf/1af06c6fd36045bab888b4ec18e45f0f,
> > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > 1294911152_255221397
> > 2014-08-22 14:49:14,780 WARN
> > [defaultRpcServer.handler=5,queue=2,port=60020] ipc.RpcServer:
> > (responseTooSlow):
> >
> >
> {"processingtimems":15494,"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiReques
> > t)","client":"192.168.3.11:53884
> >
> >
> ","starttimems":1408711739283,"queuetimems":0,"class":"HRegionServer","responsesize":483492,"method":"Multi"}
> > 2014-08-22 14:50:37,900 WARN
> > [defaultRpcServer.handler=7,queue=1,port=60020] hdfs.BlockReaderFactory:
> > I/O error constructing remote block reader.
> > 2014-08-22 14:50:37,901 WARN
> > [defaultRpcServer.handler=7,queue=1,port=60020] hdfs.DFSClient:
> Connection
> > failure: Failed to connect to /192.168.3.64:50010 for file
> > /hbase/data/default/image/a50d67b81c44c864265f5030c7c39
> > 959/cf/840ff51e591946c487413273d5341a24 for block
> >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1294309355_254555905:java.io.IOException:
> > Got error for OP_READ_BLOCK, self=/192.168.3.179:45043, remote=/
> > 192.168.3.64:5001 0, for file
> >
> >
> /hbase/data/default/image/a50d67b81c44c864265f5030c7c39959/cf/840ff51e591946c487413273d5341a24,
> > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > 1294309355_254555905
> > 2014-08-22 15:09:35,289 WARN
> > [defaultRpcServer.handler=23,queue=2,port=60020]
> hdfs.BlockReaderFactory:
> > I/O error constructing remote block reader.
> > 2014-08-22 15:09:35,289 WARN
> > [defaultRpcServer.handler=23,queue=2,port=60020] hdfs.DFSClient:
> > Connection failure: Failed to connect to /192.168.3.63:50010 for file
> > /hbase/data/default/image/1db3051c8d41943892c0230cb75b
> > c1f2/cf/75ae96ea8ca2477e8f83291d7e1fe7cb for block
> >
> >
> BP-1157637685-192.168.3.192-1382642140917:blk_1293205473_253317737:java.io.IOException:
> > Got error for OP_READ_BLOCK, self=/192.168.3.179:56326, remote=/
> > 192.168.3.63:500 10, for file
> >
> >
> /hbase/data/default/image/1db3051c8d41943892c0230cb75bc1f2/cf/75ae96ea8ca2477e8f83291d7e1fe7cb,
> > for pool BP-1157637685-192.168.3.192-1382642140917 block
> > 1293205473_253317737
> > 2014-08-22 15:18:38,891 WARN
> > [defaultRpcServer.handler=5,queue=2,port=60020] ipc.RpcServer:
> > (responseTooSlow):
> >
> >
> {"processingtimems":26092,"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiReques
> > t)","client":"192.168.3.91:40189
> >
> >
> ","starttimems":1408713492722,"queuetimems":0,"class":"HRegionServer","responsesize":483492,"method":"Multi"}
> >
> > What's interesting is that not all crashed handlers are later missing
> after
> > the freeze.
> >
> > Does that substantiate your conjecture?
> >
> >
> I saw those in your log. They are WARN from DFSClient which should be ok.
> Any other exceptions going on in your logs that could explain our handler
> loss? Or, can you see any thing particular about say the last mention of
> defaultRpcServer.handler=6 say?
>
> St.Ack
>
Re: single RegionServer stuck, causing cluster to hang
Posted by Stack <st...@duboce.net>.
On Fri, Aug 22, 2014 at 3:24 PM, Johannes Schaback <
johannes.schaback@visual-meta.com> wrote:
> ...
> I grep'ed "defaultRpcServer.handler=" on the log from that particular RS.
> The
> RS started at 15:35. After that, the handlers
>
> 6, 24, 0, 15, 28, 26, 7, 19, 21, 3, 5 and 23
>
> make an appearance in error messages of HDFS related exceptions:
>
> 2014-08-22 13:54:57,470 WARN
> [defaultRpcServer.handler=6,queue=0,port=60020] hdfs.BlockReaderFactory:
> I/O error constructing remote block reader.
> 2014-08-22 13:54:57,472 WARN
> [defaultRpcServer.handler=6,queue=0,port=60020] hdfs.DFSClient: Connection
> failure: Failed to connect to /192.168.3.233:50010 for file
> /hbase/data/default/image/dae1b8e3bfdb608571d09916bf0f
> a156/cf/866a773857654b0d83275dc4e4558be6 for block
>
> BP-1157637685-192.168.3.192-1382642140917:blk_1294949058_255264636:java.io.IOException:
> Got error for OP_READ_BLOCK, self=/192.168.3.179:37835, remote=/
> 192.168.3.233:50 010, for file
>
> /hbase/data/default/image/dae1b8e3bfdb608571d09916bf0fa156/cf/866a773857654b0d83275dc4e4558be6,
> for pool BP-1157637685-192.168.3.192-1382642140917 block
> 1294949058_255264636
> 2014-08-22 13:56:58,525 WARN
> [defaultRpcServer.handler=24,queue=0,port=60020] hdfs.BlockReaderFactory:
> I/O error constructing remote block reader.
> 2014-08-22 13:56:58,799 WARN
> [defaultRpcServer.handler=24,queue=0,port=60020] hdfs.DFSClient:
> Connection failure: Failed to connect to /192.168.3.142:50010 for file
> /hbase/data/default/image/cf493cbd9921ae6ca5e5281cc07
> 18ca2/cf/7f9618dcdeae40ddbff21165d08e0a83 for block
>
> BP-1157637685-192.168.3.192-1382642140917:blk_1294072754_254292863:java.io.IOException:
> Got error for OP_READ_BLOCK, self=/192.168.3.179:59268, remote=/
> 192.168.3.142:5 0010, for file
>
> /hbase/data/default/image/cf493cbd9921ae6ca5e5281cc0718ca2/cf/7f9618dcdeae40ddbff21165d08e0a83,
> for pool BP-1157637685-192.168.3.192-1382642140917 block
> 1294072754_254292863
> 2014-08-22 14:08:01,632 WARN
> [defaultRpcServer.handler=0,queue=0,port=60020] hdfs.BlockReaderFactory:
> I/O error constructing remote block reader.
> 2014-08-22 14:08:01,633 WARN
> [defaultRpcServer.handler=0,queue=0,port=60020] hdfs.DFSClient: Connection
> failure: Failed to connect to /192.168.3.53:50010 for file
> /hbase/data/default/image/af02bc7fb404f4c054dcd64b44b0e
> 2a9/cf/a8881b6e2d5d41b3b56fd34fd4ca8ffd for block
>
> BP-1157637685-192.168.3.192-1382642140917:blk_1294876123_255182439:java.io.IOException:
> Got error for OP_READ_BLOCK, self=/192.168.3.179:60737, remote=/
> 192.168.3.53:5001 0, for file
>
> /hbase/data/default/image/af02bc7fb404f4c054dcd64b44b0e2a9/cf/a8881b6e2d5d41b3b56fd34fd4ca8ffd,
> for pool BP-1157637685-192.168.3.192-1382642140917 block
> 1294876123_255182439
> 2014-08-22 14:11:34,192 WARN
> [defaultRpcServer.handler=15,queue=0,port=60020] ipc.RpcServer:
> (responseTooSlow):
>
> {"processingtimems":15432,"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiReque
> st)","client":"192.168.3.54:52838
>
> ","starttimems":1408709478713,"queuetimems":0,"class":"HRegionServer","responsesize":99638,"method":"Multi"}
> 2014-08-22 14:22:21,847 WARN
> [defaultRpcServer.handler=28,queue=1,port=60020] hdfs.BlockReaderFactory:
> I/O error constructing remote block reader.
> 2014-08-22 14:22:21,848 WARN
> [defaultRpcServer.handler=28,queue=1,port=60020] hdfs.DFSClient:
> Connection failure: Failed to connect to /192.168.3.233:50010 for file
> /hbase/data/default/image/1d33a251862055cd999078bbd10
> aa44c/cf/a7470351725f4cc192b6210bac9b7c44 for block
>
> BP-1157637685-192.168.3.192-1382642140917:blk_1294949316_255264902:java.io.IOException:
> Got error for OP_READ_BLOCK, self=/192.168.3.179:44787, remote=/
> 192.168.3.233:5 0010, for file
>
> /hbase/data/default/image/1d33a251862055cd999078bbd10aa44c/cf/a7470351725f4cc192b6210bac9b7c44,
> for pool BP-1157637685-192.168.3.192-1382642140917 block
> 1294949316_255264902
> 2014-08-22 14:23:22,628 WARN
> [defaultRpcServer.handler=26,queue=2,port=60020] hdfs.BlockReaderFactory:
> I/O error constructing remote block reader.
> 2014-08-22 14:23:22,628 WARN
> [defaultRpcServer.handler=26,queue=2,port=60020] hdfs.DFSClient:
> Connection failure: Failed to connect to /192.168.3.142:50010 for file
> /hbase/data/default/image/4a9c830a4fe006f0a6af7418164
> dd86d/cf/47d6f2c3f7054d40aed66cae9787c464 for block
>
> BP-1157637685-192.168.3.192-1382642140917:blk_1293459594_253612452:java.io.IOException:
> Got error for OP_READ_BLOCK, self=/192.168.3.179:37660, remote=/
> 192.168.3.142:5 0010, for file
>
> /hbase/data/default/image/4a9c830a4fe006f0a6af7418164dd86d/cf/47d6f2c3f7054d40aed66cae9787c464,
> for pool BP-1157637685-192.168.3.192-1382642140917 block
> 1293459594_253612452
> 2014-08-22 14:25:35,003 WARN
> [defaultRpcServer.handler=7,queue=1,port=60020] hdfs.BlockReaderFactory:
> I/O error constructing remote block reader.
> 2014-08-22 14:25:35,004 WARN
> [defaultRpcServer.handler=7,queue=1,port=60020] hdfs.DFSClient: Connection
> failure: Failed to connect to /192.168.3.51:50010 for file
> /hbase/data/default/run_automaton_cache/da03f8123004be3
> 2659e1c8a51afbbf8/cf/1daf234e740f4b00889bb60e574dc79b for block
>
> BP-1157637685-192.168.3.192-1382642140917:blk_1294614349_254896345:java.io.IOException:
> Got error for OP_READ_BLOCK, self=/192.168.3.179:53627, remote=/192
> .168.3.51:50010, for file
>
> /hbase/data/default/run_automaton_cache/da03f8123004be32659e1c8a51afbbf8/cf/1daf234e740f4b00889bb60e574dc79b,
> for pool BP-1157637685-192.168.3.192-1382642140917 block
> 1294614349_254896345
> 2014-08-22 14:25:46,831 WARN
> [defaultRpcServer.handler=19,queue=1,port=60020] hdfs.BlockReaderFactory:
> I/O error constructing remote block reader.
> 2014-08-22 14:25:46,832 WARN
> [defaultRpcServer.handler=19,queue=1,port=60020] hdfs.DFSClient:
> Connection failure: Failed to connect to /192.168.3.30:50010 for file
> /hbase/data/default/image/1c3bab43e260ddb46a06cf04e293
> e386/cf/740f8240ac9a4abc9e5fcf6ec7df18bc for block
>
> BP-1157637685-192.168.3.192-1382642140917:blk_1294151564_254380545:java.io.IOException:
> Got error for OP_READ_BLOCK, self=/192.168.3.179:55660, remote=/
> 192.168.3.30:500 10, for file
>
> /hbase/data/default/image/1c3bab43e260ddb46a06cf04e293e386/cf/740f8240ac9a4abc9e5fcf6ec7df18bc,
> for pool BP-1157637685-192.168.3.192-1382642140917 block
> 1294151564_254380545
> 2014-08-22 14:28:22,395 WARN
> [defaultRpcServer.handler=26,queue=2,port=60020] hdfs.BlockReaderFactory:
> I/O error constructing remote block reader.
> 2014-08-22 14:28:22,397 WARN
> [defaultRpcServer.handler=26,queue=2,port=60020] hdfs.DFSClient:
> Connection failure: Failed to connect to /192.168.3.143:50010 for file
> /hbase/data/default/image/b4b48e20a606c431e393b674f92
> 79daf/cf/76ff3628306a48f187a285b2a21d9ac9 for block
>
> BP-1157637685-192.168.3.192-1382642140917:blk_1294308080_254554506:java.io.IOException:
> Got error for OP_READ_BLOCK, self=/192.168.3.179:51759, remote=/
> 192.168.3.143:5 0010, for file
>
> /hbase/data/default/image/b4b48e20a606c431e393b674f9279daf/cf/76ff3628306a48f187a285b2a21d9ac9,
> for pool BP-1157637685-192.168.3.192-1382642140917 block
> 1294308080_254554506
> 2014-08-22 14:30:29,395 WARN
> [defaultRpcServer.handler=21,queue=0,port=60020] hdfs.BlockReaderFactory:
> I/O error constructing remote block reader.
> 2014-08-22 14:30:29,396 WARN
> [defaultRpcServer.handler=21,queue=0,port=60020] hdfs.DFSClient:
> Connection failure: Failed to connect to /192.168.3.30:50010 for file
> /hbase/data/default/image/77ceb80d73c065f1a4ba2ad3b7cf
> 04a2/cf/151c244177ac42b5996af0c9052660cc for block
>
> BP-1157637685-192.168.3.192-1382642140917:blk_1294365296_254618150:java.io.IOException:
> Got error for OP_READ_BLOCK, self=/192.168.3.179:57238, remote=/
> 192.168.3.30:500 10, for file
>
> /hbase/data/default/image/77ceb80d73c065f1a4ba2ad3b7cf04a2/cf/151c244177ac42b5996af0c9052660cc,
> for pool BP-1157637685-192.168.3.192-1382642140917 block
> 1294365296_254618150
> 2014-08-22 14:31:34,016 WARN
> [defaultRpcServer.handler=6,queue=0,port=60020] hdfs.BlockReaderFactory:
> I/O error constructing remote block reader.
> 2014-08-22 14:31:34,018 WARN
> [defaultRpcServer.handler=6,queue=0,port=60020] hdfs.DFSClient: Connection
> failure: Failed to connect to /192.168.3.212:50010 for file
> /hbase/data/default/image/45578853a5c807919578043c7715
> 1efa/cf/40c00d49a7494929a319467d82b383da for block
>
> BP-1157637685-192.168.3.192-1382642140917:blk_1294930505_255244337:java.io.IOException:
> Got error for OP_READ_BLOCK, self=/192.168.3.179:33973, remote=/
> 192.168.3.212:50 010, for file
>
> /hbase/data/default/image/45578853a5c807919578043c77151efa/cf/40c00d49a7494929a319467d82b383da,
> for pool BP-1157637685-192.168.3.192-1382642140917 block
> 1294930505_255244337
> 2014-08-22 14:33:55,994 WARN
> [defaultRpcServer.handler=3,queue=0,port=60020] hdfs.BlockReaderFactory:
> I/O error constructing remote block reader.
> 2014-08-22 14:33:55,995 WARN
> [defaultRpcServer.handler=3,queue=0,port=60020] hdfs.DFSClient: Connection
> failure: Failed to connect to /192.168.3.52:50010 for file
> /hbase/data/default/image/1f07e0160fbba29a2c26104dd5966
> 39f/cf/7e36136497fc4a5a9889797a1dfb5d3b for block
>
> BP-1157637685-192.168.3.192-1382642140917:blk_1294915607_255226575:java.io.IOException:
> Got error for OP_READ_BLOCK, self=/192.168.3.179:41017, remote=/
> 192.168.3.52:5001 0, for file
>
> /hbase/data/default/image/1f07e0160fbba29a2c26104dd596639f/cf/7e36136497fc4a5a9889797a1dfb5d3b,
> for pool BP-1157637685-192.168.3.192-1382642140917 block
> 1294915607_255226575
> 2014-08-22 14:44:14,301 WARN
> [defaultRpcServer.handler=3,queue=0,port=60020] hdfs.BlockReaderFactory:
> I/O error constructing remote block reader.
> 2014-08-22 14:44:14,302 WARN
> [defaultRpcServer.handler=3,queue=0,port=60020] hdfs.DFSClient: Connection
> failure: Failed to connect to /192.168.3.154:50010 for file
> /hbase/data/default/image/bb082a5f098d1d2a95365545349c
> 77a4/cf/1af06c6fd36045bab888b4ec18e45f0f for block
>
> BP-1157637685-192.168.3.192-1382642140917:blk_1294911152_255221397:java.io.IOException:
> Got error for OP_READ_BLOCK, self=/192.168.3.179:44711, remote=/
> 192.168.3.154:50 010, for file
>
> /hbase/data/default/image/bb082a5f098d1d2a95365545349c77a4/cf/1af06c6fd36045bab888b4ec18e45f0f,
> for pool BP-1157637685-192.168.3.192-1382642140917 block
> 1294911152_255221397
> 77a4/cf/1af06c6fd36045bab888b4ec18e45f0f for block
>
> BP-1157637685-192.168.3.192-1382642140917:blk_1294911152_255221397:java.io.IOException:
> Got error for OP_READ_BLOCK, self=/192.168.3.179:44711, remote=/
> 192.168.3.154:50 010, for file
>
> /hbase/data/default/image/bb082a5f098d1d2a95365545349c77a4/cf/1af06c6fd36045bab888b4ec18e45f0f,
> for pool BP-1157637685-192.168.3.192-1382642140917 block
> 1294911152_255221397
> 2014-08-22 14:49:14,780 WARN
> [defaultRpcServer.handler=5,queue=2,port=60020] ipc.RpcServer:
> (responseTooSlow):
>
> {"processingtimems":15494,"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiReques
> t)","client":"192.168.3.11:53884
>
> ","starttimems":1408711739283,"queuetimems":0,"class":"HRegionServer","responsesize":483492,"method":"Multi"}
> 2014-08-22 14:50:37,900 WARN
> [defaultRpcServer.handler=7,queue=1,port=60020] hdfs.BlockReaderFactory:
> I/O error constructing remote block reader.
> 2014-08-22 14:50:37,901 WARN
> [defaultRpcServer.handler=7,queue=1,port=60020] hdfs.DFSClient: Connection
> failure: Failed to connect to /192.168.3.64:50010 for file
> /hbase/data/default/image/a50d67b81c44c864265f5030c7c39
> 959/cf/840ff51e591946c487413273d5341a24 for block
>
> BP-1157637685-192.168.3.192-1382642140917:blk_1294309355_254555905:java.io.IOException:
> Got error for OP_READ_BLOCK, self=/192.168.3.179:45043, remote=/
> 192.168.3.64:5001 0, for file
>
> /hbase/data/default/image/a50d67b81c44c864265f5030c7c39959/cf/840ff51e591946c487413273d5341a24,
> for pool BP-1157637685-192.168.3.192-1382642140917 block
> 1294309355_254555905
> 2014-08-22 15:09:35,289 WARN
> [defaultRpcServer.handler=23,queue=2,port=60020] hdfs.BlockReaderFactory:
> I/O error constructing remote block reader.
> 2014-08-22 15:09:35,289 WARN
> [defaultRpcServer.handler=23,queue=2,port=60020] hdfs.DFSClient:
> Connection failure: Failed to connect to /192.168.3.63:50010 for file
> /hbase/data/default/image/1db3051c8d41943892c0230cb75b
> c1f2/cf/75ae96ea8ca2477e8f83291d7e1fe7cb for block
>
> BP-1157637685-192.168.3.192-1382642140917:blk_1293205473_253317737:java.io.IOException:
> Got error for OP_READ_BLOCK, self=/192.168.3.179:56326, remote=/
> 192.168.3.63:500 10, for file
>
> /hbase/data/default/image/1db3051c8d41943892c0230cb75bc1f2/cf/75ae96ea8ca2477e8f83291d7e1fe7cb,
> for pool BP-1157637685-192.168.3.192-1382642140917 block
> 1293205473_253317737
> 2014-08-22 15:18:38,891 WARN
> [defaultRpcServer.handler=5,queue=2,port=60020] ipc.RpcServer:
> (responseTooSlow):
>
> {"processingtimems":26092,"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiReques
> t)","client":"192.168.3.91:40189
>
> ","starttimems":1408713492722,"queuetimems":0,"class":"HRegionServer","responsesize":483492,"method":"Multi"}
>
> What's interesting is that not all crashed handlers are later missing after
> the freeze.
>
> Does that substantiate your conjecture?
>
>
I saw those in your log. They are WARN from DFSClient which should be ok.
Any other exceptions going on in your logs that could explain our handler
loss? Or, can you see any thing particular about say the last mention of
defaultRpcServer.handler=6 say?
St.Ack
Re: single RegionServer stuck, causing cluster to hang
Posted by Johannes Schaback <jo...@visual-meta.com>.
I havent managed to pull a jstack of a stuck node yet (I will do that first
thing in the morning). But...
I just killed and restarted a RS and called /dump right away to see whether
the defaultRpcServer.handler instances are present. And yes, they are. From
0 to 29, even in consecutive order. I killed and restarted that RS again to
be sure. Also, I checked several healthy RS and they got the complete set
of handler threads as well shown with /dump. So I believe it is reasonable
to assume that /dump does not miss threads.
Coming back to the particular handlers of the broken RS in question; with
the surviving handlers 27, 24, 21, 20, 18, 12, 11 and 6 after the freeze,
this set of handlers
0, 1, 2, 3, 4, 5, 7, 8, 9, 10, 13, 14, 15, 16, 17, 19, 22, 23, 25, 26, 28
and 29
did apparently get lost somehow and therefore did not appear in /dump after
the freeeze.
I grep'ed "defaultRpcServer.handler=" on the log from that particular RS. The
RS started at 15:35. After that, the handlers
6, 24, 0, 15, 28, 26, 7, 19, 21, 3, 5 and 23
make an appearance in error messages of HDFS related exceptions:
2014-08-22 13:54:57,470 WARN
[defaultRpcServer.handler=6,queue=0,port=60020] hdfs.BlockReaderFactory:
I/O error constructing remote block reader.
2014-08-22 13:54:57,472 WARN
[defaultRpcServer.handler=6,queue=0,port=60020] hdfs.DFSClient: Connection
failure: Failed to connect to /192.168.3.233:50010 for file
/hbase/data/default/image/dae1b8e3bfdb608571d09916bf0f
a156/cf/866a773857654b0d83275dc4e4558be6 for block
BP-1157637685-192.168.3.192-1382642140917:blk_1294949058_255264636:java.io.IOException:
Got error for OP_READ_BLOCK, self=/192.168.3.179:37835, remote=/
192.168.3.233:50 010, for file
/hbase/data/default/image/dae1b8e3bfdb608571d09916bf0fa156/cf/866a773857654b0d83275dc4e4558be6,
for pool BP-1157637685-192.168.3.192-1382642140917 block
1294949058_255264636
2014-08-22 13:56:58,525 WARN
[defaultRpcServer.handler=24,queue=0,port=60020] hdfs.BlockReaderFactory:
I/O error constructing remote block reader.
2014-08-22 13:56:58,799 WARN
[defaultRpcServer.handler=24,queue=0,port=60020] hdfs.DFSClient:
Connection failure: Failed to connect to /192.168.3.142:50010 for file
/hbase/data/default/image/cf493cbd9921ae6ca5e5281cc07
18ca2/cf/7f9618dcdeae40ddbff21165d08e0a83 for block
BP-1157637685-192.168.3.192-1382642140917:blk_1294072754_254292863:java.io.IOException:
Got error for OP_READ_BLOCK, self=/192.168.3.179:59268, remote=/
192.168.3.142:5 0010, for file
/hbase/data/default/image/cf493cbd9921ae6ca5e5281cc0718ca2/cf/7f9618dcdeae40ddbff21165d08e0a83,
for pool BP-1157637685-192.168.3.192-1382642140917 block
1294072754_254292863
2014-08-22 14:08:01,632 WARN
[defaultRpcServer.handler=0,queue=0,port=60020] hdfs.BlockReaderFactory:
I/O error constructing remote block reader.
2014-08-22 14:08:01,633 WARN
[defaultRpcServer.handler=0,queue=0,port=60020] hdfs.DFSClient: Connection
failure: Failed to connect to /192.168.3.53:50010 for file
/hbase/data/default/image/af02bc7fb404f4c054dcd64b44b0e
2a9/cf/a8881b6e2d5d41b3b56fd34fd4ca8ffd for block
BP-1157637685-192.168.3.192-1382642140917:blk_1294876123_255182439:java.io.IOException:
Got error for OP_READ_BLOCK, self=/192.168.3.179:60737, remote=/
192.168.3.53:5001 0, for file
/hbase/data/default/image/af02bc7fb404f4c054dcd64b44b0e2a9/cf/a8881b6e2d5d41b3b56fd34fd4ca8ffd,
for pool BP-1157637685-192.168.3.192-1382642140917 block
1294876123_255182439
2014-08-22 14:11:34,192 WARN
[defaultRpcServer.handler=15,queue=0,port=60020] ipc.RpcServer:
(responseTooSlow):
{"processingtimems":15432,"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiReque
st)","client":"192.168.3.54:52838
","starttimems":1408709478713,"queuetimems":0,"class":"HRegionServer","responsesize":99638,"method":"Multi"}
2014-08-22 14:22:21,847 WARN
[defaultRpcServer.handler=28,queue=1,port=60020] hdfs.BlockReaderFactory:
I/O error constructing remote block reader.
2014-08-22 14:22:21,848 WARN
[defaultRpcServer.handler=28,queue=1,port=60020] hdfs.DFSClient:
Connection failure: Failed to connect to /192.168.3.233:50010 for file
/hbase/data/default/image/1d33a251862055cd999078bbd10
aa44c/cf/a7470351725f4cc192b6210bac9b7c44 for block
BP-1157637685-192.168.3.192-1382642140917:blk_1294949316_255264902:java.io.IOException:
Got error for OP_READ_BLOCK, self=/192.168.3.179:44787, remote=/
192.168.3.233:5 0010, for file
/hbase/data/default/image/1d33a251862055cd999078bbd10aa44c/cf/a7470351725f4cc192b6210bac9b7c44,
for pool BP-1157637685-192.168.3.192-1382642140917 block
1294949316_255264902
2014-08-22 14:23:22,628 WARN
[defaultRpcServer.handler=26,queue=2,port=60020] hdfs.BlockReaderFactory:
I/O error constructing remote block reader.
2014-08-22 14:23:22,628 WARN
[defaultRpcServer.handler=26,queue=2,port=60020] hdfs.DFSClient:
Connection failure: Failed to connect to /192.168.3.142:50010 for file
/hbase/data/default/image/4a9c830a4fe006f0a6af7418164
dd86d/cf/47d6f2c3f7054d40aed66cae9787c464 for block
BP-1157637685-192.168.3.192-1382642140917:blk_1293459594_253612452:java.io.IOException:
Got error for OP_READ_BLOCK, self=/192.168.3.179:37660, remote=/
192.168.3.142:5 0010, for file
/hbase/data/default/image/4a9c830a4fe006f0a6af7418164dd86d/cf/47d6f2c3f7054d40aed66cae9787c464,
for pool BP-1157637685-192.168.3.192-1382642140917 block
1293459594_253612452
2014-08-22 14:25:35,003 WARN
[defaultRpcServer.handler=7,queue=1,port=60020] hdfs.BlockReaderFactory:
I/O error constructing remote block reader.
2014-08-22 14:25:35,004 WARN
[defaultRpcServer.handler=7,queue=1,port=60020] hdfs.DFSClient: Connection
failure: Failed to connect to /192.168.3.51:50010 for file
/hbase/data/default/run_automaton_cache/da03f8123004be3
2659e1c8a51afbbf8/cf/1daf234e740f4b00889bb60e574dc79b for block
BP-1157637685-192.168.3.192-1382642140917:blk_1294614349_254896345:java.io.IOException:
Got error for OP_READ_BLOCK, self=/192.168.3.179:53627, remote=/192
.168.3.51:50010, for file
/hbase/data/default/run_automaton_cache/da03f8123004be32659e1c8a51afbbf8/cf/1daf234e740f4b00889bb60e574dc79b,
for pool BP-1157637685-192.168.3.192-1382642140917 block
1294614349_254896345
2014-08-22 14:25:46,831 WARN
[defaultRpcServer.handler=19,queue=1,port=60020] hdfs.BlockReaderFactory:
I/O error constructing remote block reader.
2014-08-22 14:25:46,832 WARN
[defaultRpcServer.handler=19,queue=1,port=60020] hdfs.DFSClient:
Connection failure: Failed to connect to /192.168.3.30:50010 for file
/hbase/data/default/image/1c3bab43e260ddb46a06cf04e293
e386/cf/740f8240ac9a4abc9e5fcf6ec7df18bc for block
BP-1157637685-192.168.3.192-1382642140917:blk_1294151564_254380545:java.io.IOException:
Got error for OP_READ_BLOCK, self=/192.168.3.179:55660, remote=/
192.168.3.30:500 10, for file
/hbase/data/default/image/1c3bab43e260ddb46a06cf04e293e386/cf/740f8240ac9a4abc9e5fcf6ec7df18bc,
for pool BP-1157637685-192.168.3.192-1382642140917 block
1294151564_254380545
2014-08-22 14:28:22,395 WARN
[defaultRpcServer.handler=26,queue=2,port=60020] hdfs.BlockReaderFactory:
I/O error constructing remote block reader.
2014-08-22 14:28:22,397 WARN
[defaultRpcServer.handler=26,queue=2,port=60020] hdfs.DFSClient:
Connection failure: Failed to connect to /192.168.3.143:50010 for file
/hbase/data/default/image/b4b48e20a606c431e393b674f92
79daf/cf/76ff3628306a48f187a285b2a21d9ac9 for block
BP-1157637685-192.168.3.192-1382642140917:blk_1294308080_254554506:java.io.IOException:
Got error for OP_READ_BLOCK, self=/192.168.3.179:51759, remote=/
192.168.3.143:5 0010, for file
/hbase/data/default/image/b4b48e20a606c431e393b674f9279daf/cf/76ff3628306a48f187a285b2a21d9ac9,
for pool BP-1157637685-192.168.3.192-1382642140917 block
1294308080_254554506
2014-08-22 14:30:29,395 WARN
[defaultRpcServer.handler=21,queue=0,port=60020] hdfs.BlockReaderFactory:
I/O error constructing remote block reader.
2014-08-22 14:30:29,396 WARN
[defaultRpcServer.handler=21,queue=0,port=60020] hdfs.DFSClient:
Connection failure: Failed to connect to /192.168.3.30:50010 for file
/hbase/data/default/image/77ceb80d73c065f1a4ba2ad3b7cf
04a2/cf/151c244177ac42b5996af0c9052660cc for block
BP-1157637685-192.168.3.192-1382642140917:blk_1294365296_254618150:java.io.IOException:
Got error for OP_READ_BLOCK, self=/192.168.3.179:57238, remote=/
192.168.3.30:500 10, for file
/hbase/data/default/image/77ceb80d73c065f1a4ba2ad3b7cf04a2/cf/151c244177ac42b5996af0c9052660cc,
for pool BP-1157637685-192.168.3.192-1382642140917 block
1294365296_254618150
2014-08-22 14:31:34,016 WARN
[defaultRpcServer.handler=6,queue=0,port=60020] hdfs.BlockReaderFactory:
I/O error constructing remote block reader.
2014-08-22 14:31:34,018 WARN
[defaultRpcServer.handler=6,queue=0,port=60020] hdfs.DFSClient: Connection
failure: Failed to connect to /192.168.3.212:50010 for file
/hbase/data/default/image/45578853a5c807919578043c7715
1efa/cf/40c00d49a7494929a319467d82b383da for block
BP-1157637685-192.168.3.192-1382642140917:blk_1294930505_255244337:java.io.IOException:
Got error for OP_READ_BLOCK, self=/192.168.3.179:33973, remote=/
192.168.3.212:50 010, for file
/hbase/data/default/image/45578853a5c807919578043c77151efa/cf/40c00d49a7494929a319467d82b383da,
for pool BP-1157637685-192.168.3.192-1382642140917 block
1294930505_255244337
2014-08-22 14:33:55,994 WARN
[defaultRpcServer.handler=3,queue=0,port=60020] hdfs.BlockReaderFactory:
I/O error constructing remote block reader.
2014-08-22 14:33:55,995 WARN
[defaultRpcServer.handler=3,queue=0,port=60020] hdfs.DFSClient: Connection
failure: Failed to connect to /192.168.3.52:50010 for file
/hbase/data/default/image/1f07e0160fbba29a2c26104dd5966
39f/cf/7e36136497fc4a5a9889797a1dfb5d3b for block
BP-1157637685-192.168.3.192-1382642140917:blk_1294915607_255226575:java.io.IOException:
Got error for OP_READ_BLOCK, self=/192.168.3.179:41017, remote=/
192.168.3.52:5001 0, for file
/hbase/data/default/image/1f07e0160fbba29a2c26104dd596639f/cf/7e36136497fc4a5a9889797a1dfb5d3b,
for pool BP-1157637685-192.168.3.192-1382642140917 block
1294915607_255226575
2014-08-22 14:44:14,301 WARN
[defaultRpcServer.handler=3,queue=0,port=60020] hdfs.BlockReaderFactory:
I/O error constructing remote block reader.
2014-08-22 14:44:14,302 WARN
[defaultRpcServer.handler=3,queue=0,port=60020] hdfs.DFSClient: Connection
failure: Failed to connect to /192.168.3.154:50010 for file
/hbase/data/default/image/bb082a5f098d1d2a95365545349c
77a4/cf/1af06c6fd36045bab888b4ec18e45f0f for block
BP-1157637685-192.168.3.192-1382642140917:blk_1294911152_255221397:java.io.IOException:
Got error for OP_READ_BLOCK, self=/192.168.3.179:44711, remote=/
192.168.3.154:50 010, for file
/hbase/data/default/image/bb082a5f098d1d2a95365545349c77a4/cf/1af06c6fd36045bab888b4ec18e45f0f,
for pool BP-1157637685-192.168.3.192-1382642140917 block
1294911152_255221397
77a4/cf/1af06c6fd36045bab888b4ec18e45f0f for block
BP-1157637685-192.168.3.192-1382642140917:blk_1294911152_255221397:java.io.IOException:
Got error for OP_READ_BLOCK, self=/192.168.3.179:44711, remote=/
192.168.3.154:50 010, for file
/hbase/data/default/image/bb082a5f098d1d2a95365545349c77a4/cf/1af06c6fd36045bab888b4ec18e45f0f,
for pool BP-1157637685-192.168.3.192-1382642140917 block
1294911152_255221397
2014-08-22 14:49:14,780 WARN
[defaultRpcServer.handler=5,queue=2,port=60020] ipc.RpcServer:
(responseTooSlow):
{"processingtimems":15494,"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiReques
t)","client":"192.168.3.11:53884
","starttimems":1408711739283,"queuetimems":0,"class":"HRegionServer","responsesize":483492,"method":"Multi"}
2014-08-22 14:50:37,900 WARN
[defaultRpcServer.handler=7,queue=1,port=60020] hdfs.BlockReaderFactory:
I/O error constructing remote block reader.
2014-08-22 14:50:37,901 WARN
[defaultRpcServer.handler=7,queue=1,port=60020] hdfs.DFSClient: Connection
failure: Failed to connect to /192.168.3.64:50010 for file
/hbase/data/default/image/a50d67b81c44c864265f5030c7c39
959/cf/840ff51e591946c487413273d5341a24 for block
BP-1157637685-192.168.3.192-1382642140917:blk_1294309355_254555905:java.io.IOException:
Got error for OP_READ_BLOCK, self=/192.168.3.179:45043, remote=/
192.168.3.64:5001 0, for file
/hbase/data/default/image/a50d67b81c44c864265f5030c7c39959/cf/840ff51e591946c487413273d5341a24,
for pool BP-1157637685-192.168.3.192-1382642140917 block
1294309355_254555905
2014-08-22 15:09:35,289 WARN
[defaultRpcServer.handler=23,queue=2,port=60020] hdfs.BlockReaderFactory:
I/O error constructing remote block reader.
2014-08-22 15:09:35,289 WARN
[defaultRpcServer.handler=23,queue=2,port=60020] hdfs.DFSClient:
Connection failure: Failed to connect to /192.168.3.63:50010 for file
/hbase/data/default/image/1db3051c8d41943892c0230cb75b
c1f2/cf/75ae96ea8ca2477e8f83291d7e1fe7cb for block
BP-1157637685-192.168.3.192-1382642140917:blk_1293205473_253317737:java.io.IOException:
Got error for OP_READ_BLOCK, self=/192.168.3.179:56326, remote=/
192.168.3.63:500 10, for file
/hbase/data/default/image/1db3051c8d41943892c0230cb75bc1f2/cf/75ae96ea8ca2477e8f83291d7e1fe7cb,
for pool BP-1157637685-192.168.3.192-1382642140917 block
1293205473_253317737
2014-08-22 15:18:38,891 WARN
[defaultRpcServer.handler=5,queue=2,port=60020] ipc.RpcServer:
(responseTooSlow):
{"processingtimems":26092,"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiReques
t)","client":"192.168.3.91:40189
","starttimems":1408713492722,"queuetimems":0,"class":"HRegionServer","responsesize":483492,"method":"Multi"}
What's interesting is that not all crashed handlers are later missing after
the freeze.
Does that substantiate your conjecture?
BR, Johannes
On Fri, Aug 22, 2014 at 10:42 PM, Stack <st...@duboce.net> wrote:
> Are we losing handler threads, the workers that take from the pool we are
> blocked on?
>
> The attached thread dump has ten with non-sequential numbers:
>
> Thread 97 (defaultRpcServer.handler=27,queue=0,port=60020):
> Thread 94 (defaultRpcServer.handler=24,queue=0,port=60020):
> Thread 91 (defaultRpcServer.handler=21,queue=0,port=60020):
> Thread 90 (defaultRpcServer.handler=20,queue=2,port=60020):
> Thread 88 (defaultRpcServer.handler=18,queue=0,port=60020):
> Thread 82 (defaultRpcServer.handler=12,queue=0,port=60020):
> Thread 81 (defaultRpcServer.handler=11,queue=2,port=60020):
> Thread 76 (defaultRpcServer.handler=6,queue=0,port=60020):
>
> Perhaps this is an artifact of how the thread dump is being taken via the
> UI servlet.
>
> If you jstack, do you see hbase.regionserver.handler.count instances of
> defaultRpcServer going from 0 up to hbase.regionserver.handler.count
>
> If handlers are not taking from the call queue, yeah, it will fill.
>
> St.Ack
>
>
>
>
> On Fri, Aug 22, 2014 at 12:54 PM, Stack <st...@duboce.net> wrote:
>
> > nvm. misread. Trying to figure why the scheduling queue is filled to the
> > brim such that no more calls can be added/dispatched...
> > St.Ack
> >
> >
> > On Fri, Aug 22, 2014 at 12:45 PM, Stack <st...@duboce.net> wrote:
> >
> >> Are you replicating?
> >> St.Ack
> >>
> >>
> >> On Fri, Aug 22, 2014 at 10:28 AM, Johannes Schaback <
> >> johannes.schaback@visual-meta.com> wrote:
> >>
> >>> Dear HBase-Pros,
> >>>
> >>> we face a serious issue with our HBase production cluster for two days
> >>> now.
> >>> Every couple minutes, a random RegionServer gets stuck and does not
> >>> process
> >>> any requests. In addition this causes the other RegionServers to
> >>> freeze within a minute which brings down the entire cluster. Stopping
> the
> >>> affected RegionServer unblocks the cluster and everything comes back to
> >>> normal.
> >>>
> >>> We run 27 RegionServers, each having 31 GB JVM memory. The HBase
> Version
> >>> is
> >>> 0.98.5 on Hadoop 2.4.1. We basically have two tables, the first having
> >>> about 4,500 Regions and holding 8 TB with 1000 requests per second, the
> >>> second table is around 200 Regions with about 50,000 to 120,000
> requests
> >>> per sec over all Regions, 800 GB worth of data and with IN_MEMORY
> >>> enabled.
> >>>
> >>> While investigating the problem, I found out, that every healthy
> >>> RegionServer has the following thread:
> >>>
> >>> Thread 12 (RpcServer.listener,port=60020):
> >>> State: RUNNABLE
> >>> Blocked count: 35
> >>> Waited count: 0
> >>> Stack:
> >>> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> >>> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> >>> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
> >>>
> >>> org.apache.hadoop.hbase.ipc.RpcServer$Listener.run(RpcServer.java:684)
> >>>
> >>>
> >>> When suddenly becoming a blocked RegionServer, this particular thread
> >>> then
> >>> looks like
> >>>
> >>> Thread 12 (RpcServer.listener,port=60020):
> >>> State: BLOCKED
> >>> Blocked count: 2889
> >>> Waited count: 0
> >>> Blocked on
> >>> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader@38cba1a1
> >>> Blocked by 14 (RpcServer.reader=1,port=60020)
> >>> Stack:
> >>>
> >>>
> >>>
> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.registerChannel(RpcServer.java:619)
> >>>
> >>>
> >>>
> org.apache.hadoop.hbase.ipc.RpcServer$Listener.doAccept(RpcServer.java:774)
> >>>
> >>> org.apache.hadoop.hbase.ipc.RpcServer$Listener.run(RpcServer.java:692)
> >>>
> >>>
> >>> Also, JMX shows for an unhealthy RegionServer that
> >>>
> >>> - "queueSize" grows quickly and constantly to values greater than
> 60k,
> >>> and
> >>> - "numCallsInGeneralQueue" quickly reaches 300
> >>>
> >>> Both values are usually very small or 0 under normal circumstances, but
> >>> in
> >>> case of a RS "getting stuck" they explode, which leads me to believe
> that
> >>> the IPC-queue does not get processed properly causing the RegionServer
> to
> >>> become "deaf".
> >>>
> >>> These two symptoms appear to bring down the entire cluster. When
> killign
> >>> that RS, everyhing goes back to normal.
> >>>
> >>> I could not find any correlation between this phenomenon and
> compactions,
> >>> load or other factors. hbck says it is all fine as well.
> >>>
> >>> The servers are all 3.2.0-4-amd64 Debian, 12 cores, 96 GB RAM. Besides
> >>> the
> >>> RS and a DataNode, there isn't too much running on the boxes so the
> load
> >>> (top) is usually around 5 to 10 and bandwidth does not exceed 10 MB on
> >>> average.
> >>>
> >>> We currently survive by polling /jmx of all RegionServers constantly
> and
> >>> restarting those off that show the symptioms :(
> >>>
> >>> Do you have any idea what could be causing this?
> >>>
> >>> Thank you very much in advance!
> >>>
> >>> Johannes
> >>>
> >>
> >>
> >
>
--
LadenZeile.de <http://www.ladenzeile.de/>
powered by Visual Meta GmbH - www.visual-meta.com
Tel.: +49 30 / 609 84 88 20
Fax: +49 30 / 609 84 88 21
E-Mail: johannes.schaback@visual-meta.com
Visual Meta GmbH, Schützenstraße 25, 10117 Berlin
Geschäftsführer: Robert M. Maier, Johannes Schaback
Handelsregister HRB 115795 B, Amtsgericht Charlottenburg
USt-IdNr.: DE263760203
Re: single RegionServer stuck, causing cluster to hang
Posted by Stack <st...@duboce.net>.
Are we losing handler threads, the workers that take from the pool we are
blocked on?
The attached thread dump has ten with non-sequential numbers:
Thread 97 (defaultRpcServer.handler=27,queue=0,port=60020):
Thread 94 (defaultRpcServer.handler=24,queue=0,port=60020):
Thread 91 (defaultRpcServer.handler=21,queue=0,port=60020):
Thread 90 (defaultRpcServer.handler=20,queue=2,port=60020):
Thread 88 (defaultRpcServer.handler=18,queue=0,port=60020):
Thread 82 (defaultRpcServer.handler=12,queue=0,port=60020):
Thread 81 (defaultRpcServer.handler=11,queue=2,port=60020):
Thread 76 (defaultRpcServer.handler=6,queue=0,port=60020):
Perhaps this is an artifact of how the thread dump is being taken via the
UI servlet.
If you jstack, do you see hbase.regionserver.handler.count instances of
defaultRpcServer going from 0 up to hbase.regionserver.handler.count
If handlers are not taking from the call queue, yeah, it will fill.
St.Ack
On Fri, Aug 22, 2014 at 12:54 PM, Stack <st...@duboce.net> wrote:
> nvm. misread. Trying to figure why the scheduling queue is filled to the
> brim such that no more calls can be added/dispatched...
> St.Ack
>
>
> On Fri, Aug 22, 2014 at 12:45 PM, Stack <st...@duboce.net> wrote:
>
>> Are you replicating?
>> St.Ack
>>
>>
>> On Fri, Aug 22, 2014 at 10:28 AM, Johannes Schaback <
>> johannes.schaback@visual-meta.com> wrote:
>>
>>> Dear HBase-Pros,
>>>
>>> we face a serious issue with our HBase production cluster for two days
>>> now.
>>> Every couple minutes, a random RegionServer gets stuck and does not
>>> process
>>> any requests. In addition this causes the other RegionServers to
>>> freeze within a minute which brings down the entire cluster. Stopping the
>>> affected RegionServer unblocks the cluster and everything comes back to
>>> normal.
>>>
>>> We run 27 RegionServers, each having 31 GB JVM memory. The HBase Version
>>> is
>>> 0.98.5 on Hadoop 2.4.1. We basically have two tables, the first having
>>> about 4,500 Regions and holding 8 TB with 1000 requests per second, the
>>> second table is around 200 Regions with about 50,000 to 120,000 requests
>>> per sec over all Regions, 800 GB worth of data and with IN_MEMORY
>>> enabled.
>>>
>>> While investigating the problem, I found out, that every healthy
>>> RegionServer has the following thread:
>>>
>>> Thread 12 (RpcServer.listener,port=60020):
>>> State: RUNNABLE
>>> Blocked count: 35
>>> Waited count: 0
>>> Stack:
>>> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>>>
>>> org.apache.hadoop.hbase.ipc.RpcServer$Listener.run(RpcServer.java:684)
>>>
>>>
>>> When suddenly becoming a blocked RegionServer, this particular thread
>>> then
>>> looks like
>>>
>>> Thread 12 (RpcServer.listener,port=60020):
>>> State: BLOCKED
>>> Blocked count: 2889
>>> Waited count: 0
>>> Blocked on
>>> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader@38cba1a1
>>> Blocked by 14 (RpcServer.reader=1,port=60020)
>>> Stack:
>>>
>>>
>>> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.registerChannel(RpcServer.java:619)
>>>
>>>
>>> org.apache.hadoop.hbase.ipc.RpcServer$Listener.doAccept(RpcServer.java:774)
>>>
>>> org.apache.hadoop.hbase.ipc.RpcServer$Listener.run(RpcServer.java:692)
>>>
>>>
>>> Also, JMX shows for an unhealthy RegionServer that
>>>
>>> - "queueSize" grows quickly and constantly to values greater than 60k,
>>> and
>>> - "numCallsInGeneralQueue" quickly reaches 300
>>>
>>> Both values are usually very small or 0 under normal circumstances, but
>>> in
>>> case of a RS "getting stuck" they explode, which leads me to believe that
>>> the IPC-queue does not get processed properly causing the RegionServer to
>>> become "deaf".
>>>
>>> These two symptoms appear to bring down the entire cluster. When killign
>>> that RS, everyhing goes back to normal.
>>>
>>> I could not find any correlation between this phenomenon and compactions,
>>> load or other factors. hbck says it is all fine as well.
>>>
>>> The servers are all 3.2.0-4-amd64 Debian, 12 cores, 96 GB RAM. Besides
>>> the
>>> RS and a DataNode, there isn't too much running on the boxes so the load
>>> (top) is usually around 5 to 10 and bandwidth does not exceed 10 MB on
>>> average.
>>>
>>> We currently survive by polling /jmx of all RegionServers constantly and
>>> restarting those off that show the symptioms :(
>>>
>>> Do you have any idea what could be causing this?
>>>
>>> Thank you very much in advance!
>>>
>>> Johannes
>>>
>>
>>
>
Re: single RegionServer stuck, causing cluster to hang
Posted by Stack <st...@duboce.net>.
nvm. misread. Trying to figure why the scheduling queue is filled to the
brim such that no more calls can be added/dispatched...
St.Ack
On Fri, Aug 22, 2014 at 12:45 PM, Stack <st...@duboce.net> wrote:
> Are you replicating?
> St.Ack
>
>
> On Fri, Aug 22, 2014 at 10:28 AM, Johannes Schaback <
> johannes.schaback@visual-meta.com> wrote:
>
>> Dear HBase-Pros,
>>
>> we face a serious issue with our HBase production cluster for two days
>> now.
>> Every couple minutes, a random RegionServer gets stuck and does not
>> process
>> any requests. In addition this causes the other RegionServers to
>> freeze within a minute which brings down the entire cluster. Stopping the
>> affected RegionServer unblocks the cluster and everything comes back to
>> normal.
>>
>> We run 27 RegionServers, each having 31 GB JVM memory. The HBase Version
>> is
>> 0.98.5 on Hadoop 2.4.1. We basically have two tables, the first having
>> about 4,500 Regions and holding 8 TB with 1000 requests per second, the
>> second table is around 200 Regions with about 50,000 to 120,000 requests
>> per sec over all Regions, 800 GB worth of data and with IN_MEMORY enabled.
>>
>> While investigating the problem, I found out, that every healthy
>> RegionServer has the following thread:
>>
>> Thread 12 (RpcServer.listener,port=60020):
>> State: RUNNABLE
>> Blocked count: 35
>> Waited count: 0
>> Stack:
>> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>> org.apache.hadoop.hbase.ipc.RpcServer$Listener.run(RpcServer.java:684)
>>
>>
>> When suddenly becoming a blocked RegionServer, this particular thread then
>> looks like
>>
>> Thread 12 (RpcServer.listener,port=60020):
>> State: BLOCKED
>> Blocked count: 2889
>> Waited count: 0
>> Blocked on
>> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader@38cba1a1
>> Blocked by 14 (RpcServer.reader=1,port=60020)
>> Stack:
>>
>>
>> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.registerChannel(RpcServer.java:619)
>>
>>
>> org.apache.hadoop.hbase.ipc.RpcServer$Listener.doAccept(RpcServer.java:774)
>> org.apache.hadoop.hbase.ipc.RpcServer$Listener.run(RpcServer.java:692)
>>
>>
>> Also, JMX shows for an unhealthy RegionServer that
>>
>> - "queueSize" grows quickly and constantly to values greater than 60k,
>> and
>> - "numCallsInGeneralQueue" quickly reaches 300
>>
>> Both values are usually very small or 0 under normal circumstances, but in
>> case of a RS "getting stuck" they explode, which leads me to believe that
>> the IPC-queue does not get processed properly causing the RegionServer to
>> become "deaf".
>>
>> These two symptoms appear to bring down the entire cluster. When killign
>> that RS, everyhing goes back to normal.
>>
>> I could not find any correlation between this phenomenon and compactions,
>> load or other factors. hbck says it is all fine as well.
>>
>> The servers are all 3.2.0-4-amd64 Debian, 12 cores, 96 GB RAM. Besides the
>> RS and a DataNode, there isn't too much running on the boxes so the load
>> (top) is usually around 5 to 10 and bandwidth does not exceed 10 MB on
>> average.
>>
>> We currently survive by polling /jmx of all RegionServers constantly and
>> restarting those off that show the symptioms :(
>>
>> Do you have any idea what could be causing this?
>>
>> Thank you very much in advance!
>>
>> Johannes
>>
>
>
Re: single RegionServer stuck, causing cluster to hang
Posted by Stack <st...@duboce.net>.
Are you replicating?
St.Ack
On Fri, Aug 22, 2014 at 10:28 AM, Johannes Schaback <
johannes.schaback@visual-meta.com> wrote:
> Dear HBase-Pros,
>
> we face a serious issue with our HBase production cluster for two days now.
> Every couple minutes, a random RegionServer gets stuck and does not process
> any requests. In addition this causes the other RegionServers to
> freeze within a minute which brings down the entire cluster. Stopping the
> affected RegionServer unblocks the cluster and everything comes back to
> normal.
>
> We run 27 RegionServers, each having 31 GB JVM memory. The HBase Version is
> 0.98.5 on Hadoop 2.4.1. We basically have two tables, the first having
> about 4,500 Regions and holding 8 TB with 1000 requests per second, the
> second table is around 200 Regions with about 50,000 to 120,000 requests
> per sec over all Regions, 800 GB worth of data and with IN_MEMORY enabled.
>
> While investigating the problem, I found out, that every healthy
> RegionServer has the following thread:
>
> Thread 12 (RpcServer.listener,port=60020):
> State: RUNNABLE
> Blocked count: 35
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
> org.apache.hadoop.hbase.ipc.RpcServer$Listener.run(RpcServer.java:684)
>
>
> When suddenly becoming a blocked RegionServer, this particular thread then
> looks like
>
> Thread 12 (RpcServer.listener,port=60020):
> State: BLOCKED
> Blocked count: 2889
> Waited count: 0
> Blocked on org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader@38cba1a1
> Blocked by 14 (RpcServer.reader=1,port=60020)
> Stack:
>
>
> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.registerChannel(RpcServer.java:619)
>
> org.apache.hadoop.hbase.ipc.RpcServer$Listener.doAccept(RpcServer.java:774)
> org.apache.hadoop.hbase.ipc.RpcServer$Listener.run(RpcServer.java:692)
>
>
> Also, JMX shows for an unhealthy RegionServer that
>
> - "queueSize" grows quickly and constantly to values greater than 60k,
> and
> - "numCallsInGeneralQueue" quickly reaches 300
>
> Both values are usually very small or 0 under normal circumstances, but in
> case of a RS "getting stuck" they explode, which leads me to believe that
> the IPC-queue does not get processed properly causing the RegionServer to
> become "deaf".
>
> These two symptoms appear to bring down the entire cluster. When killign
> that RS, everyhing goes back to normal.
>
> I could not find any correlation between this phenomenon and compactions,
> load or other factors. hbck says it is all fine as well.
>
> The servers are all 3.2.0-4-amd64 Debian, 12 cores, 96 GB RAM. Besides the
> RS and a DataNode, there isn't too much running on the boxes so the load
> (top) is usually around 5 to 10 and bandwidth does not exceed 10 MB on
> average.
>
> We currently survive by polling /jmx of all RegionServers constantly and
> restarting those off that show the symptioms :(
>
> Do you have any idea what could be causing this?
>
> Thank you very much in advance!
>
> Johannes
>