You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Andrew Purtell (JIRA)" <ji...@apache.org> on 2009/01/30 03:02:59 UTC
[jira] Issue Comment Edited: (HBASE-1163) Master root scanner hung,
clients blocked indefinitely waiting for getStartKeys()
[ https://issues.apache.org/jira/browse/HBASE-1163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12668709#action_12668709 ]
apurtell edited comment on HBASE-1163 at 1/29/09 6:02 PM:
----------------------------------------------------------------
Nothing amiss on the HRS hosting ROOT as far as I can see.
Thread 311 (IPC Client (47) connection to sjdc-atr-dc-2.atr.trendmicro.com/10.30.94.31:60000 from an unknown user):
State: TIMED_WAITING
Blocked count: 1595
Waited count: 1595
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.waitForWork(HBaseClient.java:400)
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:442)
Here is the corresponding IPC thread on the master:
Thread 309 (IPC Client (47) connection to /10.30.94.32:60020 from an unknown user):
State: RUNNABLE
Blocked count: 1
Waited count: 1
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:260)
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:155)
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:150)
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:123)
java.io.FilterInputStream.read(FilterInputStream.java:116)
org.apache.hadoop.hbase.ipc.HBaseClient$Connection$PingInputStream.read(HBaseClient.java:276)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:237)
java.io.DataInputStream.readInt(DataInputStream.java:370)
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:498)
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:443)
was (Author: apurtell):
Nothing amiss on the HRS hosting ROOT as far as I can see.
Thread 311 (IPC Client (47) connection to sjdc-atr-dc-2.atr.trendmicro.com/10.30.94.31:60000 from an unknown user):
State: TIMED_WAITING
Blocked count: 1595
Waited count: 1595
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.waitForWork(HBaseClient.java:400)
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:442)
> Master root scanner hung, clients blocked indefinitely waiting for getStartKeys()
> ---------------------------------------------------------------------------------
>
> Key: HBASE-1163
> URL: https://issues.apache.org/jira/browse/HBASE-1163
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.19.0
> Reporter: Andrew Purtell
> Priority: Critical
>
> Mapreduce tasks based on TIF won't start. Clients trying to find regions by start key block indefinitely (Heritrix hbase writer eventually times out archiver).
> Master seems hung in root scan. I've dumped thread stacks 10 times in 10 minutes and the same HBaseClient$Call object appears in the trace. See below:
> Thread 21 (RegionManager.rootScanner):
> State: WAITING
> Blocked count: 500
> Waited count: 621
> Waiting on org.apache.hadoop.hbase.ipc.HBaseClient$Call@55a2896d
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:695)
> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:321)
> $Proxy2.next(Unknown Source)
> org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:161)
> org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:55)
> org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:80)
> org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:137)
> org.apache.hadoop.hbase.Chore.run(Chore.java:65)
> I only see messages from the MetaScanner scanner in the master log, nothing from RootScanner.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.