You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2010/09/08 08:05:33 UTC
[jira] Commented: (HBASE-2966) HBase client stuck on
org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1278) holding
regionLockObject lock
[ https://issues.apache.org/jira/browse/HBASE-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12907100#action_12907100 ]
Todd Lipcon commented on HBASE-2966:
------------------------------------
I replied on IRC but guess you missed it.. got a stack trace for the other ZK threads? There should be a SendThread and an EventThread
> HBase client stuck on org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1278) holding regionLockObject lock
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-2966
> URL: https://issues.apache.org/jira/browse/HBASE-2966
> Project: HBase
> Issue Type: Bug
> Reporter: Kannan Muthukkaruppan
>
> We noticed in one case the HBase client program got stuck on Zookeeper.exists() call.
>
> One of the threads was stuck here on the ZK call while holding an HBase level lock (regionLockObject in locateRegionInMeta()).
> {code}
> "thrift-0-thread-8" prio=10 tid=0x00007f189ca4c000 nid=0x550f in Object.wait() [0x0000000044241000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1278)
> - locked <0x00007f1903a0c280> (a org.apache.zookeeper.ClientCnxn$Packet)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:804)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:837)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.getRSDirectoryCount(ZooKeeperWrapper.java:765)
> at org.apache.hadoop.hbase.client.HTable.getCurrentNrHRS(HTable.java:173)
> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:147)
> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:124)
> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:89)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.prefetchRegionCache(HConnectionManager.java:734)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:785)
> - locked <0x00007f190d868848> (a java.lang.Object)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:679)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:646)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionLocation(HConnectionManager.java:472)
> at org.apache.hadoop.hbase.client.ServerCallable.instantiateServer(ServerCallable.java:57)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:1147)
> at org.apache.hadoop.hbase.client.HTable.get(HTable.java:503)
> {code}
> The remaining other threads are all waiting on the regionLockObject lock (held by the above thread) with stacks like:
>
> {code}
> thrift-0-thread-7" prio=10 tid=0x00007f189ca4a800 nid=0x550e waiting for monitor entry [0x0000000044141000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:783)
> - waiting to lock <0x00007f190d868848> (a java.lang.Object)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:679)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:646)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionLocation(HConnectionManager.java:472)
> at org.apache.hadoop.hbase.client.ServerCallable.instantiateServer(ServerCallable.java:57)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:1147)
> at org.apache.hadoop.hbase.client.HTable.get(HTable.java:503)
> {code}
> Any ideas?
>
> Meanwhile, I'll look into the ZK logs from the relevant time some more and get back if I have more information.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.