You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@hbase.apache.org by "liwei (JIRA)" <ji...@apache.org> on 2012/12/03 08:39:58 UTC

[jira] [Created] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured

liwei created HBASE-7259:
----------------------------

             Summary: Deadlock in HBaseClient when KeeperException occured
                 Key: HBASE-7259
                 URL: https://issues.apache.org/jira/browse/HBASE-7259
             Project: HBase
          Issue Type: Bug
          Components: Zookeeper
    Affects Versions: 0.94.2, 0.94.1, 0.94.0
            Reporter: liwei
            Priority: Critical


HBaseClient was running after a period of time, all of get operation became too slow.

>From the client logs I could see the following:

1. Unable to get data of znode /hbase/root-region-server
java.lang.InterruptedException
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
        at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
        at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
        at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
        at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
        at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)

2. jstack traces found one Java-level deadlock:

=============================

"catalina-exec-800":
  waiting to lock monitor 0x000000005f1f6530 (object 0x0000000731902200, a java.lang.Object),
  which is held by "catalina-exec-710"
"catalina-exec-710":
  waiting to lock monitor 0x00002aaab9a05bd0 (object 0x00000007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
  which is held by "catalina-exec-29-EventThread"
"catalina-exec-29-EventThread":
  waiting to lock monitor 0x000000005f9f0af0 (object 0x0000000732a9c7e0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
  which is held by "catalina-exec-710"
Java stack information for the threads listed above:

===================================================

"catalina-exec-800":
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:943)
        - waiting to lock <0x0000000731902200> (a java.lang.Object)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
        at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
        at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
        at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
"catalina-exec-710":
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:599)
        - waiting to lock <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:158)
        - locked <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
        at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
        at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
        - locked <0x0000000731902200> (a java.lang.Object)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
        at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
        at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
        at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
"catalina-exec-29-EventThread":
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.stop(ZooKeeperNodeTracker.java:98)
        - waiting to lock <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:604)
        - locked <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:374)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
Found 1 deadlock.

>From the source code , the reason for this problem is doing ZooKeeperNodeTracker.getData that has a KeeperException occured. And try to resetZookeeperTracker. At the same time, ClientCnxn.EventThread  also do resetZookeeperTracker ,too. Because of getData have already held the lock of  ZooKeeperNodeTracke , that lead to the order of the lock two threads to obtain does not accord. So deadlock happened.

In order to avoid the problem, we can through reduce range of the lock of getData. 
See the patch with 0.94.0.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured

Posted by "Ted Yu (JIRA)" <ji...@apache.org>.

     [ https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ted Yu updated HBASE-7259:
--------------------------

    Description: 
HBaseClient was running after a period of time, all of get operation became too slow.

>From the client logs I could see the following:

1. Unable to get data of znode /hbase/root-region-server
{code}
java.lang.InterruptedException
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
        at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
        at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
        at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
        at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
        at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
{code}

2. Catalina.out found one Java-level deadlock:
{code}
=============================

"catalina-exec-800":
  waiting to lock monitor 0x000000005f1f6530 (object 0x0000000731902200, a java.lang.Object),
  which is held by "catalina-exec-710"
"catalina-exec-710":
  waiting to lock monitor 0x00002aaab9a05bd0 (object 0x00000007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
  which is held by "catalina-exec-29-EventThread"
"catalina-exec-29-EventThread":
  waiting to lock monitor 0x000000005f9f0af0 (object 0x0000000732a9c7e0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
  which is held by "catalina-exec-710"
Java stack information for the threads listed above:

===================================================

"catalina-exec-800":
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:943)
        - waiting to lock <0x0000000731902200> (a java.lang.Object)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
        at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
        at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
        at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
"catalina-exec-710":
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:599)
        - waiting to lock <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:158)
        - locked <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
        at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
        at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
        - locked <0x0000000731902200> (a java.lang.Object)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
        at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
        at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
        at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
"catalina-exec-29-EventThread":
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.stop(ZooKeeperNodeTracker.java:98)
        - waiting to lock <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:604)
        - locked <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:374)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
Found 1 deadlock.
{code}
>From the source code , the reason for this problem is doing ZooKeeperNodeTracker.getData that lead to KeeperException. And try to resetZookeeperTracker. At the same time, ClientCnxn.EventThread  also do resetZookeeperTracker ,too. Because of getData have already held the lock of  ZooKeeperNodeTracke , that lead to the order of the lock two threads to obtain does not accord. So deadlock happened.

In order to avoid the problem, we can add if reseting condition in abortable.abort()
See the patch.

  was:
HBaseClient was running after a period of time, all of get operation became too slow.

>From the client logs I could see the following:

1. Unable to get data of znode /hbase/root-region-server
java.lang.InterruptedException
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
        at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
        at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
        at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
        at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
        at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)

2. Catalina.out found one Java-level deadlock:

=============================

"catalina-exec-800":
  waiting to lock monitor 0x000000005f1f6530 (object 0x0000000731902200, a java.lang.Object),
  which is held by "catalina-exec-710"
"catalina-exec-710":
  waiting to lock monitor 0x00002aaab9a05bd0 (object 0x00000007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
  which is held by "catalina-exec-29-EventThread"
"catalina-exec-29-EventThread":
  waiting to lock monitor 0x000000005f9f0af0 (object 0x0000000732a9c7e0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
  which is held by "catalina-exec-710"
Java stack information for the threads listed above:

===================================================

"catalina-exec-800":
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:943)
        - waiting to lock <0x0000000731902200> (a java.lang.Object)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
        at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
        at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
        at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
"catalina-exec-710":
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:599)
        - waiting to lock <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:158)
        - locked <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
        at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
        at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
        - locked <0x0000000731902200> (a java.lang.Object)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
        at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
        at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
        at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
"catalina-exec-29-EventThread":
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.stop(ZooKeeperNodeTracker.java:98)
        - waiting to lock <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:604)
        - locked <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:374)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
Found 1 deadlock.

>From the source code , the reason for this problem is doing ZooKeeperNodeTracker.getData that lead to KeeperException. And try to resetZookeeperTracker. At the same time, ClientCnxn.EventThread  also do resetZookeeperTracker ,too. Because of getData have already held the lock of  ZooKeeperNodeTracke , that lead to the order of the lock two threads to obtain does not accord. So deadlock happened.

In order to avoid the problem, we can add if reseting condition in abortable.abort()
See the patch.

    
> Deadlock in HBaseClient when KeeperException occured
> ----------------------------------------------------
>
>                 Key: HBASE-7259
>                 URL: https://issues.apache.org/jira/browse/HBASE-7259
>             Project: HBase
>          Issue Type: Bug
>          Components: Zookeeper
>    Affects Versions: 0.94.0, 0.94.1, 0.94.2
>            Reporter: liwei
>            Priority: Critical
>         Attachments: HConnectionManager.patch
>
>
> HBaseClient was running after a period of time, all of get operation became too slow.
> From the client logs I could see the following:
> 1. Unable to get data of znode /hbase/root-region-server
> {code}
> java.lang.InterruptedException
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
>         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
>         at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
>         at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
>         at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
>         at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
>         at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> {code}
> 2. Catalina.out found one Java-level deadlock:
> {code}
> =============================
> "catalina-exec-800":
>   waiting to lock monitor 0x000000005f1f6530 (object 0x0000000731902200, a java.lang.Object),
>   which is held by "catalina-exec-710"
> "catalina-exec-710":
>   waiting to lock monitor 0x00002aaab9a05bd0 (object 0x00000007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
>   which is held by "catalina-exec-29-EventThread"
> "catalina-exec-29-EventThread":
>   waiting to lock monitor 0x000000005f9f0af0 (object 0x0000000732a9c7e0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
>   which is held by "catalina-exec-710"
> Java stack information for the threads listed above:
> ===================================================
> "catalina-exec-800":
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:943)
>         - waiting to lock <0x0000000731902200> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> "catalina-exec-710":
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:599)
>         - waiting to lock <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:158)
>         - locked <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
>         at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
>         at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
>         - locked <0x0000000731902200> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> "catalina-exec-29-EventThread":
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.stop(ZooKeeperNodeTracker.java:98)
>         - waiting to lock <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:604)
>         - locked <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:374)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271)
>         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
>         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
> Found 1 deadlock.
> {code}
> From the source code , the reason for this problem is doing ZooKeeperNodeTracker.getData that lead to KeeperException. And try to resetZookeeperTracker. At the same time, ClientCnxn.EventThread  also do resetZookeeperTracker ,too. Because of getData have already held the lock of  ZooKeeperNodeTracke , that lead to the order of the lock two threads to obtain does not accord. So deadlock happened.
> In order to avoid the problem, we can add if reseting condition in abortable.abort()
> See the patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured

Posted by "liwei (JIRA)" <ji...@apache.org>.

     [ https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

liwei updated HBASE-7259:
-------------------------

    Attachment: ZookeeperNodeTracker.patch
    
> Deadlock in HBaseClient when KeeperException occured
> ----------------------------------------------------
>
>                 Key: HBASE-7259
>                 URL: https://issues.apache.org/jira/browse/HBASE-7259
>             Project: HBase
>          Issue Type: Bug
>          Components: Zookeeper
>    Affects Versions: 0.94.0, 0.94.1, 0.94.2
>            Reporter: liwei
>            Priority: Critical
>         Attachments: ZookeeperNodeTracker.patch
>
>
> HBaseClient was running after a period of time, all of get operation became too slow.
> From the client logs I could see the following:
> 1. Unable to get data of znode /hbase/root-region-server
> java.lang.InterruptedException
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
>         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
>         at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
>         at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
>         at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
>         at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
>         at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> 2. jstack traces found one Java-level deadlock:
> =============================
> "catalina-exec-800":
>   waiting to lock monitor 0x000000005f1f6530 (object 0x0000000731902200, a java.lang.Object),
>   which is held by "catalina-exec-710"
> "catalina-exec-710":
>   waiting to lock monitor 0x00002aaab9a05bd0 (object 0x00000007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
>   which is held by "catalina-exec-29-EventThread"
> "catalina-exec-29-EventThread":
>   waiting to lock monitor 0x000000005f9f0af0 (object 0x0000000732a9c7e0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
>   which is held by "catalina-exec-710"
> Java stack information for the threads listed above:
> ===================================================
> "catalina-exec-800":
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:943)
>         - waiting to lock <0x0000000731902200> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> "catalina-exec-710":
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:599)
>         - waiting to lock <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:158)
>         - locked <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
>         at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
>         at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
>         - locked <0x0000000731902200> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> "catalina-exec-29-EventThread":
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.stop(ZooKeeperNodeTracker.java:98)
>         - waiting to lock <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:604)
>         - locked <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:374)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271)
>         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
>         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
> Found 1 deadlock.
> From the source code , the reason for this problem is doing ZooKeeperNodeTracker.getData that has a KeeperException occured. And try to resetZookeeperTracker. At the same time, ClientCnxn.EventThread  also do resetZookeeperTracker ,too. Because of getData have already held the lock of  ZooKeeperNodeTracke , that lead to the order of the lock two threads to obtain does not accord. So deadlock happened.
> In order to avoid the problem, we can through reduce range of the lock of getData. 
> See the patch with 0.94.0.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured

Posted by "liwei (JIRA)" <ji...@apache.org>.

     [ https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

liwei updated HBASE-7259:
-------------------------

    Attachment: 7259-0.94-branch.txt
    
> Deadlock in HBaseClient when KeeperException occured
> ----------------------------------------------------
>
>                 Key: HBASE-7259
>                 URL: https://issues.apache.org/jira/browse/HBASE-7259
>             Project: HBase
>          Issue Type: Bug
>          Components: Zookeeper
>    Affects Versions: 0.94.0, 0.94.1, 0.94.2
>            Reporter: liwei
>            Priority: Critical
>             Fix For: 0.94.4
>
>         Attachments: 7259-0.94-branch.txt, HBASE-7259-0.94.2.txt
>
>
> HBaseClient was running after a period of time, all of get operation became too slow.
> From the client logs I could see the following:
> 1. Unable to get data of znode /hbase/root-region-server
> {code}
> java.lang.InterruptedException
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
>         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
>         at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
>         at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
>         at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
>         at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
>         at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> {code}
> 2. Catalina.out found one Java-level deadlock:
> {code}
> =============================
> "catalina-exec-800":
>   waiting to lock monitor 0x000000005f1f6530 (object 0x0000000731902200, a java.lang.Object),
>   which is held by "catalina-exec-710"
> "catalina-exec-710":
>   waiting to lock monitor 0x00002aaab9a05bd0 (object 0x00000007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
>   which is held by "catalina-exec-29-EventThread"
> "catalina-exec-29-EventThread":
>   waiting to lock monitor 0x000000005f9f0af0 (object 0x0000000732a9c7e0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
>   which is held by "catalina-exec-710"
> Java stack information for the threads listed above:
> ===================================================
> "catalina-exec-800":
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:943)
>         - waiting to lock <0x0000000731902200> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> "catalina-exec-710":
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:599)
>         - waiting to lock <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:158)
>         - locked <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
>         at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
>         at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
>         - locked <0x0000000731902200> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> "catalina-exec-29-EventThread":
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.stop(ZooKeeperNodeTracker.java:98)
>         - waiting to lock <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:604)
>         - locked <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:374)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271)
>         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
>         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
> Found 1 deadlock.
> {code}
> From the source code , the reason for this problem is doing ZooKeeperNodeTracker.getData that lead to KeeperException. And try to resetZookeeperTracker. At the same time, ClientCnxn.EventThread  also do resetZookeeperTracker ,too. Because of getData have already held the lock of  ZooKeeperNodeTracke , that lead to the order of the lock two threads to obtain does not accord. So deadlock happened.
> In order to avoid the problem, we can add if reseting condition in abortable.abort()
> See the patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured

Posted by "liwei (JIRA)" <ji...@apache.org>.

     [ https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

liwei updated HBASE-7259:
-------------------------

    Attachment:     (was: HConnectionManager.patch)
    
> Deadlock in HBaseClient when KeeperException occured
> ----------------------------------------------------
>
>                 Key: HBASE-7259
>                 URL: https://issues.apache.org/jira/browse/HBASE-7259
>             Project: HBase
>          Issue Type: Bug
>          Components: Zookeeper
>    Affects Versions: 0.94.0, 0.94.1, 0.94.2
>            Reporter: liwei
>            Priority: Critical
>
> HBaseClient was running after a period of time, all of get operation became too slow.
> From the client logs I could see the following:
> 1. Unable to get data of znode /hbase/root-region-server
> java.lang.InterruptedException
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
>         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
>         at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
>         at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
>         at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
>         at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
>         at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> 2. Catalina.out found one Java-level deadlock:
> =============================
> "catalina-exec-800":
>   waiting to lock monitor 0x000000005f1f6530 (object 0x0000000731902200, a java.lang.Object),
>   which is held by "catalina-exec-710"
> "catalina-exec-710":
>   waiting to lock monitor 0x00002aaab9a05bd0 (object 0x00000007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
>   which is held by "catalina-exec-29-EventThread"
> "catalina-exec-29-EventThread":
>   waiting to lock monitor 0x000000005f9f0af0 (object 0x0000000732a9c7e0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
>   which is held by "catalina-exec-710"
> Java stack information for the threads listed above:
> ===================================================
> "catalina-exec-800":
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:943)
>         - waiting to lock <0x0000000731902200> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> "catalina-exec-710":
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:599)
>         - waiting to lock <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:158)
>         - locked <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
>         at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
>         at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
>         - locked <0x0000000731902200> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> "catalina-exec-29-EventThread":
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.stop(ZooKeeperNodeTracker.java:98)
>         - waiting to lock <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:604)
>         - locked <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:374)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271)
>         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
>         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
> Found 1 deadlock.
> From the source code , the reason for this problem is doing ZooKeeperNodeTracker.getData that lead to KeeperException. And try to resetZookeeperTracker. At the same time, ClientCnxn.EventThread  also do resetZookeeperTracker ,too. Because of getData have already held the lock of  ZooKeeperNodeTracke , that lead to the order of the lock two threads to obtain does not accord. So deadlock happened.
> In order to avoid the problem, we can add if reseting condition in abortable.abort()
> See the patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured

Posted by "liwei (JIRA)" <ji...@apache.org>.

     [ https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

liwei updated HBASE-7259:
-------------------------

    Attachment:     (was: ZookeeperNodeTracker.patch)
    
> Deadlock in HBaseClient when KeeperException occured
> ----------------------------------------------------
>
>                 Key: HBASE-7259
>                 URL: https://issues.apache.org/jira/browse/HBASE-7259
>             Project: HBase
>          Issue Type: Bug
>          Components: Zookeeper
>    Affects Versions: 0.94.0, 0.94.1, 0.94.2
>            Reporter: liwei
>            Priority: Critical
>
> HBaseClient was running after a period of time, all of get operation became too slow.
> From the client logs I could see the following:
> 1. Unable to get data of znode /hbase/root-region-server
> java.lang.InterruptedException
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
>         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
>         at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
>         at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
>         at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
>         at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
>         at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> 2. Catalina.out found one Java-level deadlock:
> =============================
> "catalina-exec-800":
>   waiting to lock monitor 0x000000005f1f6530 (object 0x0000000731902200, a java.lang.Object),
>   which is held by "catalina-exec-710"
> "catalina-exec-710":
>   waiting to lock monitor 0x00002aaab9a05bd0 (object 0x00000007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
>   which is held by "catalina-exec-29-EventThread"
> "catalina-exec-29-EventThread":
>   waiting to lock monitor 0x000000005f9f0af0 (object 0x0000000732a9c7e0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
>   which is held by "catalina-exec-710"
> Java stack information for the threads listed above:
> ===================================================
> "catalina-exec-800":
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:943)
>         - waiting to lock <0x0000000731902200> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> "catalina-exec-710":
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:599)
>         - waiting to lock <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:158)
>         - locked <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
>         at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
>         at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
>         - locked <0x0000000731902200> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> "catalina-exec-29-EventThread":
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.stop(ZooKeeperNodeTracker.java:98)
>         - waiting to lock <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:604)
>         - locked <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:374)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271)
>         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
>         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
> Found 1 deadlock.
> From the source code , the reason for this problem is doing ZooKeeperNodeTracker.getData that lead to KeeperException. And try to resetZookeeperTracker. At the same time, ClientCnxn.EventThread  also do resetZookeeperTracker ,too. Because of getData have already held the lock of  ZooKeeperNodeTracke , that lead to the order of the lock two threads to obtain does not accord. So deadlock happened.
> In order to avoid the problem, we can through reduce range of the lock of getData. 
> See the patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured

Posted by "liwei (JIRA)" <ji...@apache.org>.

     [ https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

liwei updated HBASE-7259:
-------------------------

    Description: 
HBaseClient was running after a period of time, all of get operation became too slow.

>From the client logs I could see the following:

1. Unable to get data of znode /hbase/root-region-server
java.lang.InterruptedException
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
        at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
        at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
        at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
        at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
        at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)

2. Catalina.out found one Java-level deadlock:

=============================

"catalina-exec-800":
  waiting to lock monitor 0x000000005f1f6530 (object 0x0000000731902200, a java.lang.Object),
  which is held by "catalina-exec-710"
"catalina-exec-710":
  waiting to lock monitor 0x00002aaab9a05bd0 (object 0x00000007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
  which is held by "catalina-exec-29-EventThread"
"catalina-exec-29-EventThread":
  waiting to lock monitor 0x000000005f9f0af0 (object 0x0000000732a9c7e0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
  which is held by "catalina-exec-710"
Java stack information for the threads listed above:

===================================================

"catalina-exec-800":
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:943)
        - waiting to lock <0x0000000731902200> (a java.lang.Object)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
        at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
        at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
        at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
"catalina-exec-710":
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:599)
        - waiting to lock <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:158)
        - locked <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
        at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
        at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
        - locked <0x0000000731902200> (a java.lang.Object)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
        at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
        at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
        at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
"catalina-exec-29-EventThread":
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.stop(ZooKeeperNodeTracker.java:98)
        - waiting to lock <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:604)
        - locked <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:374)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
Found 1 deadlock.

>From the source code , the reason for this problem is doing ZooKeeperNodeTracker.getData that lead to KeeperException. And try to resetZookeeperTracker. At the same time, ClientCnxn.EventThread  also do resetZookeeperTracker ,too. Because of getData have already held the lock of  ZooKeeperNodeTracke , that lead to the order of the lock two threads to obtain does not accord. So deadlock happened.

In order to avoid the problem, we can through reduce range of the lock of getData. 
See the patch with 0.94.0.

  was:
HBaseClient was running after a period of time, all of get operation became too slow.

>From the client logs I could see the following:

1. Unable to get data of znode /hbase/root-region-server
java.lang.InterruptedException
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
        at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
        at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
        at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
        at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
        at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)

2. jstack traces found one Java-level deadlock:

=============================

"catalina-exec-800":
  waiting to lock monitor 0x000000005f1f6530 (object 0x0000000731902200, a java.lang.Object),
  which is held by "catalina-exec-710"
"catalina-exec-710":
  waiting to lock monitor 0x00002aaab9a05bd0 (object 0x00000007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
  which is held by "catalina-exec-29-EventThread"
"catalina-exec-29-EventThread":
  waiting to lock monitor 0x000000005f9f0af0 (object 0x0000000732a9c7e0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
  which is held by "catalina-exec-710"
Java stack information for the threads listed above:

===================================================

"catalina-exec-800":
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:943)
        - waiting to lock <0x0000000731902200> (a java.lang.Object)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
        at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
        at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
        at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
"catalina-exec-710":
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:599)
        - waiting to lock <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:158)
        - locked <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
        at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
        at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
        - locked <0x0000000731902200> (a java.lang.Object)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
        at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
        at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
        at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
"catalina-exec-29-EventThread":
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.stop(ZooKeeperNodeTracker.java:98)
        - waiting to lock <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:604)
        - locked <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:374)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
Found 1 deadlock.

>From the source code , the reason for this problem is doing ZooKeeperNodeTracker.getData that lead to KeeperException. And try to resetZookeeperTracker. At the same time, ClientCnxn.EventThread  also do resetZookeeperTracker ,too. Because of getData have already held the lock of  ZooKeeperNodeTracke , that lead to the order of the lock two threads to obtain does not accord. So deadlock happened.

In order to avoid the problem, we can through reduce range of the lock of getData. 
See the patch with 0.94.0.

    
> Deadlock in HBaseClient when KeeperException occured
> ----------------------------------------------------
>
>                 Key: HBASE-7259
>                 URL: https://issues.apache.org/jira/browse/HBASE-7259
>             Project: HBase
>          Issue Type: Bug
>          Components: Zookeeper
>    Affects Versions: 0.94.0, 0.94.1, 0.94.2
>            Reporter: liwei
>            Priority: Critical
>         Attachments: ZookeeperNodeTracker.patch
>
>
> HBaseClient was running after a period of time, all of get operation became too slow.
> From the client logs I could see the following:
> 1. Unable to get data of znode /hbase/root-region-server
> java.lang.InterruptedException
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
>         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
>         at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
>         at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
>         at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
>         at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
>         at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> 2. Catalina.out found one Java-level deadlock:
> =============================
> "catalina-exec-800":
>   waiting to lock monitor 0x000000005f1f6530 (object 0x0000000731902200, a java.lang.Object),
>   which is held by "catalina-exec-710"
> "catalina-exec-710":
>   waiting to lock monitor 0x00002aaab9a05bd0 (object 0x00000007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
>   which is held by "catalina-exec-29-EventThread"
> "catalina-exec-29-EventThread":
>   waiting to lock monitor 0x000000005f9f0af0 (object 0x0000000732a9c7e0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
>   which is held by "catalina-exec-710"
> Java stack information for the threads listed above:
> ===================================================
> "catalina-exec-800":
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:943)
>         - waiting to lock <0x0000000731902200> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> "catalina-exec-710":
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:599)
>         - waiting to lock <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:158)
>         - locked <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
>         at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
>         at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
>         - locked <0x0000000731902200> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> "catalina-exec-29-EventThread":
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.stop(ZooKeeperNodeTracker.java:98)
>         - waiting to lock <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:604)
>         - locked <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:374)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271)
>         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
>         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
> Found 1 deadlock.
> From the source code , the reason for this problem is doing ZooKeeperNodeTracker.getData that lead to KeeperException. And try to resetZookeeperTracker. At the same time, ClientCnxn.EventThread  also do resetZookeeperTracker ,too. Because of getData have already held the lock of  ZooKeeperNodeTracke , that lead to the order of the lock two threads to obtain does not accord. So deadlock happened.
> In order to avoid the problem, we can through reduce range of the lock of getData. 
> See the patch with 0.94.0.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured

Posted by "Ted Yu (JIRA)" <ji...@apache.org>.

    [ https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508573#comment-13508573 ] 

Ted Yu commented on HBASE-7259:
-------------------------------

Can you provide patch for trunk so that Hadoop QA can run through test suite ?

Thanks
                
> Deadlock in HBaseClient when KeeperException occured
> ----------------------------------------------------
>
>                 Key: HBASE-7259
>                 URL: https://issues.apache.org/jira/browse/HBASE-7259
>             Project: HBase
>          Issue Type: Bug
>          Components: Zookeeper
>    Affects Versions: 0.94.0, 0.94.1, 0.94.2
>            Reporter: liwei
>            Priority: Critical
>         Attachments: ZookeeperNodeTracker.patch
>
>
> HBaseClient was running after a period of time, all of get operation became too slow.
> From the client logs I could see the following:
> 1. Unable to get data of znode /hbase/root-region-server
> java.lang.InterruptedException
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
>         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
>         at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
>         at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
>         at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
>         at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
>         at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> 2. Catalina.out found one Java-level deadlock:
> =============================
> "catalina-exec-800":
>   waiting to lock monitor 0x000000005f1f6530 (object 0x0000000731902200, a java.lang.Object),
>   which is held by "catalina-exec-710"
> "catalina-exec-710":
>   waiting to lock monitor 0x00002aaab9a05bd0 (object 0x00000007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
>   which is held by "catalina-exec-29-EventThread"
> "catalina-exec-29-EventThread":
>   waiting to lock monitor 0x000000005f9f0af0 (object 0x0000000732a9c7e0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
>   which is held by "catalina-exec-710"
> Java stack information for the threads listed above:
> ===================================================
> "catalina-exec-800":
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:943)
>         - waiting to lock <0x0000000731902200> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> "catalina-exec-710":
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:599)
>         - waiting to lock <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:158)
>         - locked <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
>         at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
>         at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
>         - locked <0x0000000731902200> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> "catalina-exec-29-EventThread":
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.stop(ZooKeeperNodeTracker.java:98)
>         - waiting to lock <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:604)
>         - locked <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:374)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271)
>         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
>         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
> Found 1 deadlock.
> From the source code , the reason for this problem is doing ZooKeeperNodeTracker.getData that lead to KeeperException. And try to resetZookeeperTracker. At the same time, ClientCnxn.EventThread  also do resetZookeeperTracker ,too. Because of getData have already held the lock of  ZooKeeperNodeTracke , that lead to the order of the lock two threads to obtain does not accord. So deadlock happened.
> In order to avoid the problem, we can through reduce range of the lock of getData. 
> See the patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured

Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.

    [ https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508873#comment-13508873 ] 

Lars Hofhansl commented on HBASE-7259:
--------------------------------------

Things are much different in trunk as far as the client is concerned.
A trunk patch for this neither realistic nor useful.

Thanks for the analysis [~boneylw]

                
> Deadlock in HBaseClient when KeeperException occured
> ----------------------------------------------------
>
>                 Key: HBASE-7259
>                 URL: https://issues.apache.org/jira/browse/HBASE-7259
>             Project: HBase
>          Issue Type: Bug
>          Components: Zookeeper
>    Affects Versions: 0.94.0, 0.94.1, 0.94.2
>            Reporter: liwei
>            Priority: Critical
>         Attachments: HConnectionManager.patch
>
>
> HBaseClient was running after a period of time, all of get operation became too slow.
> From the client logs I could see the following:
> 1. Unable to get data of znode /hbase/root-region-server
> java.lang.InterruptedException
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
>         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
>         at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
>         at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
>         at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
>         at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
>         at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> 2. Catalina.out found one Java-level deadlock:
> =============================
> "catalina-exec-800":
>   waiting to lock monitor 0x000000005f1f6530 (object 0x0000000731902200, a java.lang.Object),
>   which is held by "catalina-exec-710"
> "catalina-exec-710":
>   waiting to lock monitor 0x00002aaab9a05bd0 (object 0x00000007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
>   which is held by "catalina-exec-29-EventThread"
> "catalina-exec-29-EventThread":
>   waiting to lock monitor 0x000000005f9f0af0 (object 0x0000000732a9c7e0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
>   which is held by "catalina-exec-710"
> Java stack information for the threads listed above:
> ===================================================
> "catalina-exec-800":
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:943)
>         - waiting to lock <0x0000000731902200> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> "catalina-exec-710":
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:599)
>         - waiting to lock <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:158)
>         - locked <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
>         at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
>         at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
>         - locked <0x0000000731902200> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> "catalina-exec-29-EventThread":
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.stop(ZooKeeperNodeTracker.java:98)
>         - waiting to lock <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:604)
>         - locked <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:374)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271)
>         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
>         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
> Found 1 deadlock.
> From the source code , the reason for this problem is doing ZooKeeperNodeTracker.getData that lead to KeeperException. And try to resetZookeeperTracker. At the same time, ClientCnxn.EventThread  also do resetZookeeperTracker ,too. Because of getData have already held the lock of  ZooKeeperNodeTracke , that lead to the order of the lock two threads to obtain does not accord. So deadlock happened.
> In order to avoid the problem, we can add if reseting condition in abortable.abort()
> See the patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured

Posted by "Ted Yu (JIRA)" <ji...@apache.org>.

    [ https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508779#comment-13508779 ] 

Ted Yu commented on HBASE-7259:
-------------------------------

The latest patch was generated from 0.94 branch.
Please check out trunk repo: http://svn.apache.org/repos/asf/hbase/trunk
And attach patch based on trunk.

Normally the patch filename contains the JIRA number (and optionally short description).

Thanks
                
> Deadlock in HBaseClient when KeeperException occured
> ----------------------------------------------------
>
>                 Key: HBASE-7259
>                 URL: https://issues.apache.org/jira/browse/HBASE-7259
>             Project: HBase
>          Issue Type: Bug
>          Components: Zookeeper
>    Affects Versions: 0.94.0, 0.94.1, 0.94.2
>            Reporter: liwei
>            Priority: Critical
>         Attachments: HConnectionManager.patch
>
>
> HBaseClient was running after a period of time, all of get operation became too slow.
> From the client logs I could see the following:
> 1. Unable to get data of znode /hbase/root-region-server
> java.lang.InterruptedException
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
>         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
>         at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
>         at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
>         at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
>         at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
>         at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> 2. Catalina.out found one Java-level deadlock:
> =============================
> "catalina-exec-800":
>   waiting to lock monitor 0x000000005f1f6530 (object 0x0000000731902200, a java.lang.Object),
>   which is held by "catalina-exec-710"
> "catalina-exec-710":
>   waiting to lock monitor 0x00002aaab9a05bd0 (object 0x00000007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
>   which is held by "catalina-exec-29-EventThread"
> "catalina-exec-29-EventThread":
>   waiting to lock monitor 0x000000005f9f0af0 (object 0x0000000732a9c7e0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
>   which is held by "catalina-exec-710"
> Java stack information for the threads listed above:
> ===================================================
> "catalina-exec-800":
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:943)
>         - waiting to lock <0x0000000731902200> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> "catalina-exec-710":
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:599)
>         - waiting to lock <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:158)
>         - locked <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
>         at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
>         at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
>         - locked <0x0000000731902200> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> "catalina-exec-29-EventThread":
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.stop(ZooKeeperNodeTracker.java:98)
>         - waiting to lock <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:604)
>         - locked <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:374)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271)
>         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
>         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
> Found 1 deadlock.
> From the source code , the reason for this problem is doing ZooKeeperNodeTracker.getData that lead to KeeperException. And try to resetZookeeperTracker. At the same time, ClientCnxn.EventThread  also do resetZookeeperTracker ,too. Because of getData have already held the lock of  ZooKeeperNodeTracke , that lead to the order of the lock two threads to obtain does not accord. So deadlock happened.
> In order to avoid the problem, we can add if reseting condition in abortable.abort()
> See the patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured

Posted by "liwei (JIRA)" <ji...@apache.org>.

     [ https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

liwei updated HBASE-7259:
-------------------------

    Attachment: HConnectionManager.patch
    
> Deadlock in HBaseClient when KeeperException occured
> ----------------------------------------------------
>
>                 Key: HBASE-7259
>                 URL: https://issues.apache.org/jira/browse/HBASE-7259
>             Project: HBase
>          Issue Type: Bug
>          Components: Zookeeper
>    Affects Versions: 0.94.0, 0.94.1, 0.94.2
>            Reporter: liwei
>            Priority: Critical
>         Attachments: HConnectionManager.patch
>
>
> HBaseClient was running after a period of time, all of get operation became too slow.
> From the client logs I could see the following:
> 1. Unable to get data of znode /hbase/root-region-server
> java.lang.InterruptedException
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
>         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
>         at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
>         at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
>         at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
>         at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
>         at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> 2. Catalina.out found one Java-level deadlock:
> =============================
> "catalina-exec-800":
>   waiting to lock monitor 0x000000005f1f6530 (object 0x0000000731902200, a java.lang.Object),
>   which is held by "catalina-exec-710"
> "catalina-exec-710":
>   waiting to lock monitor 0x00002aaab9a05bd0 (object 0x00000007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
>   which is held by "catalina-exec-29-EventThread"
> "catalina-exec-29-EventThread":
>   waiting to lock monitor 0x000000005f9f0af0 (object 0x0000000732a9c7e0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
>   which is held by "catalina-exec-710"
> Java stack information for the threads listed above:
> ===================================================
> "catalina-exec-800":
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:943)
>         - waiting to lock <0x0000000731902200> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> "catalina-exec-710":
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:599)
>         - waiting to lock <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:158)
>         - locked <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
>         at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
>         at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
>         - locked <0x0000000731902200> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> "catalina-exec-29-EventThread":
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.stop(ZooKeeperNodeTracker.java:98)
>         - waiting to lock <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:604)
>         - locked <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:374)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271)
>         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
>         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
> Found 1 deadlock.
> From the source code , the reason for this problem is doing ZooKeeperNodeTracker.getData that lead to KeeperException. And try to resetZookeeperTracker. At the same time, ClientCnxn.EventThread  also do resetZookeeperTracker ,too. Because of getData have already held the lock of  ZooKeeperNodeTracke , that lead to the order of the lock two threads to obtain does not accord. So deadlock happened.
> In order to avoid the problem, we can add if reseting condition in abortable.abort()
> See the patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured

Posted by "liwei (JIRA)" <ji...@apache.org>.

     [ https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

liwei updated HBASE-7259:
-------------------------

    Attachment: HBASE-7259-0.94.2.txt

from 0.94.2
                
> Deadlock in HBaseClient when KeeperException occured
> ----------------------------------------------------
>
>                 Key: HBASE-7259
>                 URL: https://issues.apache.org/jira/browse/HBASE-7259
>             Project: HBase
>          Issue Type: Bug
>          Components: Zookeeper
>    Affects Versions: 0.94.0, 0.94.1, 0.94.2
>            Reporter: liwei
>            Priority: Critical
>             Fix For: 0.94.4
>
>         Attachments: HBASE-7259-0.94.2.txt, HConnectionManager.patch
>
>
> HBaseClient was running after a period of time, all of get operation became too slow.
> From the client logs I could see the following:
> 1. Unable to get data of znode /hbase/root-region-server
> {code}
> java.lang.InterruptedException
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
>         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
>         at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
>         at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
>         at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
>         at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
>         at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> {code}
> 2. Catalina.out found one Java-level deadlock:
> {code}
> =============================
> "catalina-exec-800":
>   waiting to lock monitor 0x000000005f1f6530 (object 0x0000000731902200, a java.lang.Object),
>   which is held by "catalina-exec-710"
> "catalina-exec-710":
>   waiting to lock monitor 0x00002aaab9a05bd0 (object 0x00000007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
>   which is held by "catalina-exec-29-EventThread"
> "catalina-exec-29-EventThread":
>   waiting to lock monitor 0x000000005f9f0af0 (object 0x0000000732a9c7e0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
>   which is held by "catalina-exec-710"
> Java stack information for the threads listed above:
> ===================================================
> "catalina-exec-800":
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:943)
>         - waiting to lock <0x0000000731902200> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> "catalina-exec-710":
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:599)
>         - waiting to lock <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:158)
>         - locked <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
>         at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
>         at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
>         - locked <0x0000000731902200> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> "catalina-exec-29-EventThread":
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.stop(ZooKeeperNodeTracker.java:98)
>         - waiting to lock <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:604)
>         - locked <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:374)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271)
>         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
>         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
> Found 1 deadlock.
> {code}
> From the source code , the reason for this problem is doing ZooKeeperNodeTracker.getData that lead to KeeperException. And try to resetZookeeperTracker. At the same time, ClientCnxn.EventThread  also do resetZookeeperTracker ,too. Because of getData have already held the lock of  ZooKeeperNodeTracke , that lead to the order of the lock two threads to obtain does not accord. So deadlock happened.
> In order to avoid the problem, we can add if reseting condition in abortable.abort()
> See the patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured

Posted by "liwei (JIRA)" <ji...@apache.org>.

     [ https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

liwei updated HBASE-7259:
-------------------------

    Description: 
HBaseClient was running after a period of time, all of get operation became too slow.

>From the client logs I could see the following:

1. Unable to get data of znode /hbase/root-region-server
java.lang.InterruptedException
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
        at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
        at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
        at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
        at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
        at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)

2. jstack traces found one Java-level deadlock:

=============================

"catalina-exec-800":
  waiting to lock monitor 0x000000005f1f6530 (object 0x0000000731902200, a java.lang.Object),
  which is held by "catalina-exec-710"
"catalina-exec-710":
  waiting to lock monitor 0x00002aaab9a05bd0 (object 0x00000007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
  which is held by "catalina-exec-29-EventThread"
"catalina-exec-29-EventThread":
  waiting to lock monitor 0x000000005f9f0af0 (object 0x0000000732a9c7e0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
  which is held by "catalina-exec-710"
Java stack information for the threads listed above:

===================================================

"catalina-exec-800":
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:943)
        - waiting to lock <0x0000000731902200> (a java.lang.Object)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
        at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
        at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
        at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
"catalina-exec-710":
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:599)
        - waiting to lock <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:158)
        - locked <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
        at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
        at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
        - locked <0x0000000731902200> (a java.lang.Object)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
        at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
        at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
        at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
"catalina-exec-29-EventThread":
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.stop(ZooKeeperNodeTracker.java:98)
        - waiting to lock <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:604)
        - locked <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:374)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
Found 1 deadlock.

>From the source code , the reason for this problem is doing ZooKeeperNodeTracker.getData that lead to KeeperException. And try to resetZookeeperTracker. At the same time, ClientCnxn.EventThread  also do resetZookeeperTracker ,too. Because of getData have already held the lock of  ZooKeeperNodeTracke , that lead to the order of the lock two threads to obtain does not accord. So deadlock happened.

In order to avoid the problem, we can through reduce range of the lock of getData. 
See the patch with 0.94.0.

  was:
HBaseClient was running after a period of time, all of get operation became too slow.

>From the client logs I could see the following:

1. Unable to get data of znode /hbase/root-region-server
java.lang.InterruptedException
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
        at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
        at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
        at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
        at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
        at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)

2. jstack traces found one Java-level deadlock:

=============================

"catalina-exec-800":
  waiting to lock monitor 0x000000005f1f6530 (object 0x0000000731902200, a java.lang.Object),
  which is held by "catalina-exec-710"
"catalina-exec-710":
  waiting to lock monitor 0x00002aaab9a05bd0 (object 0x00000007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
  which is held by "catalina-exec-29-EventThread"
"catalina-exec-29-EventThread":
  waiting to lock monitor 0x000000005f9f0af0 (object 0x0000000732a9c7e0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
  which is held by "catalina-exec-710"
Java stack information for the threads listed above:

===================================================

"catalina-exec-800":
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:943)
        - waiting to lock <0x0000000731902200> (a java.lang.Object)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
        at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
        at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
        at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
"catalina-exec-710":
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:599)
        - waiting to lock <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:158)
        - locked <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
        at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
        at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
        - locked <0x0000000731902200> (a java.lang.Object)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
        at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
        at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
        at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
"catalina-exec-29-EventThread":
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.stop(ZooKeeperNodeTracker.java:98)
        - waiting to lock <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:604)
        - locked <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:374)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
Found 1 deadlock.

>From the source code , the reason for this problem is doing ZooKeeperNodeTracker.getData that has a KeeperException occured. And try to resetZookeeperTracker. At the same time, ClientCnxn.EventThread  also do resetZookeeperTracker ,too. Because of getData have already held the lock of  ZooKeeperNodeTracke , that lead to the order of the lock two threads to obtain does not accord. So deadlock happened.

In order to avoid the problem, we can through reduce range of the lock of getData. 
See the patch with 0.94.0.

    
> Deadlock in HBaseClient when KeeperException occured
> ----------------------------------------------------
>
>                 Key: HBASE-7259
>                 URL: https://issues.apache.org/jira/browse/HBASE-7259
>             Project: HBase
>          Issue Type: Bug
>          Components: Zookeeper
>    Affects Versions: 0.94.0, 0.94.1, 0.94.2
>            Reporter: liwei
>            Priority: Critical
>         Attachments: ZookeeperNodeTracker.patch
>
>
> HBaseClient was running after a period of time, all of get operation became too slow.
> From the client logs I could see the following:
> 1. Unable to get data of znode /hbase/root-region-server
> java.lang.InterruptedException
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
>         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
>         at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
>         at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
>         at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
>         at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
>         at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> 2. jstack traces found one Java-level deadlock:
> =============================
> "catalina-exec-800":
>   waiting to lock monitor 0x000000005f1f6530 (object 0x0000000731902200, a java.lang.Object),
>   which is held by "catalina-exec-710"
> "catalina-exec-710":
>   waiting to lock monitor 0x00002aaab9a05bd0 (object 0x00000007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
>   which is held by "catalina-exec-29-EventThread"
> "catalina-exec-29-EventThread":
>   waiting to lock monitor 0x000000005f9f0af0 (object 0x0000000732a9c7e0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
>   which is held by "catalina-exec-710"
> Java stack information for the threads listed above:
> ===================================================
> "catalina-exec-800":
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:943)
>         - waiting to lock <0x0000000731902200> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> "catalina-exec-710":
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:599)
>         - waiting to lock <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:158)
>         - locked <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
>         at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
>         at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
>         - locked <0x0000000731902200> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> "catalina-exec-29-EventThread":
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.stop(ZooKeeperNodeTracker.java:98)
>         - waiting to lock <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:604)
>         - locked <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:374)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271)
>         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
>         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
> Found 1 deadlock.
> From the source code , the reason for this problem is doing ZooKeeperNodeTracker.getData that lead to KeeperException. And try to resetZookeeperTracker. At the same time, ClientCnxn.EventThread  also do resetZookeeperTracker ,too. Because of getData have already held the lock of  ZooKeeperNodeTracke , that lead to the order of the lock two threads to obtain does not accord. So deadlock happened.
> In order to avoid the problem, we can through reduce range of the lock of getData. 
> See the patch with 0.94.0.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured

Posted by "liwei (JIRA)" <ji...@apache.org>.

     [ https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

liwei updated HBASE-7259:
-------------------------

    Attachment:     (was: HConnectionManager.patch)
    
> Deadlock in HBaseClient when KeeperException occured
> ----------------------------------------------------
>
>                 Key: HBASE-7259
>                 URL: https://issues.apache.org/jira/browse/HBASE-7259
>             Project: HBase
>          Issue Type: Bug
>          Components: Zookeeper
>    Affects Versions: 0.94.0, 0.94.1, 0.94.2
>            Reporter: liwei
>            Priority: Critical
>             Fix For: 0.94.4
>
>         Attachments: HBASE-7259-0.94.2.txt
>
>
> HBaseClient was running after a period of time, all of get operation became too slow.
> From the client logs I could see the following:
> 1. Unable to get data of znode /hbase/root-region-server
> {code}
> java.lang.InterruptedException
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
>         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
>         at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
>         at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
>         at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
>         at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
>         at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> {code}
> 2. Catalina.out found one Java-level deadlock:
> {code}
> =============================
> "catalina-exec-800":
>   waiting to lock monitor 0x000000005f1f6530 (object 0x0000000731902200, a java.lang.Object),
>   which is held by "catalina-exec-710"
> "catalina-exec-710":
>   waiting to lock monitor 0x00002aaab9a05bd0 (object 0x00000007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
>   which is held by "catalina-exec-29-EventThread"
> "catalina-exec-29-EventThread":
>   waiting to lock monitor 0x000000005f9f0af0 (object 0x0000000732a9c7e0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
>   which is held by "catalina-exec-710"
> Java stack information for the threads listed above:
> ===================================================
> "catalina-exec-800":
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:943)
>         - waiting to lock <0x0000000731902200> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> "catalina-exec-710":
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:599)
>         - waiting to lock <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:158)
>         - locked <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
>         at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
>         at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
>         - locked <0x0000000731902200> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> "catalina-exec-29-EventThread":
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.stop(ZooKeeperNodeTracker.java:98)
>         - waiting to lock <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:604)
>         - locked <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:374)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271)
>         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
>         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
> Found 1 deadlock.
> {code}
> From the source code , the reason for this problem is doing ZooKeeperNodeTracker.getData that lead to KeeperException. And try to resetZookeeperTracker. At the same time, ClientCnxn.EventThread  also do resetZookeeperTracker ,too. Because of getData have already held the lock of  ZooKeeperNodeTracke , that lead to the order of the lock two threads to obtain does not accord. So deadlock happened.
> In order to avoid the problem, we can add if reseting condition in abortable.abort()
> See the patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured

Posted by "Ted Yu (JIRA)" <ji...@apache.org>.

    [ https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508871#comment-13508871 ] 

Ted Yu commented on HBASE-7259:
-------------------------------

I was talking about how to create the patch.
1. svn co http://svn.apache.org/repos/asf/hbase/trunk
2. cd trunk
3. modify HConnectionManager.java
4. at the root of trunk (your workspace) issue: svn diff > 7259-remove-deadlock-in-case-keeper-exception.txt
                
> Deadlock in HBaseClient when KeeperException occured
> ----------------------------------------------------
>
>                 Key: HBASE-7259
>                 URL: https://issues.apache.org/jira/browse/HBASE-7259
>             Project: HBase
>          Issue Type: Bug
>          Components: Zookeeper
>    Affects Versions: 0.94.0, 0.94.1, 0.94.2
>            Reporter: liwei
>            Priority: Critical
>         Attachments: HConnectionManager.patch
>
>
> HBaseClient was running after a period of time, all of get operation became too slow.
> From the client logs I could see the following:
> 1. Unable to get data of znode /hbase/root-region-server
> java.lang.InterruptedException
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
>         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
>         at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
>         at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
>         at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
>         at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
>         at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> 2. Catalina.out found one Java-level deadlock:
> =============================
> "catalina-exec-800":
>   waiting to lock monitor 0x000000005f1f6530 (object 0x0000000731902200, a java.lang.Object),
>   which is held by "catalina-exec-710"
> "catalina-exec-710":
>   waiting to lock monitor 0x00002aaab9a05bd0 (object 0x00000007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
>   which is held by "catalina-exec-29-EventThread"
> "catalina-exec-29-EventThread":
>   waiting to lock monitor 0x000000005f9f0af0 (object 0x0000000732a9c7e0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
>   which is held by "catalina-exec-710"
> Java stack information for the threads listed above:
> ===================================================
> "catalina-exec-800":
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:943)
>         - waiting to lock <0x0000000731902200> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> "catalina-exec-710":
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:599)
>         - waiting to lock <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:158)
>         - locked <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
>         at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
>         at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
>         - locked <0x0000000731902200> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> "catalina-exec-29-EventThread":
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.stop(ZooKeeperNodeTracker.java:98)
>         - waiting to lock <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:604)
>         - locked <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:374)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271)
>         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
>         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
> Found 1 deadlock.
> From the source code , the reason for this problem is doing ZooKeeperNodeTracker.getData that lead to KeeperException. And try to resetZookeeperTracker. At the same time, ClientCnxn.EventThread  also do resetZookeeperTracker ,too. Because of getData have already held the lock of  ZooKeeperNodeTracke , that lead to the order of the lock two threads to obtain does not accord. So deadlock happened.
> In order to avoid the problem, we can add if reseting condition in abortable.abort()
> See the patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured

Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.

     [ https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Lars Hofhansl updated HBASE-7259:
---------------------------------

    Fix Version/s: 0.94.4
    
> Deadlock in HBaseClient when KeeperException occured
> ----------------------------------------------------
>
>                 Key: HBASE-7259
>                 URL: https://issues.apache.org/jira/browse/HBASE-7259
>             Project: HBase
>          Issue Type: Bug
>          Components: Zookeeper
>    Affects Versions: 0.94.0, 0.94.1, 0.94.2
>            Reporter: liwei
>            Priority: Critical
>             Fix For: 0.94.4
>
>         Attachments: HConnectionManager.patch
>
>
> HBaseClient was running after a period of time, all of get operation became too slow.
> From the client logs I could see the following:
> 1. Unable to get data of znode /hbase/root-region-server
> {code}
> java.lang.InterruptedException
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
>         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
>         at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
>         at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
>         at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
>         at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
>         at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> {code}
> 2. Catalina.out found one Java-level deadlock:
> {code}
> =============================
> "catalina-exec-800":
>   waiting to lock monitor 0x000000005f1f6530 (object 0x0000000731902200, a java.lang.Object),
>   which is held by "catalina-exec-710"
> "catalina-exec-710":
>   waiting to lock monitor 0x00002aaab9a05bd0 (object 0x00000007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
>   which is held by "catalina-exec-29-EventThread"
> "catalina-exec-29-EventThread":
>   waiting to lock monitor 0x000000005f9f0af0 (object 0x0000000732a9c7e0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
>   which is held by "catalina-exec-710"
> Java stack information for the threads listed above:
> ===================================================
> "catalina-exec-800":
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:943)
>         - waiting to lock <0x0000000731902200> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> "catalina-exec-710":
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:599)
>         - waiting to lock <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:158)
>         - locked <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
>         at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
>         at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
>         - locked <0x0000000731902200> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> "catalina-exec-29-EventThread":
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.stop(ZooKeeperNodeTracker.java:98)
>         - waiting to lock <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:604)
>         - locked <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:374)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271)
>         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
>         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
> Found 1 deadlock.
> {code}
> From the source code , the reason for this problem is doing ZooKeeperNodeTracker.getData that lead to KeeperException. And try to resetZookeeperTracker. At the same time, ClientCnxn.EventThread  also do resetZookeeperTracker ,too. Because of getData have already held the lock of  ZooKeeperNodeTracke , that lead to the order of the lock two threads to obtain does not accord. So deadlock happened.
> In order to avoid the problem, we can add if reseting condition in abortable.abort()
> See the patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured

Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.

    [ https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508884#comment-13508884 ] 

Lars Hofhansl commented on HBASE-7259:
--------------------------------------

Patch looks good.
                
> Deadlock in HBaseClient when KeeperException occured
> ----------------------------------------------------
>
>                 Key: HBASE-7259
>                 URL: https://issues.apache.org/jira/browse/HBASE-7259
>             Project: HBase
>          Issue Type: Bug
>          Components: Zookeeper
>    Affects Versions: 0.94.0, 0.94.1, 0.94.2
>            Reporter: liwei
>            Priority: Critical
>         Attachments: HConnectionManager.patch
>
>
> HBaseClient was running after a period of time, all of get operation became too slow.
> From the client logs I could see the following:
> 1. Unable to get data of znode /hbase/root-region-server
> {code}
> java.lang.InterruptedException
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
>         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
>         at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
>         at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
>         at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
>         at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
>         at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> {code}
> 2. Catalina.out found one Java-level deadlock:
> {code}
> =============================
> "catalina-exec-800":
>   waiting to lock monitor 0x000000005f1f6530 (object 0x0000000731902200, a java.lang.Object),
>   which is held by "catalina-exec-710"
> "catalina-exec-710":
>   waiting to lock monitor 0x00002aaab9a05bd0 (object 0x00000007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
>   which is held by "catalina-exec-29-EventThread"
> "catalina-exec-29-EventThread":
>   waiting to lock monitor 0x000000005f9f0af0 (object 0x0000000732a9c7e0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
>   which is held by "catalina-exec-710"
> Java stack information for the threads listed above:
> ===================================================
> "catalina-exec-800":
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:943)
>         - waiting to lock <0x0000000731902200> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> "catalina-exec-710":
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:599)
>         - waiting to lock <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:158)
>         - locked <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
>         at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
>         at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
>         - locked <0x0000000731902200> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> "catalina-exec-29-EventThread":
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.stop(ZooKeeperNodeTracker.java:98)
>         - waiting to lock <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:604)
>         - locked <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:374)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271)
>         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
>         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
> Found 1 deadlock.
> {code}
> From the source code , the reason for this problem is doing ZooKeeperNodeTracker.getData that lead to KeeperException. And try to resetZookeeperTracker. At the same time, ClientCnxn.EventThread  also do resetZookeeperTracker ,too. Because of getData have already held the lock of  ZooKeeperNodeTracke , that lead to the order of the lock two threads to obtain does not accord. So deadlock happened.
> In order to avoid the problem, we can add if reseting condition in abortable.abort()
> See the patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured

Posted by "liwei (JIRA)" <ji...@apache.org>.

     [ https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

liwei updated HBASE-7259:
-------------------------

    Description: 
HBaseClient was running after a period of time, all of get operation became too slow.

>From the client logs I could see the following:

1. Unable to get data of znode /hbase/root-region-server
java.lang.InterruptedException
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
        at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
        at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
        at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
        at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
        at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)

2. Catalina.out found one Java-level deadlock:

=============================

"catalina-exec-800":
  waiting to lock monitor 0x000000005f1f6530 (object 0x0000000731902200, a java.lang.Object),
  which is held by "catalina-exec-710"
"catalina-exec-710":
  waiting to lock monitor 0x00002aaab9a05bd0 (object 0x00000007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
  which is held by "catalina-exec-29-EventThread"
"catalina-exec-29-EventThread":
  waiting to lock monitor 0x000000005f9f0af0 (object 0x0000000732a9c7e0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
  which is held by "catalina-exec-710"
Java stack information for the threads listed above:

===================================================

"catalina-exec-800":
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:943)
        - waiting to lock <0x0000000731902200> (a java.lang.Object)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
        at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
        at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
        at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
"catalina-exec-710":
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:599)
        - waiting to lock <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:158)
        - locked <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
        at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
        at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
        - locked <0x0000000731902200> (a java.lang.Object)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
        at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
        at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
        at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
"catalina-exec-29-EventThread":
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.stop(ZooKeeperNodeTracker.java:98)
        - waiting to lock <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:604)
        - locked <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:374)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
Found 1 deadlock.

>From the source code , the reason for this problem is doing ZooKeeperNodeTracker.getData that lead to KeeperException. And try to resetZookeeperTracker. At the same time, ClientCnxn.EventThread  also do resetZookeeperTracker ,too. Because of getData have already held the lock of  ZooKeeperNodeTracke , that lead to the order of the lock two threads to obtain does not accord. So deadlock happened.

In order to avoid the problem, we can add if reseting condition in abortable.abort()
See the patch.

  was:
HBaseClient was running after a period of time, all of get operation became too slow.

>From the client logs I could see the following:

1. Unable to get data of znode /hbase/root-region-server
java.lang.InterruptedException
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
        at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
        at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
        at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
        at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
        at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)

2. Catalina.out found one Java-level deadlock:

=============================

"catalina-exec-800":
  waiting to lock monitor 0x000000005f1f6530 (object 0x0000000731902200, a java.lang.Object),
  which is held by "catalina-exec-710"
"catalina-exec-710":
  waiting to lock monitor 0x00002aaab9a05bd0 (object 0x00000007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
  which is held by "catalina-exec-29-EventThread"
"catalina-exec-29-EventThread":
  waiting to lock monitor 0x000000005f9f0af0 (object 0x0000000732a9c7e0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
  which is held by "catalina-exec-710"
Java stack information for the threads listed above:

===================================================

"catalina-exec-800":
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:943)
        - waiting to lock <0x0000000731902200> (a java.lang.Object)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
        at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
        at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
        at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
"catalina-exec-710":
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:599)
        - waiting to lock <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:158)
        - locked <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
        at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
        at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
        - locked <0x0000000731902200> (a java.lang.Object)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
        at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
        at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
        at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
"catalina-exec-29-EventThread":
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.stop(ZooKeeperNodeTracker.java:98)
        - waiting to lock <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:604)
        - locked <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:374)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
Found 1 deadlock.

>From the source code , the reason for this problem is doing ZooKeeperNodeTracker.getData that lead to KeeperException. And try to resetZookeeperTracker. At the same time, ClientCnxn.EventThread  also do resetZookeeperTracker ,too. Because of getData have already held the lock of  ZooKeeperNodeTracke , that lead to the order of the lock two threads to obtain does not accord. So deadlock happened.

In order to avoid the problem, we can through reduce range of the lock of getData. 
See the patch.

    
> Deadlock in HBaseClient when KeeperException occured
> ----------------------------------------------------
>
>                 Key: HBASE-7259
>                 URL: https://issues.apache.org/jira/browse/HBASE-7259
>             Project: HBase
>          Issue Type: Bug
>          Components: Zookeeper
>    Affects Versions: 0.94.0, 0.94.1, 0.94.2
>            Reporter: liwei
>            Priority: Critical
>
> HBaseClient was running after a period of time, all of get operation became too slow.
> From the client logs I could see the following:
> 1. Unable to get data of znode /hbase/root-region-server
> java.lang.InterruptedException
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
>         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
>         at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
>         at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
>         at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
>         at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
>         at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> 2. Catalina.out found one Java-level deadlock:
> =============================
> "catalina-exec-800":
>   waiting to lock monitor 0x000000005f1f6530 (object 0x0000000731902200, a java.lang.Object),
>   which is held by "catalina-exec-710"
> "catalina-exec-710":
>   waiting to lock monitor 0x00002aaab9a05bd0 (object 0x00000007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
>   which is held by "catalina-exec-29-EventThread"
> "catalina-exec-29-EventThread":
>   waiting to lock monitor 0x000000005f9f0af0 (object 0x0000000732a9c7e0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
>   which is held by "catalina-exec-710"
> Java stack information for the threads listed above:
> ===================================================
> "catalina-exec-800":
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:943)
>         - waiting to lock <0x0000000731902200> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> "catalina-exec-710":
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:599)
>         - waiting to lock <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:158)
>         - locked <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
>         at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
>         at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
>         - locked <0x0000000731902200> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> "catalina-exec-29-EventThread":
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.stop(ZooKeeperNodeTracker.java:98)
>         - waiting to lock <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:604)
>         - locked <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:374)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271)
>         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
>         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
> Found 1 deadlock.
> From the source code , the reason for this problem is doing ZooKeeperNodeTracker.getData that lead to KeeperException. And try to resetZookeeperTracker. At the same time, ClientCnxn.EventThread  also do resetZookeeperTracker ,too. Because of getData have already held the lock of  ZooKeeperNodeTracke , that lead to the order of the lock two threads to obtain does not accord. So deadlock happened.
> In order to avoid the problem, we can add if reseting condition in abortable.abort()
> See the patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured

Posted by "liwei (JIRA)" <ji...@apache.org>.

    [ https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508863#comment-13508863 ] 

liwei commented on HBASE-7259:
------------------------------

I don't have privilege to commit.
How to do it? 
                
> Deadlock in HBaseClient when KeeperException occured
> ----------------------------------------------------
>
>                 Key: HBASE-7259
>                 URL: https://issues.apache.org/jira/browse/HBASE-7259
>             Project: HBase
>          Issue Type: Bug
>          Components: Zookeeper
>    Affects Versions: 0.94.0, 0.94.1, 0.94.2
>            Reporter: liwei
>            Priority: Critical
>         Attachments: HConnectionManager.patch
>
>
> HBaseClient was running after a period of time, all of get operation became too slow.
> From the client logs I could see the following:
> 1. Unable to get data of znode /hbase/root-region-server
> java.lang.InterruptedException
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
>         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
>         at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
>         at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
>         at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
>         at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
>         at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> 2. Catalina.out found one Java-level deadlock:
> =============================
> "catalina-exec-800":
>   waiting to lock monitor 0x000000005f1f6530 (object 0x0000000731902200, a java.lang.Object),
>   which is held by "catalina-exec-710"
> "catalina-exec-710":
>   waiting to lock monitor 0x00002aaab9a05bd0 (object 0x00000007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
>   which is held by "catalina-exec-29-EventThread"
> "catalina-exec-29-EventThread":
>   waiting to lock monitor 0x000000005f9f0af0 (object 0x0000000732a9c7e0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
>   which is held by "catalina-exec-710"
> Java stack information for the threads listed above:
> ===================================================
> "catalina-exec-800":
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:943)
>         - waiting to lock <0x0000000731902200> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> "catalina-exec-710":
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:599)
>         - waiting to lock <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:158)
>         - locked <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
>         at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
>         at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
>         - locked <0x0000000731902200> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> "catalina-exec-29-EventThread":
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.stop(ZooKeeperNodeTracker.java:98)
>         - waiting to lock <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:604)
>         - locked <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:374)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271)
>         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
>         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
> Found 1 deadlock.
> From the source code , the reason for this problem is doing ZooKeeperNodeTracker.getData that lead to KeeperException. And try to resetZookeeperTracker. At the same time, ClientCnxn.EventThread  also do resetZookeeperTracker ,too. Because of getData have already held the lock of  ZooKeeperNodeTracke , that lead to the order of the lock two threads to obtain does not accord. So deadlock happened.
> In order to avoid the problem, we can add if reseting condition in abortable.abort()
> See the patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured

Posted by "liwei (JIRA)" <ji...@apache.org>.

     [ https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

liwei updated HBASE-7259:
-------------------------

    Attachment: HConnectionManager.patch
    
> Deadlock in HBaseClient when KeeperException occured
> ----------------------------------------------------
>
>                 Key: HBASE-7259
>                 URL: https://issues.apache.org/jira/browse/HBASE-7259
>             Project: HBase
>          Issue Type: Bug
>          Components: Zookeeper
>    Affects Versions: 0.94.0, 0.94.1, 0.94.2
>            Reporter: liwei
>            Priority: Critical
>         Attachments: HConnectionManager.patch
>
>
> HBaseClient was running after a period of time, all of get operation became too slow.
> From the client logs I could see the following:
> 1. Unable to get data of znode /hbase/root-region-server
> java.lang.InterruptedException
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
>         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
>         at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
>         at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
>         at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
>         at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
>         at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> 2. Catalina.out found one Java-level deadlock:
> =============================
> "catalina-exec-800":
>   waiting to lock monitor 0x000000005f1f6530 (object 0x0000000731902200, a java.lang.Object),
>   which is held by "catalina-exec-710"
> "catalina-exec-710":
>   waiting to lock monitor 0x00002aaab9a05bd0 (object 0x00000007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
>   which is held by "catalina-exec-29-EventThread"
> "catalina-exec-29-EventThread":
>   waiting to lock monitor 0x000000005f9f0af0 (object 0x0000000732a9c7e0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
>   which is held by "catalina-exec-710"
> Java stack information for the threads listed above:
> ===================================================
> "catalina-exec-800":
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:943)
>         - waiting to lock <0x0000000731902200> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> "catalina-exec-710":
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:599)
>         - waiting to lock <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:158)
>         - locked <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
>         at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
>         at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
>         - locked <0x0000000731902200> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> "catalina-exec-29-EventThread":
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.stop(ZooKeeperNodeTracker.java:98)
>         - waiting to lock <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:604)
>         - locked <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:374)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271)
>         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
>         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
> Found 1 deadlock.
> From the source code , the reason for this problem is doing ZooKeeperNodeTracker.getData that lead to KeeperException. And try to resetZookeeperTracker. At the same time, ClientCnxn.EventThread  also do resetZookeeperTracker ,too. Because of getData have already held the lock of  ZooKeeperNodeTracke , that lead to the order of the lock two threads to obtain does not accord. So deadlock happened.
> In order to avoid the problem, we can add if reseting condition in abortable.abort()
> See the patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured

Posted by "liwei (JIRA)" <ji...@apache.org>.

     [ https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

liwei updated HBASE-7259:
-------------------------

    Description: 
HBaseClient was running after a period of time, all of get operation became too slow.

>From the client logs I could see the following:

1. Unable to get data of znode /hbase/root-region-server
java.lang.InterruptedException
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
        at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
        at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
        at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
        at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
        at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)

2. Catalina.out found one Java-level deadlock:

=============================

"catalina-exec-800":
  waiting to lock monitor 0x000000005f1f6530 (object 0x0000000731902200, a java.lang.Object),
  which is held by "catalina-exec-710"
"catalina-exec-710":
  waiting to lock monitor 0x00002aaab9a05bd0 (object 0x00000007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
  which is held by "catalina-exec-29-EventThread"
"catalina-exec-29-EventThread":
  waiting to lock monitor 0x000000005f9f0af0 (object 0x0000000732a9c7e0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
  which is held by "catalina-exec-710"
Java stack information for the threads listed above:

===================================================

"catalina-exec-800":
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:943)
        - waiting to lock <0x0000000731902200> (a java.lang.Object)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
        at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
        at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
        at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
"catalina-exec-710":
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:599)
        - waiting to lock <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:158)
        - locked <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
        at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
        at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
        - locked <0x0000000731902200> (a java.lang.Object)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
        at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
        at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
        at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
"catalina-exec-29-EventThread":
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.stop(ZooKeeperNodeTracker.java:98)
        - waiting to lock <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:604)
        - locked <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:374)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
Found 1 deadlock.

>From the source code , the reason for this problem is doing ZooKeeperNodeTracker.getData that lead to KeeperException. And try to resetZookeeperTracker. At the same time, ClientCnxn.EventThread  also do resetZookeeperTracker ,too. Because of getData have already held the lock of  ZooKeeperNodeTracke , that lead to the order of the lock two threads to obtain does not accord. So deadlock happened.

In order to avoid the problem, we can through reduce range of the lock of getData. 
See the patch.

  was:
HBaseClient was running after a period of time, all of get operation became too slow.

>From the client logs I could see the following:

1. Unable to get data of znode /hbase/root-region-server
java.lang.InterruptedException
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
        at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
        at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
        at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
        at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
        at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)

2. Catalina.out found one Java-level deadlock:

=============================

"catalina-exec-800":
  waiting to lock monitor 0x000000005f1f6530 (object 0x0000000731902200, a java.lang.Object),
  which is held by "catalina-exec-710"
"catalina-exec-710":
  waiting to lock monitor 0x00002aaab9a05bd0 (object 0x00000007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
  which is held by "catalina-exec-29-EventThread"
"catalina-exec-29-EventThread":
  waiting to lock monitor 0x000000005f9f0af0 (object 0x0000000732a9c7e0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
  which is held by "catalina-exec-710"
Java stack information for the threads listed above:

===================================================

"catalina-exec-800":
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:943)
        - waiting to lock <0x0000000731902200> (a java.lang.Object)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
        at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
        at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
        at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
"catalina-exec-710":
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:599)
        - waiting to lock <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:158)
        - locked <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
        at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
        at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
        at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
        - locked <0x0000000731902200> (a java.lang.Object)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
        at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
        at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
        at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
"catalina-exec-29-EventThread":
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.stop(ZooKeeperNodeTracker.java:98)
        - waiting to lock <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:604)
        - locked <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:374)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
Found 1 deadlock.

>From the source code , the reason for this problem is doing ZooKeeperNodeTracker.getData that lead to KeeperException. And try to resetZookeeperTracker. At the same time, ClientCnxn.EventThread  also do resetZookeeperTracker ,too. Because of getData have already held the lock of  ZooKeeperNodeTracke , that lead to the order of the lock two threads to obtain does not accord. So deadlock happened.

In order to avoid the problem, we can through reduce range of the lock of getData. 
See the patch with 0.94.0.

    
> Deadlock in HBaseClient when KeeperException occured
> ----------------------------------------------------
>
>                 Key: HBASE-7259
>                 URL: https://issues.apache.org/jira/browse/HBASE-7259
>             Project: HBase
>          Issue Type: Bug
>          Components: Zookeeper
>    Affects Versions: 0.94.0, 0.94.1, 0.94.2
>            Reporter: liwei
>            Priority: Critical
>         Attachments: ZookeeperNodeTracker.patch
>
>
> HBaseClient was running after a period of time, all of get operation became too slow.
> From the client logs I could see the following:
> 1. Unable to get data of znode /hbase/root-region-server
> java.lang.InterruptedException
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
>         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
>         at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
>         at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
>         at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
>         at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
>         at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> 2. Catalina.out found one Java-level deadlock:
> =============================
> "catalina-exec-800":
>   waiting to lock monitor 0x000000005f1f6530 (object 0x0000000731902200, a java.lang.Object),
>   which is held by "catalina-exec-710"
> "catalina-exec-710":
>   waiting to lock monitor 0x00002aaab9a05bd0 (object 0x00000007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
>   which is held by "catalina-exec-29-EventThread"
> "catalina-exec-29-EventThread":
>   waiting to lock monitor 0x000000005f9f0af0 (object 0x0000000732a9c7e0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
>   which is held by "catalina-exec-710"
> Java stack information for the threads listed above:
> ===================================================
> "catalina-exec-800":
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:943)
>         - waiting to lock <0x0000000731902200> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> "catalina-exec-710":
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:599)
>         - waiting to lock <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:158)
>         - locked <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
>         at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>         at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>         at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
>         at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>         at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
>         - locked <0x0000000731902200> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
>         at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
>         at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> "catalina-exec-29-EventThread":
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.stop(ZooKeeperNodeTracker.java:98)
>         - waiting to lock <0x0000000732a9c7e0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:604)
>         - locked <0x00000007321f8708> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1660)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:374)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271)
>         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
>         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
> Found 1 deadlock.
> From the source code , the reason for this problem is doing ZooKeeperNodeTracker.getData that lead to KeeperException. And try to resetZookeeperTracker. At the same time, ClientCnxn.EventThread  also do resetZookeeperTracker ,too. Because of getData have already held the lock of  ZooKeeperNodeTracke , that lead to the order of the lock two threads to obtain does not accord. So deadlock happened.
> In order to avoid the problem, we can through reduce range of the lock of getData. 
> See the patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira