You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Namit Maheshwari (JIRA)" <ji...@apache.org> on 2015/07/21 10:41:05 UTC
[jira] [Reopened] (HBASE-12185) Deadlock in
HConnectionManager$HConnectionImplementation
[ https://issues.apache.org/jira/browse/HBASE-12185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Maheshwari reopened HBASE-12185:
--------------------------------------
This issue seems to be recurring. Adding relevant portions of stack trace.
Found one Java-level deadlock:
=============================
"qtp446544274-1565 - POST /operations/storeFile":
waiting to lock monitor 0x0000000046a92b78 (object 0x00000007801ed8a8, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
which is held by "main-EventThread"
"main-EventThread":
waiting to lock monitor 0x0000000041fd1f90 (object 0x00000007801f2c08, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
which is held by "qtp865211569-588"
"qtp865211569-588":
waiting to lock monitor 0x0000000046a92b78 (object 0x00000007801ed8a8, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
which is held by "main-EventThread"
Java stack information for the threads listed above:
===================================================
"qtp446544274-1565 - POST /operations/storeFile":
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:598)
- waiting to lock <0x00000007801ed8a8> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1681)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.checkIfBaseNodeAvailable(ZooKeeperNodeTracker.java:218)
at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.waitRootRegionLocation(RootRegionTracker.java:77)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:841)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:954)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:852)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:954)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:856)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:813)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:271)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:211)
at org.apache.hadoop.hbase.client.HTableFactory.createHTableInterface(HTableFactory.java:36)
at org.apache.hadoop.hbase.client.HTablePool.createHTable(HTablePool.java:265)
at org.apache.hadoop.hbase.client.HTablePool.findOrCreateTable(HTablePool.java:195)
at org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:174)
"main-EventThread":
at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.stop(ZooKeeperNodeTracker.java:98)
- waiting to lock <0x00000007801f2c08> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:603)
- locked <0x00000007801ed8a8> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1681)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:389)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:286)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
"qtp865211569-588":
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:598)
- waiting to lock <0x00000007801ed8a8> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1681)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:132)
- locked <0x00000007801f2c08> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.waitRootRegionLocation(RootRegionTracker.java:83)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:841)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:954)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:852)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:954)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:856)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:813)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:737)
at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:695)
at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:362)
Found one Java-level deadlock:
=============================
"qtp321887181-1129 - GET /operations/downloadFile/?TenantId=spslm&Namespace=spslm_seller_processed_feeds&Id=ult6s38dku2rwvrp_Bags+%26+Fashion+Accessories_CATALOG_REQE9DLSKKKP9_uploaded":
waiting to lock monitor 0x000000004171f298 (object 0x0000000780452848, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
which is held by "main-EventThread"
"main-EventThread":
waiting to lock monitor 0x0000000043d1e1a0 (object 0x00000007804aa068, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
which is held by "qtp321887181-1108 - POST /operations/storeFile"
"qtp321887181-1108 - POST /operations/storeFile":
waiting to lock monitor 0x000000004171f298 (object 0x0000000780452848, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
which is held by "main-EventThread"
Java stack information for the threads listed above:
===================================================
"qtp321887181-1129 - GET /operations/downloadFile/?TenantId=spslm&Namespace=spslm_seller_processed_feeds&Id=ult6s38dku2rwvrp_Bags+%26+Fashion+Accessories_CATALOG_REQE9DLSKKKP9_uploaded":
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:598)
- waiting to lock <0x0000000780452848> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1681)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.checkIfBaseNodeAvailable(ZooKeeperNodeTracker.java:218)
at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.waitRootRegionLocation(RootRegionTracker.java:77)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:841)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:954)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:852)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:954)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:856)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:813)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:737)
at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:695)
at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:362)
"main-EventThread":
at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.stop(ZooKeeperNodeTracker.java:98)
- waiting to lock <0x00000007804aa068> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:603)
- locked <0x0000000780452848> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1681)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:389)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:286)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
"qtp321887181-1108 - POST /operations/storeFile":
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:598)
- waiting to lock <0x0000000780452848> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1681)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:132)
- locked <0x00000007804aa068> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.waitRootRegionLocation(RootRegionTracker.java:83)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:841)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:954)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:852)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:954)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:856)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:813)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:737)
at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:695)
at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:362)
> Deadlock in HConnectionManager$HConnectionImplementation
> --------------------------------------------------------
>
> Key: HBASE-12185
> URL: https://issues.apache.org/jira/browse/HBASE-12185
> Project: HBase
> Issue Type: Bug
> Components: Client
> Affects Versions: 0.94.2
> Environment: CDH 4.2.0
> Reporter: Michael Tamm
> Priority: Critical
>
> Here you can see the relevant section of a thread dump:
> {noformat}
> Found one Java-level deadlock:
> =============================
> "AsyncSave-700512-Worker-EventThread":
> waiting to lock monitor 0x00007f8d90eecd20 (object 0x00000005c0a8e1d0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
> which is held by "AsyncSave-700546-Worker"
> "AsyncSave-700546-Worker":
> waiting to lock monitor 0x00007f8d90149700 (object 0x0000000571404180, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
> which is held by "AsyncSave-700512-Worker-EventThread"
> Java stack information for the threads listed above:
> ===================================================
> "AsyncSave-700512-Worker-EventThread":
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.stop(ZooKeeperNodeTracker.java:98)
> - waiting to lock <0x00000005c0a8e1d0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:603)
> - locked <0x0000000571404180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1681)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:389)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:286)
> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
> "AsyncSave-700546-Worker":
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:598)
> - waiting to lock <0x0000000571404180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1681)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:132)
> - locked <0x00000005c0a8e1d0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
> at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.waitRootRegionLocation(RootRegionTracker.java:83)
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:841)
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:954)
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:852)
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:954)
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:856)
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:813)
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1503)
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1388)
> at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:955)
> at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.flushCommits(HTablePool.java:449)
> at ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)