You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "zhoushuaifeng (JIRA)" <ji...@apache.org> on 2011/04/29 09:40:03 UTC

[jira] [Created] (HBASE-3830) dumb JVM figure out a deadlock on hbase

dumb JVM figure out a deadlock on hbase
---------------------------------------

                 Key: HBASE-3830
                 URL: https://issues.apache.org/jira/browse/HBASE-3830
             Project: HBase
          Issue Type: Bug
          Components: regionserver
    Affects Versions: 0.90.1
            Reporter: zhoushuaifeng


Found one Java-level deadlock:
=============================
"IPC Server handler 9 on 60020":
  waiting to lock monitor 0x00000000409f3908 (object 0x00007fe7cbacbd48, a org.apache.hadoop.hbase.regionserver.MemStoreFlusher),
  which is held by "IPC Server handler 7 on 60020"
"IPC Server handler 7 on 60020":
  waiting for ownable synchronizer 0x00007fe7cbb06228, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
  which is held by "regionserver60020.cacheFlusher"
"regionserver60020.cacheFlusher":
  waiting to lock monitor 0x00000000409f3908 (object 0x00007fe7cbacbd48, a org.apache.hadoop.hbase.regionserver.MemStoreFlusher),
  which is held by "IPC Server handler 7 on 60020"

Java stack information for the threads listed above:
===================================================
"IPC Server handler 9 on 60020":
                at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(MemStoreFlusher.java)
                - waiting to lock <0x00007fe7cbacbd48> (a org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
                at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:2558)
                at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
                at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
"IPC Server handler 7 on 60020":
                at sun.misc.Unsafe.$$YJP$$park(Native Method)
                - parking to wait for  <0x00007fe7cbb06228> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
                at sun.misc.Unsafe.park(Unsafe.java)
                at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
                at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
                at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
                at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(MemStoreFlusher.java:429)
                - locked <0x00007fe7cbacbd48> (a org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
                at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:2558)
                at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
                at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
"regionserver60020.cacheFlusher":
                at java.util.ResourceBundle.endLoading(ResourceBundle.java:1506)
                - waiting to lock <0x00007fe7cbacbd48> (a org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
                at java.util.ResourceBundle.findBundle(ResourceBundle.java:1379)
                at java.util.ResourceBundle.findBundle(ResourceBundle.java:1292)
                at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1234)
                at java.util.ResourceBundle.getBundle(ResourceBundle.java:832)
                at sun.util.resources.LocaleData$1.run(LocaleData.java:127)
                at java.security.AccessController.$$YJP$$doPrivileged(Native Method)
                at java.security.AccessController.doPrivileged(AccessController.java)
                at sun.util.resources.LocaleData.getBundle(LocaleData.java:125)
                at sun.util.resources.LocaleData.getTimeZoneNames(LocaleData.java:97)
                at sun.util.TimeZoneNameUtility.getBundle(TimeZoneNameUtility.java:115)
                at sun.util.TimeZoneNameUtility.retrieveDisplayNames(TimeZoneNameUtility.java:80)
                at java.util.TimeZone.getDisplayNames(TimeZone.java:399)
                at java.util.TimeZone.getDisplayName(TimeZone.java:350)
                at java.util.Date.toString(Date.java:1025)
                at java.lang.String.valueOf(String.java:2826)
                at java.lang.StringBuilder.append(StringBuilder.java:115)
                at org.apache.hadoop.hbase.regionserver.PriorityCompactionQueue$CompactionRequest.toString(PriorityCompactionQueue.java:114)
                at java.lang.String.valueOf(String.java:2826)
                at java.lang.StringBuilder.append(StringBuilder.java:115)
                at org.apache.hadoop.hbase.regionserver.PriorityCompactionQueue.addToRegionsInQueue(PriorityCompactionQueue.java:145)
                - locked <0x00007fe7ccabd258> (a java.util.HashMap)
                at org.apache.hadoop.hbase.regionserver.PriorityCompactionQueue.add(PriorityCompactionQueue.java:188)
                at org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompaction(CompactSplitThread.java:140)
                - locked <0x00007fe7cbaf08c8> (a org.apache.hadoop.hbase.regionserver.CompactSplitThread)
                at org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompaction(CompactSplitThread.java:118)
                - locked <0x00007fe7cbaf08c8> (a org.apache.hadoop.hbase.regionserver.CompactSplitThread)
                at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:387)
                at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:194)
                at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:214)

Found 1 deadlock.


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3830) MemStoreFlusher deadlock detected by JVM

Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13234090#comment-13234090 ] 

stack commented on HBASE-3830:
------------------------------

Because its doing ResourceBundle#findBundle?  Lets assume it is.
                
> MemStoreFlusher deadlock detected by JVM
> ----------------------------------------
>
>                 Key: HBASE-3830
>                 URL: https://issues.apache.org/jira/browse/HBASE-3830
>             Project: HBase
>          Issue Type: Bug
>          Components: regionserver
>    Affects Versions: 0.90.1
>            Reporter: zhoushuaifeng
>            Priority: Trivial
>
> Found one Java-level deadlock:
> =============================
> "IPC Server handler 9 on 60020":
>   waiting to lock monitor 0x00000000409f3908 (object 0x00007fe7cbacbd48, a org.apache.hadoop.hbase.regionserver.MemStoreFlusher),
>   which is held by "IPC Server handler 7 on 60020"
> "IPC Server handler 7 on 60020":
>   waiting for ownable synchronizer 0x00007fe7cbb06228, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
>   which is held by "regionserver60020.cacheFlusher"
> "regionserver60020.cacheFlusher":
>   waiting to lock monitor 0x00000000409f3908 (object 0x00007fe7cbacbd48, a org.apache.hadoop.hbase.regionserver.MemStoreFlusher),
>   which is held by "IPC Server handler 7 on 60020"
> Java stack information for the threads listed above:
> ===================================================
> "IPC Server handler 9 on 60020":
>                 at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(MemStoreFlusher.java)
>                 - waiting to lock <0x00007fe7cbacbd48> (a org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
>                 at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:2558)
>                 at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>                 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>                 at java.lang.reflect.Method.invoke(Method.java:597)
>                 at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>                 at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
> "IPC Server handler 7 on 60020":
>                 at sun.misc.Unsafe.$$YJP$$park(Native Method)
>                 - parking to wait for  <0x00007fe7cbb06228> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
>                 at sun.misc.Unsafe.park(Unsafe.java)
>                 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>                 at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
>                 at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
>                 at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
>                 at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
>                 at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
>                 at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(MemStoreFlusher.java:429)
>                 - locked <0x00007fe7cbacbd48> (a org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
>                 at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:2558)
>                 at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>                 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>                 at java.lang.reflect.Method.invoke(Method.java:597)
>                 at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>                 at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
> "regionserver60020.cacheFlusher":
>                 at java.util.ResourceBundle.endLoading(ResourceBundle.java:1506)
>                 - waiting to lock <0x00007fe7cbacbd48> (a org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
>                 at java.util.ResourceBundle.findBundle(ResourceBundle.java:1379)
>                 at java.util.ResourceBundle.findBundle(ResourceBundle.java:1292)
>                 at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1234)
>                 at java.util.ResourceBundle.getBundle(ResourceBundle.java:832)
>                 at sun.util.resources.LocaleData$1.run(LocaleData.java:127)
>                 at java.security.AccessController.$$YJP$$doPrivileged(Native Method)
>                 at java.security.AccessController.doPrivileged(AccessController.java)
>                 at sun.util.resources.LocaleData.getBundle(LocaleData.java:125)
>                 at sun.util.resources.LocaleData.getTimeZoneNames(LocaleData.java:97)
>                 at sun.util.TimeZoneNameUtility.getBundle(TimeZoneNameUtility.java:115)
>                 at sun.util.TimeZoneNameUtility.retrieveDisplayNames(TimeZoneNameUtility.java:80)
>                 at java.util.TimeZone.getDisplayNames(TimeZone.java:399)
>                 at java.util.TimeZone.getDisplayName(TimeZone.java:350)
>                 at java.util.Date.toString(Date.java:1025)
>                 at java.lang.String.valueOf(String.java:2826)
>                 at java.lang.StringBuilder.append(StringBuilder.java:115)
>                 at org.apache.hadoop.hbase.regionserver.PriorityCompactionQueue$CompactionRequest.toString(PriorityCompactionQueue.java:114)
>                 at java.lang.String.valueOf(String.java:2826)
>                 at java.lang.StringBuilder.append(StringBuilder.java:115)
>                 at org.apache.hadoop.hbase.regionserver.PriorityCompactionQueue.addToRegionsInQueue(PriorityCompactionQueue.java:145)
>                 - locked <0x00007fe7ccabd258> (a java.util.HashMap)
>                 at org.apache.hadoop.hbase.regionserver.PriorityCompactionQueue.add(PriorityCompactionQueue.java:188)
>                 at org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompaction(CompactSplitThread.java:140)
>                 - locked <0x00007fe7cbaf08c8> (a org.apache.hadoop.hbase.regionserver.CompactSplitThread)
>                 at org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompaction(CompactSplitThread.java:118)
>                 - locked <0x00007fe7cbaf08c8> (a org.apache.hadoop.hbase.regionserver.CompactSplitThread)
>                 at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:387)
>                 at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:194)
>                 at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:214)
> Found 1 deadlock.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3830) MemStoreFlusher deadlock detected by JVM

Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13234092#comment-13234092 ] 

stack commented on HBASE-3830:
------------------------------

Because its doing ResourceBundle#findBundle?  Lets assume it is.
                
> MemStoreFlusher deadlock detected by JVM
> ----------------------------------------
>
>                 Key: HBASE-3830
>                 URL: https://issues.apache.org/jira/browse/HBASE-3830
>             Project: HBase
>          Issue Type: Bug
>          Components: regionserver
>    Affects Versions: 0.90.1
>            Reporter: zhoushuaifeng
>            Priority: Trivial
>
> Found one Java-level deadlock:
> =============================
> "IPC Server handler 9 on 60020":
>   waiting to lock monitor 0x00000000409f3908 (object 0x00007fe7cbacbd48, a org.apache.hadoop.hbase.regionserver.MemStoreFlusher),
>   which is held by "IPC Server handler 7 on 60020"
> "IPC Server handler 7 on 60020":
>   waiting for ownable synchronizer 0x00007fe7cbb06228, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
>   which is held by "regionserver60020.cacheFlusher"
> "regionserver60020.cacheFlusher":
>   waiting to lock monitor 0x00000000409f3908 (object 0x00007fe7cbacbd48, a org.apache.hadoop.hbase.regionserver.MemStoreFlusher),
>   which is held by "IPC Server handler 7 on 60020"
> Java stack information for the threads listed above:
> ===================================================
> "IPC Server handler 9 on 60020":
>                 at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(MemStoreFlusher.java)
>                 - waiting to lock <0x00007fe7cbacbd48> (a org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
>                 at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:2558)
>                 at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>                 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>                 at java.lang.reflect.Method.invoke(Method.java:597)
>                 at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>                 at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
> "IPC Server handler 7 on 60020":
>                 at sun.misc.Unsafe.$$YJP$$park(Native Method)
>                 - parking to wait for  <0x00007fe7cbb06228> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
>                 at sun.misc.Unsafe.park(Unsafe.java)
>                 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>                 at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
>                 at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
>                 at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
>                 at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
>                 at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
>                 at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(MemStoreFlusher.java:429)
>                 - locked <0x00007fe7cbacbd48> (a org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
>                 at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:2558)
>                 at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>                 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>                 at java.lang.reflect.Method.invoke(Method.java:597)
>                 at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>                 at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
> "regionserver60020.cacheFlusher":
>                 at java.util.ResourceBundle.endLoading(ResourceBundle.java:1506)
>                 - waiting to lock <0x00007fe7cbacbd48> (a org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
>                 at java.util.ResourceBundle.findBundle(ResourceBundle.java:1379)
>                 at java.util.ResourceBundle.findBundle(ResourceBundle.java:1292)
>                 at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1234)
>                 at java.util.ResourceBundle.getBundle(ResourceBundle.java:832)
>                 at sun.util.resources.LocaleData$1.run(LocaleData.java:127)
>                 at java.security.AccessController.$$YJP$$doPrivileged(Native Method)
>                 at java.security.AccessController.doPrivileged(AccessController.java)
>                 at sun.util.resources.LocaleData.getBundle(LocaleData.java:125)
>                 at sun.util.resources.LocaleData.getTimeZoneNames(LocaleData.java:97)
>                 at sun.util.TimeZoneNameUtility.getBundle(TimeZoneNameUtility.java:115)
>                 at sun.util.TimeZoneNameUtility.retrieveDisplayNames(TimeZoneNameUtility.java:80)
>                 at java.util.TimeZone.getDisplayNames(TimeZone.java:399)
>                 at java.util.TimeZone.getDisplayName(TimeZone.java:350)
>                 at java.util.Date.toString(Date.java:1025)
>                 at java.lang.String.valueOf(String.java:2826)
>                 at java.lang.StringBuilder.append(StringBuilder.java:115)
>                 at org.apache.hadoop.hbase.regionserver.PriorityCompactionQueue$CompactionRequest.toString(PriorityCompactionQueue.java:114)
>                 at java.lang.String.valueOf(String.java:2826)
>                 at java.lang.StringBuilder.append(StringBuilder.java:115)
>                 at org.apache.hadoop.hbase.regionserver.PriorityCompactionQueue.addToRegionsInQueue(PriorityCompactionQueue.java:145)
>                 - locked <0x00007fe7ccabd258> (a java.util.HashMap)
>                 at org.apache.hadoop.hbase.regionserver.PriorityCompactionQueue.add(PriorityCompactionQueue.java:188)
>                 at org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompaction(CompactSplitThread.java:140)
>                 - locked <0x00007fe7cbaf08c8> (a org.apache.hadoop.hbase.regionserver.CompactSplitThread)
>                 at org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompaction(CompactSplitThread.java:118)
>                 - locked <0x00007fe7cbaf08c8> (a org.apache.hadoop.hbase.regionserver.CompactSplitThread)
>                 at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:387)
>                 at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:194)
>                 at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:214)
> Found 1 deadlock.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-3830) MemStoreFlusher deadlock detected by JVM

Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Purtell updated HBASE-3830:
----------------------------------

    Summary: MemStoreFlusher deadlock detected by JVM  (was: dumb JVM figure out a deadlock on hbase)

> MemStoreFlusher deadlock detected by JVM
> ----------------------------------------
>
>                 Key: HBASE-3830
>                 URL: https://issues.apache.org/jira/browse/HBASE-3830
>             Project: HBase
>          Issue Type: Bug
>          Components: regionserver
>    Affects Versions: 0.90.1
>            Reporter: zhoushuaifeng
>
> Found one Java-level deadlock:
> =============================
> "IPC Server handler 9 on 60020":
>   waiting to lock monitor 0x00000000409f3908 (object 0x00007fe7cbacbd48, a org.apache.hadoop.hbase.regionserver.MemStoreFlusher),
>   which is held by "IPC Server handler 7 on 60020"
> "IPC Server handler 7 on 60020":
>   waiting for ownable synchronizer 0x00007fe7cbb06228, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
>   which is held by "regionserver60020.cacheFlusher"
> "regionserver60020.cacheFlusher":
>   waiting to lock monitor 0x00000000409f3908 (object 0x00007fe7cbacbd48, a org.apache.hadoop.hbase.regionserver.MemStoreFlusher),
>   which is held by "IPC Server handler 7 on 60020"
> Java stack information for the threads listed above:
> ===================================================
> "IPC Server handler 9 on 60020":
>                 at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(MemStoreFlusher.java)
>                 - waiting to lock <0x00007fe7cbacbd48> (a org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
>                 at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:2558)
>                 at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>                 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>                 at java.lang.reflect.Method.invoke(Method.java:597)
>                 at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>                 at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
> "IPC Server handler 7 on 60020":
>                 at sun.misc.Unsafe.$$YJP$$park(Native Method)
>                 - parking to wait for  <0x00007fe7cbb06228> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
>                 at sun.misc.Unsafe.park(Unsafe.java)
>                 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>                 at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
>                 at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
>                 at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
>                 at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
>                 at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
>                 at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(MemStoreFlusher.java:429)
>                 - locked <0x00007fe7cbacbd48> (a org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
>                 at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:2558)
>                 at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>                 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>                 at java.lang.reflect.Method.invoke(Method.java:597)
>                 at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>                 at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
> "regionserver60020.cacheFlusher":
>                 at java.util.ResourceBundle.endLoading(ResourceBundle.java:1506)
>                 - waiting to lock <0x00007fe7cbacbd48> (a org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
>                 at java.util.ResourceBundle.findBundle(ResourceBundle.java:1379)
>                 at java.util.ResourceBundle.findBundle(ResourceBundle.java:1292)
>                 at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1234)
>                 at java.util.ResourceBundle.getBundle(ResourceBundle.java:832)
>                 at sun.util.resources.LocaleData$1.run(LocaleData.java:127)
>                 at java.security.AccessController.$$YJP$$doPrivileged(Native Method)
>                 at java.security.AccessController.doPrivileged(AccessController.java)
>                 at sun.util.resources.LocaleData.getBundle(LocaleData.java:125)
>                 at sun.util.resources.LocaleData.getTimeZoneNames(LocaleData.java:97)
>                 at sun.util.TimeZoneNameUtility.getBundle(TimeZoneNameUtility.java:115)
>                 at sun.util.TimeZoneNameUtility.retrieveDisplayNames(TimeZoneNameUtility.java:80)
>                 at java.util.TimeZone.getDisplayNames(TimeZone.java:399)
>                 at java.util.TimeZone.getDisplayName(TimeZone.java:350)
>                 at java.util.Date.toString(Date.java:1025)
>                 at java.lang.String.valueOf(String.java:2826)
>                 at java.lang.StringBuilder.append(StringBuilder.java:115)
>                 at org.apache.hadoop.hbase.regionserver.PriorityCompactionQueue$CompactionRequest.toString(PriorityCompactionQueue.java:114)
>                 at java.lang.String.valueOf(String.java:2826)
>                 at java.lang.StringBuilder.append(StringBuilder.java:115)
>                 at org.apache.hadoop.hbase.regionserver.PriorityCompactionQueue.addToRegionsInQueue(PriorityCompactionQueue.java:145)
>                 - locked <0x00007fe7ccabd258> (a java.util.HashMap)
>                 at org.apache.hadoop.hbase.regionserver.PriorityCompactionQueue.add(PriorityCompactionQueue.java:188)
>                 at org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompaction(CompactSplitThread.java:140)
>                 - locked <0x00007fe7cbaf08c8> (a org.apache.hadoop.hbase.regionserver.CompactSplitThread)
>                 at org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompaction(CompactSplitThread.java:118)
>                 - locked <0x00007fe7cbaf08c8> (a org.apache.hadoop.hbase.regionserver.CompactSplitThread)
>                 at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:387)
>                 at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:194)
>                 at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:214)
> Found 1 deadlock.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3830) MemStoreFlusher deadlock detected by JVM

Posted by "Chris Waterson (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13234044#comment-13234044 ] 

Chris Waterson commented on HBASE-3830:
---------------------------------------

This looks a lot like HBASE-4367.  Is it a dup?
                
> MemStoreFlusher deadlock detected by JVM
> ----------------------------------------
>
>                 Key: HBASE-3830
>                 URL: https://issues.apache.org/jira/browse/HBASE-3830
>             Project: HBase
>          Issue Type: Bug
>          Components: regionserver
>    Affects Versions: 0.90.1
>            Reporter: zhoushuaifeng
>            Priority: Trivial
>
> Found one Java-level deadlock:
> =============================
> "IPC Server handler 9 on 60020":
>   waiting to lock monitor 0x00000000409f3908 (object 0x00007fe7cbacbd48, a org.apache.hadoop.hbase.regionserver.MemStoreFlusher),
>   which is held by "IPC Server handler 7 on 60020"
> "IPC Server handler 7 on 60020":
>   waiting for ownable synchronizer 0x00007fe7cbb06228, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
>   which is held by "regionserver60020.cacheFlusher"
> "regionserver60020.cacheFlusher":
>   waiting to lock monitor 0x00000000409f3908 (object 0x00007fe7cbacbd48, a org.apache.hadoop.hbase.regionserver.MemStoreFlusher),
>   which is held by "IPC Server handler 7 on 60020"
> Java stack information for the threads listed above:
> ===================================================
> "IPC Server handler 9 on 60020":
>                 at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(MemStoreFlusher.java)
>                 - waiting to lock <0x00007fe7cbacbd48> (a org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
>                 at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:2558)
>                 at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>                 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>                 at java.lang.reflect.Method.invoke(Method.java:597)
>                 at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>                 at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
> "IPC Server handler 7 on 60020":
>                 at sun.misc.Unsafe.$$YJP$$park(Native Method)
>                 - parking to wait for  <0x00007fe7cbb06228> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
>                 at sun.misc.Unsafe.park(Unsafe.java)
>                 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>                 at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
>                 at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
>                 at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
>                 at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
>                 at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
>                 at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(MemStoreFlusher.java:429)
>                 - locked <0x00007fe7cbacbd48> (a org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
>                 at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:2558)
>                 at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>                 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>                 at java.lang.reflect.Method.invoke(Method.java:597)
>                 at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>                 at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
> "regionserver60020.cacheFlusher":
>                 at java.util.ResourceBundle.endLoading(ResourceBundle.java:1506)
>                 - waiting to lock <0x00007fe7cbacbd48> (a org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
>                 at java.util.ResourceBundle.findBundle(ResourceBundle.java:1379)
>                 at java.util.ResourceBundle.findBundle(ResourceBundle.java:1292)
>                 at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1234)
>                 at java.util.ResourceBundle.getBundle(ResourceBundle.java:832)
>                 at sun.util.resources.LocaleData$1.run(LocaleData.java:127)
>                 at java.security.AccessController.$$YJP$$doPrivileged(Native Method)
>                 at java.security.AccessController.doPrivileged(AccessController.java)
>                 at sun.util.resources.LocaleData.getBundle(LocaleData.java:125)
>                 at sun.util.resources.LocaleData.getTimeZoneNames(LocaleData.java:97)
>                 at sun.util.TimeZoneNameUtility.getBundle(TimeZoneNameUtility.java:115)
>                 at sun.util.TimeZoneNameUtility.retrieveDisplayNames(TimeZoneNameUtility.java:80)
>                 at java.util.TimeZone.getDisplayNames(TimeZone.java:399)
>                 at java.util.TimeZone.getDisplayName(TimeZone.java:350)
>                 at java.util.Date.toString(Date.java:1025)
>                 at java.lang.String.valueOf(String.java:2826)
>                 at java.lang.StringBuilder.append(StringBuilder.java:115)
>                 at org.apache.hadoop.hbase.regionserver.PriorityCompactionQueue$CompactionRequest.toString(PriorityCompactionQueue.java:114)
>                 at java.lang.String.valueOf(String.java:2826)
>                 at java.lang.StringBuilder.append(StringBuilder.java:115)
>                 at org.apache.hadoop.hbase.regionserver.PriorityCompactionQueue.addToRegionsInQueue(PriorityCompactionQueue.java:145)
>                 - locked <0x00007fe7ccabd258> (a java.util.HashMap)
>                 at org.apache.hadoop.hbase.regionserver.PriorityCompactionQueue.add(PriorityCompactionQueue.java:188)
>                 at org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompaction(CompactSplitThread.java:140)
>                 - locked <0x00007fe7cbaf08c8> (a org.apache.hadoop.hbase.regionserver.CompactSplitThread)
>                 at org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompaction(CompactSplitThread.java:118)
>                 - locked <0x00007fe7cbaf08c8> (a org.apache.hadoop.hbase.regionserver.CompactSplitThread)
>                 at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:387)
>                 at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:194)
>                 at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:214)
> Found 1 deadlock.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-3830) MemStoreFlusher deadlock detected by JVM

Posted by "stack (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HBASE-3830:
-------------------------

    Priority: Trivial  (was: Major)

Zhou was running a profiler at the time.  The deadlock stack traces do not make sense.  We discussed it over here: http://search-hadoop.com/m/pzf8rNv8aT/0x00007fe7cbacbd48&subj=found+one+deadlock+on+hbase+

> MemStoreFlusher deadlock detected by JVM
> ----------------------------------------
>
>                 Key: HBASE-3830
>                 URL: https://issues.apache.org/jira/browse/HBASE-3830
>             Project: HBase
>          Issue Type: Bug
>          Components: regionserver
>    Affects Versions: 0.90.1
>            Reporter: zhoushuaifeng
>            Priority: Trivial
>
> Found one Java-level deadlock:
> =============================
> "IPC Server handler 9 on 60020":
>   waiting to lock monitor 0x00000000409f3908 (object 0x00007fe7cbacbd48, a org.apache.hadoop.hbase.regionserver.MemStoreFlusher),
>   which is held by "IPC Server handler 7 on 60020"
> "IPC Server handler 7 on 60020":
>   waiting for ownable synchronizer 0x00007fe7cbb06228, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
>   which is held by "regionserver60020.cacheFlusher"
> "regionserver60020.cacheFlusher":
>   waiting to lock monitor 0x00000000409f3908 (object 0x00007fe7cbacbd48, a org.apache.hadoop.hbase.regionserver.MemStoreFlusher),
>   which is held by "IPC Server handler 7 on 60020"
> Java stack information for the threads listed above:
> ===================================================
> "IPC Server handler 9 on 60020":
>                 at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(MemStoreFlusher.java)
>                 - waiting to lock <0x00007fe7cbacbd48> (a org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
>                 at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:2558)
>                 at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>                 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>                 at java.lang.reflect.Method.invoke(Method.java:597)
>                 at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>                 at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
> "IPC Server handler 7 on 60020":
>                 at sun.misc.Unsafe.$$YJP$$park(Native Method)
>                 - parking to wait for  <0x00007fe7cbb06228> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
>                 at sun.misc.Unsafe.park(Unsafe.java)
>                 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>                 at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
>                 at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
>                 at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
>                 at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
>                 at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
>                 at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(MemStoreFlusher.java:429)
>                 - locked <0x00007fe7cbacbd48> (a org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
>                 at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:2558)
>                 at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>                 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>                 at java.lang.reflect.Method.invoke(Method.java:597)
>                 at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>                 at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
> "regionserver60020.cacheFlusher":
>                 at java.util.ResourceBundle.endLoading(ResourceBundle.java:1506)
>                 - waiting to lock <0x00007fe7cbacbd48> (a org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
>                 at java.util.ResourceBundle.findBundle(ResourceBundle.java:1379)
>                 at java.util.ResourceBundle.findBundle(ResourceBundle.java:1292)
>                 at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1234)
>                 at java.util.ResourceBundle.getBundle(ResourceBundle.java:832)
>                 at sun.util.resources.LocaleData$1.run(LocaleData.java:127)
>                 at java.security.AccessController.$$YJP$$doPrivileged(Native Method)
>                 at java.security.AccessController.doPrivileged(AccessController.java)
>                 at sun.util.resources.LocaleData.getBundle(LocaleData.java:125)
>                 at sun.util.resources.LocaleData.getTimeZoneNames(LocaleData.java:97)
>                 at sun.util.TimeZoneNameUtility.getBundle(TimeZoneNameUtility.java:115)
>                 at sun.util.TimeZoneNameUtility.retrieveDisplayNames(TimeZoneNameUtility.java:80)
>                 at java.util.TimeZone.getDisplayNames(TimeZone.java:399)
>                 at java.util.TimeZone.getDisplayName(TimeZone.java:350)
>                 at java.util.Date.toString(Date.java:1025)
>                 at java.lang.String.valueOf(String.java:2826)
>                 at java.lang.StringBuilder.append(StringBuilder.java:115)
>                 at org.apache.hadoop.hbase.regionserver.PriorityCompactionQueue$CompactionRequest.toString(PriorityCompactionQueue.java:114)
>                 at java.lang.String.valueOf(String.java:2826)
>                 at java.lang.StringBuilder.append(StringBuilder.java:115)
>                 at org.apache.hadoop.hbase.regionserver.PriorityCompactionQueue.addToRegionsInQueue(PriorityCompactionQueue.java:145)
>                 - locked <0x00007fe7ccabd258> (a java.util.HashMap)
>                 at org.apache.hadoop.hbase.regionserver.PriorityCompactionQueue.add(PriorityCompactionQueue.java:188)
>                 at org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompaction(CompactSplitThread.java:140)
>                 - locked <0x00007fe7cbaf08c8> (a org.apache.hadoop.hbase.regionserver.CompactSplitThread)
>                 at org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompaction(CompactSplitThread.java:118)
>                 - locked <0x00007fe7cbaf08c8> (a org.apache.hadoop.hbase.regionserver.CompactSplitThread)
>                 at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:387)
>                 at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:194)
>                 at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:214)
> Found 1 deadlock.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-3830) MemStoreFlusher deadlock detected by JVM

Posted by "stack (Resolved) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack resolved HBASE-3830.
--------------------------

    Resolution: Duplicate

Marking dup of hbase-4367.  Thanks Chris.
                
> MemStoreFlusher deadlock detected by JVM
> ----------------------------------------
>
>                 Key: HBASE-3830
>                 URL: https://issues.apache.org/jira/browse/HBASE-3830
>             Project: HBase
>          Issue Type: Bug
>          Components: regionserver
>    Affects Versions: 0.90.1
>            Reporter: zhoushuaifeng
>            Priority: Trivial
>
> Found one Java-level deadlock:
> =============================
> "IPC Server handler 9 on 60020":
>   waiting to lock monitor 0x00000000409f3908 (object 0x00007fe7cbacbd48, a org.apache.hadoop.hbase.regionserver.MemStoreFlusher),
>   which is held by "IPC Server handler 7 on 60020"
> "IPC Server handler 7 on 60020":
>   waiting for ownable synchronizer 0x00007fe7cbb06228, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
>   which is held by "regionserver60020.cacheFlusher"
> "regionserver60020.cacheFlusher":
>   waiting to lock monitor 0x00000000409f3908 (object 0x00007fe7cbacbd48, a org.apache.hadoop.hbase.regionserver.MemStoreFlusher),
>   which is held by "IPC Server handler 7 on 60020"
> Java stack information for the threads listed above:
> ===================================================
> "IPC Server handler 9 on 60020":
>                 at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(MemStoreFlusher.java)
>                 - waiting to lock <0x00007fe7cbacbd48> (a org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
>                 at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:2558)
>                 at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>                 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>                 at java.lang.reflect.Method.invoke(Method.java:597)
>                 at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>                 at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
> "IPC Server handler 7 on 60020":
>                 at sun.misc.Unsafe.$$YJP$$park(Native Method)
>                 - parking to wait for  <0x00007fe7cbb06228> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
>                 at sun.misc.Unsafe.park(Unsafe.java)
>                 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>                 at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
>                 at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
>                 at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
>                 at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
>                 at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
>                 at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(MemStoreFlusher.java:429)
>                 - locked <0x00007fe7cbacbd48> (a org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
>                 at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:2558)
>                 at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>                 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>                 at java.lang.reflect.Method.invoke(Method.java:597)
>                 at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>                 at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
> "regionserver60020.cacheFlusher":
>                 at java.util.ResourceBundle.endLoading(ResourceBundle.java:1506)
>                 - waiting to lock <0x00007fe7cbacbd48> (a org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
>                 at java.util.ResourceBundle.findBundle(ResourceBundle.java:1379)
>                 at java.util.ResourceBundle.findBundle(ResourceBundle.java:1292)
>                 at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1234)
>                 at java.util.ResourceBundle.getBundle(ResourceBundle.java:832)
>                 at sun.util.resources.LocaleData$1.run(LocaleData.java:127)
>                 at java.security.AccessController.$$YJP$$doPrivileged(Native Method)
>                 at java.security.AccessController.doPrivileged(AccessController.java)
>                 at sun.util.resources.LocaleData.getBundle(LocaleData.java:125)
>                 at sun.util.resources.LocaleData.getTimeZoneNames(LocaleData.java:97)
>                 at sun.util.TimeZoneNameUtility.getBundle(TimeZoneNameUtility.java:115)
>                 at sun.util.TimeZoneNameUtility.retrieveDisplayNames(TimeZoneNameUtility.java:80)
>                 at java.util.TimeZone.getDisplayNames(TimeZone.java:399)
>                 at java.util.TimeZone.getDisplayName(TimeZone.java:350)
>                 at java.util.Date.toString(Date.java:1025)
>                 at java.lang.String.valueOf(String.java:2826)
>                 at java.lang.StringBuilder.append(StringBuilder.java:115)
>                 at org.apache.hadoop.hbase.regionserver.PriorityCompactionQueue$CompactionRequest.toString(PriorityCompactionQueue.java:114)
>                 at java.lang.String.valueOf(String.java:2826)
>                 at java.lang.StringBuilder.append(StringBuilder.java:115)
>                 at org.apache.hadoop.hbase.regionserver.PriorityCompactionQueue.addToRegionsInQueue(PriorityCompactionQueue.java:145)
>                 - locked <0x00007fe7ccabd258> (a java.util.HashMap)
>                 at org.apache.hadoop.hbase.regionserver.PriorityCompactionQueue.add(PriorityCompactionQueue.java:188)
>                 at org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompaction(CompactSplitThread.java:140)
>                 - locked <0x00007fe7cbaf08c8> (a org.apache.hadoop.hbase.regionserver.CompactSplitThread)
>                 at org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompaction(CompactSplitThread.java:118)
>                 - locked <0x00007fe7cbaf08c8> (a org.apache.hadoop.hbase.regionserver.CompactSplitThread)
>                 at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:387)
>                 at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:194)
>                 at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:214)
> Found 1 deadlock.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira