You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2008/08/12 01:54:44 UTC

[jira] Created: (HBASE-818) Deadlock running 'flushSomeRegions'

Deadlock running 'flushSomeRegions'
-----------------------------------

                 Key: HBASE-818
                 URL: https://issues.apache.org/jira/browse/HBASE-818
             Project: Hadoop HBase
          Issue Type: Bug
            Reporter: stack
            Priority: Blocker
             Fix For: 0.2.1, 0.3.0


Playing with MR uploading no a regionserver with 60+ regions, I ran into a deadlock:

{code}
Found one Java-level deadlock:
=============================
"IPC Server handler 19 on 60020":
  waiting to lock monitor 0x084be38c (object 0xb6f69a70, a org.apache.hadoop.hbase.regionserver.Flusher),
  which is held by "IPC Server handler 16 on 60020"
"IPC Server handler 16 on 60020":
  waiting to lock monitor 0x080f8dec (object 0xb73610c0, a org.apache.hadoop.hbase.regionserver.HRegion$WriteState),
  which is held by "IPC Server handler 2 on 60020"
"IPC Server handler 2 on 60020":
  waiting to lock monitor 0x086e8fe8 (object 0xb6f69cf0, a java.util.HashSet),
  which is held by "IPC Server handler 16 on 60020"

Java stack information for the threads listed above:
===================================================
"IPC Server handler 19 on 60020":
        at org.apache.hadoop.hbase.regionserver.Flusher.flushSomeRegions(Flusher.java:261)
        - waiting to lock <0xb6f69a70> (a org.apache.hadoop.hbase.regionserver.Flusher)
        at org.apache.hadoop.hbase.regionserver.Flusher.reclaimMemcacheMemory(Flusher.java:252)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1136)
        at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:623)
        at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
"IPC Server handler 16 on 60020":
        at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:948)
        - waiting to lock <0xb73610c0> (a org.apache.hadoop.hbase.regionserver.HRegion$WriteState)
        at org.apache.hadoop.hbase.regionserver.Flusher.flushRegion(Flusher.java:173)
        - locked <0xb6f69cf0> (a java.util.HashSet)
        at org.apache.hadoop.hbase.regionserver.Flusher.flushSomeRegions(Flusher.java:267)
        - locked <0xb6f69a70> (a org.apache.hadoop.hbase.regionserver.Flusher)
        at org.apache.hadoop.hbase.regionserver.Flusher.reclaimMemcacheMemory(Flusher.java:252)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1136)
        at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:623)
        at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
"IPC Server handler 2 on 60020":
        at org.apache.hadoop.hbase.regionserver.Flusher.addRegion(Flusher.java:237)
        - waiting to lock <0xb6f69cf0> (a java.util.HashSet)
        at org.apache.hadoop.hbase.regionserver.Flusher.request(Flusher.java:114)
        at org.apache.hadoop.hbase.regionserver.HRegion.requestFlush(HRegion.java:1627)
        - locked <0xb73610c0> (a org.apache.hadoop.hbase.regionserver.HRegion$WriteState)
        at org.apache.hadoop.hbase.regionserver.HRegion.update(HRegion.java:1614)
        at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1398)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1137)
        at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:623)
        at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)

Found 1 deadlock.
{code}

Regionserver is hosed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HBASE-818) Deadlock running 'flushSomeRegions'

Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Purtell updated HBASE-818:
---------------------------------

    Attachment: 818.patch

> Deadlock running 'flushSomeRegions'
> -----------------------------------
>
>                 Key: HBASE-818
>                 URL: https://issues.apache.org/jira/browse/HBASE-818
>             Project: Hadoop HBase
>          Issue Type: Bug
>    Affects Versions: 0.2.0
>            Reporter: stack
>            Priority: Blocker
>             Fix For: 0.2.1, 0.3.0
>
>         Attachments: 818.patch
>
>
> Playing with MR uploading no a regionserver with 60+ regions, I ran into a deadlock:
> {code}
> Found one Java-level deadlock:
> =============================
> "IPC Server handler 19 on 60020":
>   waiting to lock monitor 0x084be38c (object 0xb6f69a70, a org.apache.hadoop.hbase.regionserver.Flusher),
>   which is held by "IPC Server handler 16 on 60020"
> "IPC Server handler 16 on 60020":
>   waiting to lock monitor 0x080f8dec (object 0xb73610c0, a org.apache.hadoop.hbase.regionserver.HRegion$WriteState),
>   which is held by "IPC Server handler 2 on 60020"
> "IPC Server handler 2 on 60020":
>   waiting to lock monitor 0x086e8fe8 (object 0xb6f69cf0, a java.util.HashSet),
>   which is held by "IPC Server handler 16 on 60020"
> Java stack information for the threads listed above:
> ===================================================
> "IPC Server handler 19 on 60020":
>         at org.apache.hadoop.hbase.regionserver.Flusher.flushSomeRegions(Flusher.java:261)
>         - waiting to lock <0xb6f69a70> (a org.apache.hadoop.hbase.regionserver.Flusher)
>         at org.apache.hadoop.hbase.regionserver.Flusher.reclaimMemcacheMemory(Flusher.java:252)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1136)
>         at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:623)
>         at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> "IPC Server handler 16 on 60020":
>         at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:948)
>         - waiting to lock <0xb73610c0> (a org.apache.hadoop.hbase.regionserver.HRegion$WriteState)
>         at org.apache.hadoop.hbase.regionserver.Flusher.flushRegion(Flusher.java:173)
>         - locked <0xb6f69cf0> (a java.util.HashSet)
>         at org.apache.hadoop.hbase.regionserver.Flusher.flushSomeRegions(Flusher.java:267)
>         - locked <0xb6f69a70> (a org.apache.hadoop.hbase.regionserver.Flusher)
>         at org.apache.hadoop.hbase.regionserver.Flusher.reclaimMemcacheMemory(Flusher.java:252)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1136)
>         at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:623)
>         at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> "IPC Server handler 2 on 60020":
>         at org.apache.hadoop.hbase.regionserver.Flusher.addRegion(Flusher.java:237)
>         - waiting to lock <0xb6f69cf0> (a java.util.HashSet)
>         at org.apache.hadoop.hbase.regionserver.Flusher.request(Flusher.java:114)
>         at org.apache.hadoop.hbase.regionserver.HRegion.requestFlush(HRegion.java:1627)
>         - locked <0xb73610c0> (a org.apache.hadoop.hbase.regionserver.HRegion$WriteState)
>         at org.apache.hadoop.hbase.regionserver.HRegion.update(HRegion.java:1614)
>         at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1398)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1137)
>         at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:623)
>         at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> Found 1 deadlock.
> {code}
> Regionserver is hosed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HBASE-818) Deadlock running 'flushSomeRegions'

Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Purtell updated HBASE-818:
---------------------------------

             Assignee: Andrew Purtell
    Affects Version/s: 0.2.0
               Status: Patch Available  (was: Open)

> Deadlock running 'flushSomeRegions'
> -----------------------------------
>
>                 Key: HBASE-818
>                 URL: https://issues.apache.org/jira/browse/HBASE-818
>             Project: Hadoop HBase
>          Issue Type: Bug
>    Affects Versions: 0.2.0
>            Reporter: stack
>            Assignee: Andrew Purtell
>            Priority: Blocker
>             Fix For: 0.2.1, 0.3.0
>
>         Attachments: 818.patch
>
>
> Playing with MR uploading no a regionserver with 60+ regions, I ran into a deadlock:
> {code}
> Found one Java-level deadlock:
> =============================
> "IPC Server handler 19 on 60020":
>   waiting to lock monitor 0x084be38c (object 0xb6f69a70, a org.apache.hadoop.hbase.regionserver.Flusher),
>   which is held by "IPC Server handler 16 on 60020"
> "IPC Server handler 16 on 60020":
>   waiting to lock monitor 0x080f8dec (object 0xb73610c0, a org.apache.hadoop.hbase.regionserver.HRegion$WriteState),
>   which is held by "IPC Server handler 2 on 60020"
> "IPC Server handler 2 on 60020":
>   waiting to lock monitor 0x086e8fe8 (object 0xb6f69cf0, a java.util.HashSet),
>   which is held by "IPC Server handler 16 on 60020"
> Java stack information for the threads listed above:
> ===================================================
> "IPC Server handler 19 on 60020":
>         at org.apache.hadoop.hbase.regionserver.Flusher.flushSomeRegions(Flusher.java:261)
>         - waiting to lock <0xb6f69a70> (a org.apache.hadoop.hbase.regionserver.Flusher)
>         at org.apache.hadoop.hbase.regionserver.Flusher.reclaimMemcacheMemory(Flusher.java:252)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1136)
>         at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:623)
>         at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> "IPC Server handler 16 on 60020":
>         at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:948)
>         - waiting to lock <0xb73610c0> (a org.apache.hadoop.hbase.regionserver.HRegion$WriteState)
>         at org.apache.hadoop.hbase.regionserver.Flusher.flushRegion(Flusher.java:173)
>         - locked <0xb6f69cf0> (a java.util.HashSet)
>         at org.apache.hadoop.hbase.regionserver.Flusher.flushSomeRegions(Flusher.java:267)
>         - locked <0xb6f69a70> (a org.apache.hadoop.hbase.regionserver.Flusher)
>         at org.apache.hadoop.hbase.regionserver.Flusher.reclaimMemcacheMemory(Flusher.java:252)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1136)
>         at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:623)
>         at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> "IPC Server handler 2 on 60020":
>         at org.apache.hadoop.hbase.regionserver.Flusher.addRegion(Flusher.java:237)
>         - waiting to lock <0xb6f69cf0> (a java.util.HashSet)
>         at org.apache.hadoop.hbase.regionserver.Flusher.request(Flusher.java:114)
>         at org.apache.hadoop.hbase.regionserver.HRegion.requestFlush(HRegion.java:1627)
>         - locked <0xb73610c0> (a org.apache.hadoop.hbase.regionserver.HRegion$WriteState)
>         at org.apache.hadoop.hbase.regionserver.HRegion.update(HRegion.java:1614)
>         at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1398)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1137)
>         at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:623)
>         at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> Found 1 deadlock.
> {code}
> Regionserver is hosed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HBASE-818) Deadlock running 'flushSomeRegions'

Posted by "stack (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HBASE-818:
------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

Applied branch and trunk. Thanks for the patch Andrew.

> Deadlock running 'flushSomeRegions'
> -----------------------------------
>
>                 Key: HBASE-818
>                 URL: https://issues.apache.org/jira/browse/HBASE-818
>             Project: Hadoop HBase
>          Issue Type: Bug
>    Affects Versions: 0.2.0
>            Reporter: stack
>            Assignee: Andrew Purtell
>            Priority: Blocker
>             Fix For: 0.2.1, 0.3.0
>
>         Attachments: 818.patch
>
>
> Playing with MR uploading no a regionserver with 60+ regions, I ran into a deadlock:
> {code}
> Found one Java-level deadlock:
> =============================
> "IPC Server handler 19 on 60020":
>   waiting to lock monitor 0x084be38c (object 0xb6f69a70, a org.apache.hadoop.hbase.regionserver.Flusher),
>   which is held by "IPC Server handler 16 on 60020"
> "IPC Server handler 16 on 60020":
>   waiting to lock monitor 0x080f8dec (object 0xb73610c0, a org.apache.hadoop.hbase.regionserver.HRegion$WriteState),
>   which is held by "IPC Server handler 2 on 60020"
> "IPC Server handler 2 on 60020":
>   waiting to lock monitor 0x086e8fe8 (object 0xb6f69cf0, a java.util.HashSet),
>   which is held by "IPC Server handler 16 on 60020"
> Java stack information for the threads listed above:
> ===================================================
> "IPC Server handler 19 on 60020":
>         at org.apache.hadoop.hbase.regionserver.Flusher.flushSomeRegions(Flusher.java:261)
>         - waiting to lock <0xb6f69a70> (a org.apache.hadoop.hbase.regionserver.Flusher)
>         at org.apache.hadoop.hbase.regionserver.Flusher.reclaimMemcacheMemory(Flusher.java:252)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1136)
>         at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:623)
>         at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> "IPC Server handler 16 on 60020":
>         at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:948)
>         - waiting to lock <0xb73610c0> (a org.apache.hadoop.hbase.regionserver.HRegion$WriteState)
>         at org.apache.hadoop.hbase.regionserver.Flusher.flushRegion(Flusher.java:173)
>         - locked <0xb6f69cf0> (a java.util.HashSet)
>         at org.apache.hadoop.hbase.regionserver.Flusher.flushSomeRegions(Flusher.java:267)
>         - locked <0xb6f69a70> (a org.apache.hadoop.hbase.regionserver.Flusher)
>         at org.apache.hadoop.hbase.regionserver.Flusher.reclaimMemcacheMemory(Flusher.java:252)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1136)
>         at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:623)
>         at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> "IPC Server handler 2 on 60020":
>         at org.apache.hadoop.hbase.regionserver.Flusher.addRegion(Flusher.java:237)
>         - waiting to lock <0xb6f69cf0> (a java.util.HashSet)
>         at org.apache.hadoop.hbase.regionserver.Flusher.request(Flusher.java:114)
>         at org.apache.hadoop.hbase.regionserver.HRegion.requestFlush(HRegion.java:1627)
>         - locked <0xb73610c0> (a org.apache.hadoop.hbase.regionserver.HRegion$WriteState)
>         at org.apache.hadoop.hbase.regionserver.HRegion.update(HRegion.java:1614)
>         at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1398)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1137)
>         at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:623)
>         at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> Found 1 deadlock.
> {code}
> Regionserver is hosed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HBASE-818) Deadlock running 'flushSomeRegions'

Posted by "stack (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12621704#action_12621704 ] 

stack commented on HBASE-818:
-----------------------------

Suggest moving the line 'this.flushListener.request(this);' outside of the synchronize on WriteState.  Testing here, it seems to do the trick.  There's no need of the flush request being inside the WriteState synchronize block since the scheduling of flushes is async anyways.

> Deadlock running 'flushSomeRegions'
> -----------------------------------
>
>                 Key: HBASE-818
>                 URL: https://issues.apache.org/jira/browse/HBASE-818
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Priority: Blocker
>             Fix For: 0.2.1, 0.3.0
>
>
> Playing with MR uploading no a regionserver with 60+ regions, I ran into a deadlock:
> {code}
> Found one Java-level deadlock:
> =============================
> "IPC Server handler 19 on 60020":
>   waiting to lock monitor 0x084be38c (object 0xb6f69a70, a org.apache.hadoop.hbase.regionserver.Flusher),
>   which is held by "IPC Server handler 16 on 60020"
> "IPC Server handler 16 on 60020":
>   waiting to lock monitor 0x080f8dec (object 0xb73610c0, a org.apache.hadoop.hbase.regionserver.HRegion$WriteState),
>   which is held by "IPC Server handler 2 on 60020"
> "IPC Server handler 2 on 60020":
>   waiting to lock monitor 0x086e8fe8 (object 0xb6f69cf0, a java.util.HashSet),
>   which is held by "IPC Server handler 16 on 60020"
> Java stack information for the threads listed above:
> ===================================================
> "IPC Server handler 19 on 60020":
>         at org.apache.hadoop.hbase.regionserver.Flusher.flushSomeRegions(Flusher.java:261)
>         - waiting to lock <0xb6f69a70> (a org.apache.hadoop.hbase.regionserver.Flusher)
>         at org.apache.hadoop.hbase.regionserver.Flusher.reclaimMemcacheMemory(Flusher.java:252)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1136)
>         at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:623)
>         at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> "IPC Server handler 16 on 60020":
>         at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:948)
>         - waiting to lock <0xb73610c0> (a org.apache.hadoop.hbase.regionserver.HRegion$WriteState)
>         at org.apache.hadoop.hbase.regionserver.Flusher.flushRegion(Flusher.java:173)
>         - locked <0xb6f69cf0> (a java.util.HashSet)
>         at org.apache.hadoop.hbase.regionserver.Flusher.flushSomeRegions(Flusher.java:267)
>         - locked <0xb6f69a70> (a org.apache.hadoop.hbase.regionserver.Flusher)
>         at org.apache.hadoop.hbase.regionserver.Flusher.reclaimMemcacheMemory(Flusher.java:252)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1136)
>         at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:623)
>         at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> "IPC Server handler 2 on 60020":
>         at org.apache.hadoop.hbase.regionserver.Flusher.addRegion(Flusher.java:237)
>         - waiting to lock <0xb6f69cf0> (a java.util.HashSet)
>         at org.apache.hadoop.hbase.regionserver.Flusher.request(Flusher.java:114)
>         at org.apache.hadoop.hbase.regionserver.HRegion.requestFlush(HRegion.java:1627)
>         - locked <0xb73610c0> (a org.apache.hadoop.hbase.regionserver.HRegion$WriteState)
>         at org.apache.hadoop.hbase.regionserver.HRegion.update(HRegion.java:1614)
>         at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1398)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1137)
>         at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:623)
>         at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> Found 1 deadlock.
> {code}
> Regionserver is hosed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.