You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2008/09/03 19:31:44 UTC
[jira] Created: (HBASE-864) Deadlock in regionserver
Deadlock in regionserver
------------------------
Key: HBASE-864
URL: https://issues.apache.org/jira/browse/HBASE-864
Project: Hadoop HBase
Issue Type: Bug
Reporter: stack
Found in 0.2.0 by Michael Bieniosk:
{code}
Found one Java-level deadlock:============================="IPC Server handler 9 on 60020": waiting to lock monitor 0x00000000405b6688 (object 0x00007f328f64f260, a org.apache.hadoop.hbase.regionserver.HRegion$WriteState), which is held by "IPC Server handler 3 on 60020""IPC Server handler 3 on 60020": waiting to lock monitor 0x00000000403d05a0 (object 0x00007f328ce1e988, a java.util.HashSet), which is held by "regionserver/0:0:0:0:0:0:0:0:60020.cacheFlusher""regionserver/0:0:0:0:0:0:0:0:60020.cacheFlusher": waiting to lock monitor 0x00000000405b6688 (object 0x00007f328f64f260, a org.apache.hadoop.hbase.regionserver.HRegion$WriteState),
which is held by "IPC Server handler 3 on 60020"
Java stack information for the threads listed above:
===================================================
"IPC Server handler 9 on 60020":
at org.apache.hadoop.hbase.regionserver.HRegion.requestFlush(HRegion.java:1623)
- waiting to lock <0x00007f328f64f260> (a org.apache.hadoop.hbase.regionserver.HRegion$WriteState)
at org.apache.hadoop.hbase.regionserver.HRegion.update(HRegion.java:1614)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1398)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1151)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
"IPC Server handler 3 on 60020":
at org.apache.hadoop.hbase.regionserver.Flusher.addRegion(Flusher.java:237)
- waiting to lock <0x00007f328ce1e988> (a java.util.HashSet)
at org.apache.hadoop.hbase.regionserver.Flusher.request(Flusher.java:114)
at org.apache.hadoop.hbase.regionserver.HRegion.requestFlush(HRegion.java:1627)
- locked <0x00007f328f64f260> (a org.apache.hadoop.hbase.regionserver.HRegion$WriteState)
at org.apache.hadoop.hbase.regionserver.HRegion.update(HRegion.java:1614)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1398)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1151)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
"regionserver/0:0:0:0:0:0:0:0:60020.cacheFlusher":
at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:948)
- waiting to lock <0x00007f328f64f260> (a org.apache.hadoop.hbase.regionserver.HRegion$WriteState)
at org.apache.hadoop.hbase.regionserver.Flusher.flushRegion(Flusher.java:173)
- locked <0x00007f328ce1e988> (a java.util.HashSet)
at org.apache.hadoop.hbase.regionserver.Flusher.run(Flusher.java:91)
Found 1 deadlock.
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-864) Deadlock in regionserver
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-864:
-------------------------------------
Attachment: hbase-864-branch-0.2.patch
Patchn arrows down the lock on regionsInQueue after we get the lock. Review please.
> Deadlock in regionserver
> ------------------------
>
> Key: HBASE-864
> URL: https://issues.apache.org/jira/browse/HBASE-864
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Attachments: hbase-864-branch-0.2.patch
>
>
> Found in 0.2.0 by Michael Bieniosk:
> {code}
> Found one Java-level deadlock:============================="IPC Server handler 9 on 60020": waiting to lock monitor 0x00000000405b6688 (object 0x00007f328f64f260, a org.apache.hadoop.hbase.regionserver.HRegion$WriteState), which is held by "IPC Server handler 3 on 60020""IPC Server handler 3 on 60020": waiting to lock monitor 0x00000000403d05a0 (object 0x00007f328ce1e988, a java.util.HashSet), which is held by "regionserver/0:0:0:0:0:0:0:0:60020.cacheFlusher""regionserver/0:0:0:0:0:0:0:0:60020.cacheFlusher": waiting to lock monitor 0x00000000405b6688 (object 0x00007f328f64f260, a org.apache.hadoop.hbase.regionserver.HRegion$WriteState),
> which is held by "IPC Server handler 3 on 60020"
> Java stack information for the threads listed above:
> ===================================================
> "IPC Server handler 9 on 60020":
> at org.apache.hadoop.hbase.regionserver.HRegion.requestFlush(HRegion.java:1623)
> - waiting to lock <0x00007f328f64f260> (a org.apache.hadoop.hbase.regionserver.HRegion$WriteState)
> at org.apache.hadoop.hbase.regionserver.HRegion.update(HRegion.java:1614)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1398)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1151)
> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> "IPC Server handler 3 on 60020":
> at org.apache.hadoop.hbase.regionserver.Flusher.addRegion(Flusher.java:237)
> - waiting to lock <0x00007f328ce1e988> (a java.util.HashSet)
> at org.apache.hadoop.hbase.regionserver.Flusher.request(Flusher.java:114)
> at org.apache.hadoop.hbase.regionserver.HRegion.requestFlush(HRegion.java:1627)
> - locked <0x00007f328f64f260> (a org.apache.hadoop.hbase.regionserver.HRegion$WriteState)
> at org.apache.hadoop.hbase.regionserver.HRegion.update(HRegion.java:1614)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1398)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1151)
> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> "regionserver/0:0:0:0:0:0:0:0:60020.cacheFlusher":
> at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:948)
> - waiting to lock <0x00007f328f64f260> (a org.apache.hadoop.hbase.regionserver.HRegion$WriteState)
> at org.apache.hadoop.hbase.regionserver.Flusher.flushRegion(Flusher.java:173)
> - locked <0x00007f328ce1e988> (a java.util.HashSet)
> at org.apache.hadoop.hbase.regionserver.Flusher.run(Flusher.java:91)
> Found 1 deadlock.
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-864) Deadlock in regionserver
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-864.
-------------------------
Resolution: Fixed
Fix Version/s: 0.18.0
0.2.1
Applied to branch and trunk. Thanks for the patch J-D.
> Deadlock in regionserver
> ------------------------
>
> Key: HBASE-864
> URL: https://issues.apache.org/jira/browse/HBASE-864
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.2.1, 0.18.0
>
> Attachments: hbase-864-branch-0.2.patch
>
>
> Found in 0.2.0 by Michael Bieniosk:
> {code}
> Found one Java-level deadlock:============================="IPC Server handler 9 on 60020": waiting to lock monitor 0x00000000405b6688 (object 0x00007f328f64f260, a org.apache.hadoop.hbase.regionserver.HRegion$WriteState), which is held by "IPC Server handler 3 on 60020""IPC Server handler 3 on 60020": waiting to lock monitor 0x00000000403d05a0 (object 0x00007f328ce1e988, a java.util.HashSet), which is held by "regionserver/0:0:0:0:0:0:0:0:60020.cacheFlusher""regionserver/0:0:0:0:0:0:0:0:60020.cacheFlusher": waiting to lock monitor 0x00000000405b6688 (object 0x00007f328f64f260, a org.apache.hadoop.hbase.regionserver.HRegion$WriteState),
> which is held by "IPC Server handler 3 on 60020"
> Java stack information for the threads listed above:
> ===================================================
> "IPC Server handler 9 on 60020":
> at org.apache.hadoop.hbase.regionserver.HRegion.requestFlush(HRegion.java:1623)
> - waiting to lock <0x00007f328f64f260> (a org.apache.hadoop.hbase.regionserver.HRegion$WriteState)
> at org.apache.hadoop.hbase.regionserver.HRegion.update(HRegion.java:1614)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1398)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1151)
> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> "IPC Server handler 3 on 60020":
> at org.apache.hadoop.hbase.regionserver.Flusher.addRegion(Flusher.java:237)
> - waiting to lock <0x00007f328ce1e988> (a java.util.HashSet)
> at org.apache.hadoop.hbase.regionserver.Flusher.request(Flusher.java:114)
> at org.apache.hadoop.hbase.regionserver.HRegion.requestFlush(HRegion.java:1627)
> - locked <0x00007f328f64f260> (a org.apache.hadoop.hbase.regionserver.HRegion$WriteState)
> at org.apache.hadoop.hbase.regionserver.HRegion.update(HRegion.java:1614)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1398)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1151)
> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> "regionserver/0:0:0:0:0:0:0:0:60020.cacheFlusher":
> at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:948)
> - waiting to lock <0x00007f328f64f260> (a org.apache.hadoop.hbase.regionserver.HRegion$WriteState)
> at org.apache.hadoop.hbase.regionserver.Flusher.flushRegion(Flusher.java:173)
> - locked <0x00007f328ce1e988> (a java.util.HashSet)
> at org.apache.hadoop.hbase.regionserver.Flusher.run(Flusher.java:91)
> Found 1 deadlock.
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.