You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Jing Zhao (JIRA)" <ji...@apache.org> on 2013/08/27 02:47:51 UTC
[jira] [Commented] (HDFS-5132) Deadlock in namenode while running
load generator with 15 threads
[ https://issues.apache.org/jira/browse/HDFS-5132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13750824#comment-13750824 ]
Jing Zhao commented on HDFS-5132:
---------------------------------
Looks like the deadlock was caused by the following locking scenario:
1. A thread is trying to acquire datanodeMap's monitor while holding the monitor of SafeModeInfo
{noformat}
Thread 14109: (state = BLOCKED)
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.getNumLiveDataNodes() @bci=8, line=877 (Compiled frame)
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNumLiveDataNodes() @bci=7, line=5287 (Compiled frame)
org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeInfo.needEnter() @bci=24, line=4487 (Compiled frame)
org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeInfo.canLeave() @bci=38, line=4479 (Compiled frame)
org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeInfo.access$1300(org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeInfo) @bci=1, line=4275 (Compiled frame)
org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeMonitor.run() @bci=27, line=4749 (Interpreted frame)
java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
{noformat}
2. Another thread is trying to get the monitor of SafeModeInfo while holding the datanodeMap's monitor
{noformat}
Thread 13921: (state = BLOCKED)
org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeInfo.isOn() @bci=0, line=4385 (Compiled frame)
org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeInfo.access$1500(org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeInfo) @bci=1, line=4275 (Compiled frame)
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.isInSafeMode() @bci=12, line=4798 (Interpreted frame)
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.handleHeartbeat(org.apache.hadoop.hdfs.server.protocol.DatanodeRegistration, java.lang.String, long, long, long, long, int, int, int) @bci=131, line=1180 (Interpreted frame)
{noformat}
> Deadlock in namenode while running load generator with 15 threads
> -----------------------------------------------------------------
>
> Key: HDFS-5132
> URL: https://issues.apache.org/jira/browse/HDFS-5132
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
> Affects Versions: 2.1.0-beta
> Reporter: Arpit Gupta
> Attachments: jstack.log
>
>
> While running nn load generator with 15 threads for 20 mins the standby namenode deadlocked.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira