You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Daryn Sharp (JIRA)" <ji...@apache.org> on 2014/09/11 16:20:34 UTC
[jira] [Commented] (HDFS-7046) HA NN can NPE upon transition to
active
[ https://issues.apache.org/jira/browse/HDFS-7046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14130072#comment-14130072 ]
Daryn Sharp commented on HDFS-7046:
-----------------------------------
Stack trace from an internal 2.5.1-ish build. Line numbers may be off, but demonstrates a delete adjusted safe mode block totals, triggering exit of safe mode, which called {{startSecretManagerIfNecessary}} which only starts if {{getEditLog().isOpenForWrite()}} is true. The edit log stream is actually null.
{noformat}
[IPC Server handler 84 on 8020] ERROR namenode.FSEditLogLoader: Encountered exception on operation DeleteOp [length=0, path=..., timestamp=1410390000919,
RpcClientId=..., RpcCallId=..., opCode=OP_DELETE, txid=...]
java.lang.NullPointerException
at org.apache.hadoop.hdfs.server.namenode.FSEditLog.logEdit(FSEditLog.java:410)
at org.apache.hadoop.hdfs.server.namenode.FSEditLog.logUpdateMasterKey(FSEditLog.java:954)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.logUpdateMasterKey(FSNamesystem.java:6842)
at org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenSecretManager.logUpdateMasterKey(DelegationTokenSecretManager.java:378)
at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.updateCurrentKey(AbstractDelegationTokenSecretManager.java:232)
at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.startThreads(AbstractDelegationTokenSecretManager.java:116)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startSecretManager(FSNamesystem.java:990)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startSecretManagerIfNecessary(FSNamesystem.java:1004)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.access$1200(FSNamesystem.java:294)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeInfo.leave(FSNamesystem.java:5208)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeInfo.checkMode(FSNamesystem.java:5283)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeInfo.setBlockTotal(FSNamesystem.java:5320)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeInfo.adjustBlockTotals(FSNamesystem.java:5509)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeInfo.access$2100(FSNamesystem.java:5054)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.adjustSafeModeBlockTotals(FSNamesystem.java:5654)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeBlocksAndUpdateSafemodeTotal(FSNamesystem.java:3646)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removePathAndBlocks(FSNamesystem.java:3610)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedDelete(FSDirectory.java:1196)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:506)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:228)
{noformat}
> HA NN can NPE upon transition to active
> ---------------------------------------
>
> Key: HDFS-7046
> URL: https://issues.apache.org/jira/browse/HDFS-7046
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
> Affects Versions: 3.0.0, 2.5.0
> Reporter: Daryn Sharp
> Priority: Critical
>
> While processing edits, the NN may decide after adjusting block totals to leave safe mode - in the middle of the edit. Going active starts the secret manager which generates a new secret key, which in turn generates an edit, which NPEs because the edit log is not open.
> # Transitions should _not_ occur in the middle of an edit.
> # The edit log appears to claim it's open for write when the stream isn't even open
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)