You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Lohit Vijayarenu (JIRA)" <ji...@apache.org> on 2008/07/21 20:31:31 UTC
[jira] Issue Comment Edited: (HADOOP-3724) Namenode does not start
due to exception throw while saving Image
[ https://issues.apache.org/jira/browse/HADOOP-3724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12615310#action_12615310 ]
lohit edited comment on HADOOP-3724 at 7/21/08 11:29 AM:
--------------------------------------------------------------------
After looking at the logs, here is something which might have triggered this
- 00:06:35,042 /mapredsystem/user/mapredsystem/machine/job_200807182151_0100/job.jar was created
- 00:06:36,641 request for NameSystem.allocateBlock blk_353852126366132690_6689141 of file /mapredsystem/user/mapredsystem/machine/job_200807182151_0100/job.jar
- 00:10:18,138 request for rename of /mapredsystem/user/mapredsystem/machine to /user/user/.Trash/Current/machine
- 01:06:35,583 After around one hour, lease recovery for the open file happens but with new path under .Trash With this exception going on forever
{noformat}
INFO org.apache.hadoop.dfs.LeaseManager: Lease Monitor: Removing lease [Lease. Holder: DFSClient_-2067534834, pendingcreates: 1], sortedLeases.size()=: 13
INFO org.apache.hadoop.fs.FSNamesystem: Recovering lease=[Lease. Holder: DFSClient_-2067534834, pendingcreates: 1], src=/user/user/.Trash/Current/machine/machine/job_200807182151_0100/job.jar
WARN org.apache.hadoop.dfs.StateChange: DIR* NameSystem.internalReleaseCreate: attempt to release a create lock on /user/user/.Trash/Current/machine/machine/job_200807182151_0100/job.jar file does not exist.
{noformat}
Also,
- For this particular file, allocating a block was not followed by addStoredBlock, none of the datanodes reported back to namenode with block length, hence the file was still open
was (Author: lohit):
After looking at the logs, here is something which might have triggered this
- 00:06:35,042 /mapredsystem/user/mapredsystem/machine/job_200807182151_0100/job.jar was created
- 00:06:36,641 request for NameSystem.allocateBlock blk_353852126366132690_6689141 of file /mapredsystem/user/mapredsystem/machine/job_200807182151_0100/job.jar
- 00:10:18,138 request for rename of /mapredsystem/user/mapredsystem/machine to /user/user/.Trash/Current/machine
- 01:06:35,583 After around one hour, lease recovery for the open file happens but with new path under .Trash With this exception going on forever
{noformat}
INFO org.apache.hadoop.fs.FSNamesystem: Recovering lease=[Lease. Holder: DFSClient_-2067534834, pendingcreates: 1], src=/user/user/.Trash/Current/machine/machine/job_200807182151_0100/job.jar
{noformat}
Also,
- For this particular file, allocating a block was not followed by addStoredBlock, none of the datanodes reported back to namenode with block length, hence the file was still open
> Namenode does not start due to exception throw while saving Image
> -----------------------------------------------------------------
>
> Key: HADOOP-3724
> URL: https://issues.apache.org/jira/browse/HADOOP-3724
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.18.0
> Reporter: Lohit Vijayarenu
> Assignee: dhruba borthakur
> Priority: Blocker
> Fix For: 0.18.0
>
>
> Re-start of namenode failed with this stack trace while savingImage during initialization
> {noformat}
> 2008-07-09 00:20:21,470 INFO org.apache.hadoop.ipc.Server: Stopping server on 9000
> 2008-07-09 00:20:21,493 ERROR org.apache.hadoop.dfs.NameNode: java.io.IOException: saveLeases found path /foo/bar/jambajuice but no matching entry in namespace.
> at org.apache.hadoop.dfs.FSNamesystem.saveFilesUnderConstruction(FSNamesystem.java:4376)
> at org.apache.hadoop.dfs.FSImage.saveFSImage(FSImage.java:874)
> at org.apache.hadoop.dfs.FSImage.saveFSImage(FSImage.java:892)
> at org.apache.hadoop.dfs.FSDirectory.loadFSImage(FSDirectory.java:81)
> at org.apache.hadoop.dfs.FSNamesystem.initialize(FSNamesystem.java:273)
> at org.apache.hadoop.dfs.FSNamesystem.<init>(FSNamesystem.java:252)
> at org.apache.hadoop.dfs.NameNode.initialize(NameNode.java:148)
> at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:193)
> at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:179)
> at org.apache.hadoop.dfs.NameNode.createNameNode(NameNode.java:822)
> at org.apache.hadoop.dfs.NameNode.main(NameNode.java:831)
> {noformat}
> Looks like it was throwing IOException in saveFilesUnderConstruction
> Before restart NameNode was killed while some jobs were running. Upon looking at the namenode log before the stopping of namenode, there were many entries like this
> {noformat}
> 2008-07-09 00:12:55,301 INFO org.apache.hadoop.fs.FSNamesystem: Recovering lease=[Lease. Holder: DFSClient_-510679348, pendingcreates: 1], src=/foo/bar/jambajuice
> 2008-07-09 00:12:55,301 WARN org.apache.hadoop.dfs.StateChange: DIR* NameSystem.internalReleaseCreate: attempt to release a create lock on /foo/bar/jambajuice file does not exist.
> {noformat}
> These 2 lines are repeated forever every second, to a point where I see that a 7 node cluster had namenode log with size close to 41G.
> Could not find any other information about the file as there were not previous namenode logs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.