Loading FSEditLog fails

[Andrew <gu...@gmail.com>]

Namenode fails to start and writes this error. What can I do to repair it?


2009-05-13 20:54:55,098 INFO  namenode.FSNamesystem 
(FSNamesystem.java:registerMBean(4464)) - Registered F
SNamesystemStatusMBean
2009-05-13 20:54:55,133 INFO  common.Storage (FSImage.java:loadFSImage(852)) - 
Number of files = 52756
2009-05-13 20:54:55,133 INFO  common.Storage (FSImage.java:loadFSImage(852)) - 
Number of files = 52756
2009-05-13 20:54:56,492 INFO  common.Storage 
(FSImage.java:loadFilesUnderConstruction(1183)) - Number of f
iles under construction = 0
2009-05-13 20:54:56,492 INFO  common.Storage 
(FSImage.java:loadFilesUnderConstruction(1183)) - Number of files under 
construction = 0
2009-05-13 20:54:56,493 INFO  common.Storage (FSImage.java:loadFSImage(789)) - 
Image file of size 6551593 loaded in 1 seconds.
2009-05-13 20:54:56,493 INFO  common.Storage (FSImage.java:loadFSImage(789)) - 
Image file of size 6551593 loaded in 1 seconds.
2009-05-13 20:54:56,676 ERROR namenode.FSNamesystem 
(FSNamesystem.java:<init>(290)) - FSNamesystem initialization failed.
java.io.IOException: Incorrect data format. logVersion is -18 but 
writables.length is 0.
        at 
org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.java:542)
        at 
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:973)
        at 
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:793)
        at 
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:352)
        at 
org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:87)
        at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:309)
        at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:288)
        at 
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:163)
        at 
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:208)
        at 
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:194)
        at 
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:859)
        at 
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:868)

Re: Loading FSEditLog fails

[Andrew <gu...@gmail.com>]

OK, I've just solved problem with minor data lost. Steps to solve:

1) comment out FSEditLog.java:542
2) compile hadoop-core jar
3) start cluster with new jar
Namenode will skip bad records in "name/current/edits" and write new edits 
file back into fs. As bad records stand for actual IO operations, some files 
in HDFS may be deleted as they consist of blocks which do not correspond to 
edits entries. In my situtation, I've lost files of last fortnight period.
4) wait for some time while datanodes are removing blocks, that do not 
corresond to entries in edits file
5) stop cluster
6) replace hadoop-core jar with release one
7) start cluster