primary namenode not starting

[Colin Freas <co...@gmail.com>]

The primary namenode on my cluster seems to have stopped working.  The
secondary name node starts, but the primary fails with the error message
below.

I've scoured the cluster, particularly this node for changes, but I haven't
found any that I believe would cause this problem.

If anyone has an idea what I might look for, I'd appreciate any help.  Also,
is there any way to increase the verbosity of the logging?

-Colin

-------

************************************************************/
2008-05-09 11:31:46,484 INFO org.apache.hadoop.dfs.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = dev04/10.0.2.12
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 0.16.1
STARTUP_MSG:   build =
http://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.16 -r 635123;
compiled by 'hadoopqa' on Sun Mar  9 05:44:19 UTC 2008
************************************************************/
2008-05-09 11:31:46,656 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
Initializing RPC Metrics with hostName=NameNode, port=54310
2008-05-09 11:31:46,665 INFO org.apache.hadoop.dfs.NameNode: Namenode up at:
dev04/10.0.2.12:54310
2008-05-09 11:31:46,671 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=NameNode, sessionId=null
2008-05-09 11:31:46,676 INFO org.apache.hadoop.dfs.NameNodeMetrics:
Initializing NameNodeMeterics using context
object:org.apache.hadoop.metrics.spi.NullContext
2008-05-09 11:31:46,761 INFO org.apache.hadoop.fs.FSNamesystem:
fsOwner=hadoop,hadoop
2008-05-09 11:31:46,761 INFO org.apache.hadoop.fs.FSNamesystem:
supergroup=supergroup
2008-05-09 11:31:46,761 INFO org.apache.hadoop.fs.FSNamesystem:
isPermissionEnabled=true
2008-05-09 11:31:47,132 INFO org.apache.hadoop.ipc.Server: Stopping server
on 54310
2008-05-09 11:31:47,135 ERROR org.apache.hadoop.dfs.NameNode:
java.io.EOFException
        at java.io.DataInputStream.readFully(DataInputStream.java:178)
        at org.apache.hadoop.io.UTF8.readFields(UTF8.java:106)
        at
org.apache.hadoop.io.ArrayWritable.readFields(ArrayWritable.java:90)
        at org.apache.hadoop.dfs.FSEditLog.loadFSEdits(FSEditLog.java:433)
        at org.apache.hadoop.dfs.FSImage.loadFSEdits(FSImage.java:756)
        at org.apache.hadoop.dfs.FSImage.loadFSImage(FSImage.java:639)
        at
org.apache.hadoop.dfs.FSImage.recoverTransitionRead(FSImage.java:222)
        at
org.apache.hadoop.dfs.FSDirectory.loadFSImage(FSDirectory.java:79)
        at
org.apache.hadoop.dfs.FSNamesystem.initialize(FSNamesystem.java:254)
        at org.apache.hadoop.dfs.FSNamesystem.<init>(FSNamesystem.java:235)
        at org.apache.hadoop.dfs.NameNode.initialize(NameNode.java:131)
        at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:176)
        at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:162)
        at org.apache.hadoop.dfs.NameNode.createNameNode(NameNode.java:846)
        at org.apache.hadoop.dfs.NameNode.main(NameNode.java:855)

2008-05-09 11:31:47,135 INFO org.apache.hadoop.dfs.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at dev04/10.0.2.12
************************************************************/