Problem after turning hadoop machines power-off without shutting down

[Hieu Le <hi...@gmail.com>]

Hello,

We have a cluster with ~1.5TB of data on it. After the power is off
unexpectedly, I restart the cluster again but the administrative webpage is
not available. When I ran $hadoop fsck /, I got bellow error message:

hadoop@b1:~/hadoop/bin$ ./hadoop fsck /
Exception in thread "main" java.io.IOException: Server returned HTTP
response code: 503 for URL: http://0.0.0.0:50070/fsck?path=%2F
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1313)
at org.apache.hadoop.hdfs.tools.DFSck.run(DFSck.java:123)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.hdfs.tools.DFSck.main(DFSck.java:159)
hadoop@b1:~/hadoop/bin$

Checking the log of the namenode, bellow is what I see:

2010-08-04 06:21:57,624 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* ask
192.168.1.5:50010 to delete  blk_7399677479348617513_221053
2010-08-04 06:22:00,625 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* ask
192.168.1.1:50010 to delete  blk_7399677479348617513_221053
2010-08-04 06:23:09,067 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
192.168.1.3
2010-08-04 06:28:09,073 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
192.168.1.3
2010-08-04 06:33:09,080 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
192.168.1.3
2010-08-04 06:38:09,086 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
192.168.1.3
2010-08-04 06:43:09,092 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
192.168.1.3
2010-08-04 06:48:09,098 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
192.168.1.3

And bellow is log of a datanode (other datanodes have quite similar log
patten):

2010-08-04 06:21:18,070 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Deleting block
blk_7399677479348617513_221053 file
/home/hadoop/hdfs/data/current/blk_7399677479348617513
2010-08-04 06:23:20,602 INFO
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
succeeded for blk_-1342019480713368018_215616
2010-08-04 06:25:32,919 INFO
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
succeeded for blk_-4748941997410465848_220293
2010-08-04 06:29:49,380 INFO
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
succeeded for blk_4296166818468293227_220300
2010-08-04 06:32:20,033 INFO
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
succeeded for blk_3798415444126765289_220122
2010-08-04 06:32:20,634 INFO
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
succeeded for blk_-8132282518829193527_207589

Any help would be greatly appreciated,

Thanks,

~Hieu