Hadoop strange behaviour! Please help

[Tali K <nc...@hotmail.com>]

Hi all,
 
We were running hadoop for couple month.
It started behave weird. I can't even run a simple wordcount job it stays with Map 0  , Reduce 0 forever :
 
10/12/03 18:05:40 INFO mapred.JobClient:  map 0% reduce 0%.
 
I 've tried to restart hadoop , the same behaviour. (stop-all/start-all)
report showing nothing suspicious, except that 1 node is dead. But hadoop should run despite that some node is dead, isn't it?
It was 10 nodes before. I couldn't find anything suspicious in the logs except some strange LeaseException
hadoop-hadoop-namenode-ti-13.proteinspire.local.log.2010-12-02:org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /tmp/to_delete
Any help is appreciated.
 
Here is report: 
 
Configured Capacity: 17721062572032 (16.12 TB)
Present Capacity: 15938830940883 (14.5 TB)
DFS Remaining: 9830502768640 (8.94 TB)
DFS Used: 6108328172243 (5.56 TB)
DFS Used%: 38.32%
Under replicated blocks: 3
Blocks with corrupt replicas: 0
Missing blocks: 0
-------------------------------------------------
Datanodes available: 9 (9 total, 0 dead)
Name: 10.255.255.139:50010
Decommission Status : Normal
Configured Capacity: 1969006952448 (1.79 TB)
DFS Used: 725233823744 (675.43 GB)
Non DFS Used: 100232089600 (93.35 GB)
DFS Remaining: 1143541039104(1.04 TB)
DFS Used%: 36.83%
DFS Remaining%: 58.08%
Last contact: Fri Dec 03 18:09:06 PST 2010

Name: 10.255.255.145:50010
Decommission Status : Normal
Configured Capacity: 1969006952448 (1.79 TB)
DFS Used: 901043232768 (839.16 GB)
Non DFS Used: 100228562944 (93.35 GB)
DFS Remaining: 967735156736(901.27 GB)
DFS Used%: 45.76%
DFS Remaining%: 49.15%
Last contact: Fri Dec 03 18:09:07 PST 2010

Name: 10.255.255.144:50010
Decommission Status : Normal
Configured Capacity: 1969006952448 (1.79 TB)
DFS Used: 682028363776 (635.19 GB)
Non DFS Used: 100232114176 (93.35 GB)
DFS Remaining: 1186746474496(1.08 TB)
DFS Used%: 34.64%
DFS Remaining%: 60.27%
Last contact: Fri Dec 03 18:09:05 PST 2010

Name: 10.255.255.147:50010
Decommission Status : Normal
Configured Capacity: 1969006952448 (1.79 TB)
DFS Used: 896293472650 (834.74 GB)
Non DFS Used: 100228576886 (93.35 GB)
DFS Remaining: 972484902912(905.7 GB)
DFS Used%: 45.52%
DFS Remaining%: 49.39%
Last contact: Fri Dec 03 18:09:06 PST 2010

Name: 10.255.255.142:50010
Decommission Status : Normal
Configured Capacity: 1969006952448 (1.79 TB)
DFS Used: 732124628542 (681.84 GB)