Re: Reduce hangs 2

[Stefan Groschupf <sg...@101tec.com>]

Hi,
not sure if this is the same source of problem, but I also run in  
problems with a hanging reduce.
It is reproducible for me, though I did not find the source of the  
problem yet.
I run a series of jobs and my last job, the last reduce task hangs for  
about 15 to 20 minutes doing nothing, but than resumes. I running  
hadoop 15.1

Below the log entries during the hang. So I think it is not the copy  
problem mentioned before. I also checked our dfs is healthy.


2008-01-22 21:22:09,327 INFO org.apache.hadoop.mapred.ReduceTask:  
task_200801221313_0003_r_000046_1 Need 2 map output(s)
2008-01-22 21:22:09,327 INFO org.apache.hadoop.mapred.ReduceTask:  
task_200801221313_0003_r_000046_1: Got 0 new map-outputs & 0 obsolete  
map-outputs from tasktracker and 0 map-outputs from previous failures
2008-01-22 21:22:09,327 INFO org.apache.hadoop.mapred.ReduceTask:  
task_200801221313_0003_r_000046_1 Got 2 known map output location(s);  
scheduling...
2008-01-22 21:22:09,327 INFO org.apache.hadoop.mapred.ReduceTask:  
task_200801221313_0003_r_000046_1 Scheduled 2 of 2 known outputs (0  
slow hosts and 0 dup hosts)
2008-01-22 21:22:09,327 INFO org.apache.hadoop.mapred.ReduceTask:  
task_200801221313_0003_r_000046_1 Copying  
task_200801221313_0003_m_000035_0 output from hadoop5.dev.company.com.
2008-01-22 21:22:09,328 INFO org.apache.hadoop.mapred.ReduceTask:  
task_200801221313_0003_r_000046_1 Copying  
task_200801221313_0003_m_000040_0 output from hadoop1.dev.company.com.
2008-01-22 21:22:11,243 INFO org.apache.hadoop.mapred.ReduceTask:  
task_200801221313_0003_r_000046_1 done copying  
task_200801221313_0003_m_000040_0 output from hadoop1.dev.company.com.
2008-01-22 21:22:11,610 INFO org.apache.hadoop.mapred.ReduceTask:  
task_200801221313_0003_r_000046_1 done copying  
task_200801221313_0003_m_000035_0 output from hadoop5.dev.company.com.
2008-01-22 21:22:11,611 INFO org.apache.hadoop.mapred.ReduceTask:  
task_200801221313_0003_r_000046_1 Copying of all map outputs complete.  
Initiating the last merge on the remaining files in ramfs:// 
mapoutput169937755
2008-01-22 21:22:11,635 INFO org.apache.hadoop.mapred.ReduceTask:  
task_200801221313_0003_r_000046_1 Merge of the 1 files in  
InMemoryFileSystem complete. Local file is /home/hadoop/data/hadoop- 
hadoop/mapred/local/task_200801221313_0003_r_000046_1/map_34.out

Any ideas? Thanks!
Stefan