[jira] [Created] (HADOOP-10145) Reduce task stuck on 0.16666667%

["vikash kumar (JIRA)" <ji...@apache.org>]

vikash kumar created HADOOP-10145:
-------------------------------------

             Summary: Reduce task stuck on 0.16666667%
                 Key: HADOOP-10145
                 URL: https://issues.apache.org/jira/browse/HADOOP-10145
             Project: Hadoop Common
          Issue Type: Bug
          Components: conf
    Affects Versions: 0.20.2
         Environment: OS:  RHEL 6.4
Hadoop version:  0.20.2-cdh3u6
            Reporter: vikash kumar


All of sudden, one of the Hadoop jobs is stuck, basically the reduce takes forever to complete(we have waited for 30 hours, usually it takes an hour to complete).
in tasktracker logs i see tons of following messages, however at times, resubmitting the same job works fine. 

2013-12-04 00:00:00,381 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201310070546_159167_r_000041_0 0.16666667% reduce > copy (1 of 2 at 0.01 MB/s) >
2013-12-04 00:00:00,750 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201310070546_159167_r_000048_0 0.16666667% reduce > copy (1 of 2 at 0.01 MB/s) >
2013-12-04 00:00:01,729 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201310070546_159262_r_000046_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
2013-12-04 00:00:01,918 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201310070546_159262_r_000055_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
2013-12-04 00:00:01,919 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201310070546_159262_r_000021_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
2013-12-04 00:00:01,922 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201310070546_159262_r_000031_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
2013-12-04 00:00:01,940 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201310070546_159262_r_000057_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
2013-12-04 00:00:02,443 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201310070546_159167_r_000047_0 0.16666667% reduce > copy (1 of 2 at 0.01 MB/s) >


there are no other resonable clues in log for me to get a direction on, what am i looking for. with my setup, upgrading to new version is not an option.

please help!



--
This message was sent by Atlassian JIRA
(v6.1#6144)