You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Amit Kumar Singh (JIRA)" <ji...@apache.org> on 2008/05/07 21:26:55 UTC

[jira] Created: (HADOOP-3362) Reduce Task wont complete goes till 16% and halts

Reduce Task wont complete goes till 16% and halts
-------------------------------------------------

                 Key: HADOOP-3362
                 URL: https://issues.apache.org/jira/browse/HADOOP-3362
             Project: Hadoop Core
          Issue Type: Bug
    Affects Versions: 0.16.3
         Environment: Distributor ID: Ubuntu
Description:    Ubuntu 7.10
Release:        7.10
Codename:       gutsy

JDK 1.6

            Reporter: Amit Kumar Singh


I have been trying word count example distributed with Hadoop 0.16.3.
It works fine on single machine mode. But the moment i add an extra slave reduce phase stalls.
Went through some of the post like http://www.mail-archive.com/hadoop-user@lucene.apache.org/msg01688.html
but to no avail.

Can you please give some pointers

Environment
JDK 6.0
Ubuntu

I Get following message in my logs SLAVE
.
2008-05-07 23:37:27,860 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
2008-05-07 23:37:33,862 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
2008-05-07 23:37:39,864 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
2008-05-07 23:37:42,866 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
2008-05-07 23:37:48,868 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
2008-05-07 23:37:54,870 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
2008-05-07 23:38:00,872 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
2008-05-07 23:38:03,873 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
2008-05-07 23:38:09,875 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
2008-05-07 23:38:15,876 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
2008-05-07 23:38:18,878 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
2008-05-07 23:38:24,880 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
2008-05-07 23:38:30,882 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
2008-05-07 23:38:33,883 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
2008-05-07 23:39:18,898 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
2008-05-07 23:39:24,900 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
2008-05-07 23:39:30,902 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
2008-05-07 23:39:33,903 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
2008-05-07 23:39:39,905 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
2008-05-07 23:39:45,907 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
2008-05-07 23:39:48,908 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
2008-05-07 23:39:54,910 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
2008-05-07 23:40:00,912 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
2008-05-07 23:40:03,913 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
2008-05-07 23:40:09,915 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
2008-05-07 23:40:15,917 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
2008-05-07 23:40:18,919 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
2008-05-07 23:40:24,921 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
2008-05-07 23:40:27,120 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction: task_200805080929_0001_m_000001_1
2008-05-07 23:40:28,705 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_m_000001_1 1.0% hdfs://master:54310/user/hadoop/d3:337381+337381
2008-05-07 23:40:28,708 INFO org.apache.hadoop.mapred.TaskTracker: Task task_200805080929_0001_m_000001_1 is done.
2008-05-07 23:40:30,923 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
2008-05-07 23:40:36,925 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.33333334% reduce > copy (2 of 2 at 0.00 MB/s)
2008-05-07 23:40:37,558 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.8684772% reduce > reduce
2008-05-07 23:40:37,559 INFO org.apache.hadoop.mapred.TaskTracker: Task task_200805080929_0001_r_000000_0 is done.
2008-05-07 23:40:39,692 INFO org.apache.hadoop.mapred.TaskTracker: Received 'KillJobAction' for job: job_200805080929_0001
2008-05-07 23:40:39,692 INFO org.apache.hadoop.mapred.TaskRunner: task_200805080929_0001_m_000001_1 done; removing files.
2008-05-07 23:40:39,695 INFO org.apache.hadoop.mapred.TaskRunner: task_200805080929_0001_m_000000_0 done; removing files.
2008-05-07 23:40:39,698 INFO org.apache.hadoop.mapred.TaskRunner: task_200805080929_0001_r_000000_0 done; removing files.
2008-05-07 23:45:49,869 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction: task_200805080929_0002_m_000001_0


And in MASTER(which is also a slave)
008-05-08 09:30:58,991 INFO org.apache.hadoop.mapred.JobInProgress: Task 'task_200805080929_0001_m_000000_0' has completed tip_200805080929_0001_m_000000 successfully.
2008-05-08 09:33:39,111 INFO org.apache.hadoop.mapred.JobInProgress: Failed fetch notification #1 for task task_200805080929_0001_m_000001_0
2008-05-08 09:36:14,303 INFO org.apache.hadoop.conf.Configuration: found resource webapps/static/jobconf.xsl at file:/home/hadoop/HADOOP/hadoop-0.16.3/webapps/static/jobconf.xsl
2008-05-08 09:38:36,511 INFO org.apache.hadoop.mapred.JobInProgress: Failed fetch notification #2 for task task_200805080929_0001_m_000001_0
2008-05-08 09:43:44,540 INFO org.apache.hadoop.mapred.JobInProgress: Failed fetch notification #3 for task task_200805080929_0001_m_000001_0
2008-05-08 09:43:44,540 INFO org.apache.hadoop.mapred.JobInProgress:* Too many fetch-failures for output of task: ta*sk_200805080929_0001_m_000001_0 ... killing it
2008-05-08 09:43:44,540 INFO org.apache.hadoop.mapred.TaskInProgress: Error from task_200805080929_0001_m_000001_0: Too many fetch-failures
2008-05-08 09:43:44,541 INFO org.apache.hadoop.mapred.JobInProgress: Choosing normal task tip_200805080929_0001_m_000001
2008-05-08 09:43:44,541 INFO org.apache.hadoop.mapred.JobTracker: Adding task 'task_200805080929_0001_m_000001_1' to tip tip_200805080929_0001_m_000001, for tracker 'tracker_mtech-desktop:localhost/127.0.0.1:39716'
2008-05-08 09:43:46,695 INFO org.apache.hadoop.mapred.TaskRunner: Saved output of task 'task_200805080929_0001_m_000001_1' to hdfs://master:54310/user/hadoop/d4
2008-05-08 09:43:46,695 INFO org.apache.hadoop.mapred.JobInProgress: Task 'task_200805080929_0001_m_000001_1' has completed tip_200805080929_0001_m_000001 successfully.
2008-05-08 09:43:46,757 INFO org.apache.hadoop.mapred.JobTracker: Removed completed task 'task_200805080929_0001_m_000001_0' from 'tracker_cse


Can any one give some ideas as to what might be the problem.

Configuration of cluster is as per (http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Multi-Node_Cluster%29) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HADOOP-3362) Reduce Task wont complete goes till 16% and halts

Posted by "Amit Kumar Singh (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-3362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amit Kumar Singh updated HADOOP-3362:
-------------------------------------

    Attachment: hadoop-hadoop-secondarynamenode-cse-desktop.log

> Reduce Task wont complete goes till 16% and halts
> -------------------------------------------------
>
>                 Key: HADOOP-3362
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3362
>             Project: Hadoop Core
>          Issue Type: Bug
>    Affects Versions: 0.16.3
>         Environment: Distributor ID: Ubuntu
> Description:    Ubuntu 7.10
> Release:        7.10
> Codename:       gutsy
> JDK 1.6
>            Reporter: Amit Kumar Singh
>         Attachments: hadoop-hadoop-datanode-cse-desktop.log, hadoop-hadoop-jobtracker-cse-desktop.log, hadoop-hadoop-namenode-cse-desktop.log, hadoop-hadoop-secondarynamenode-cse-desktop.log, hadoop-hadoop-tasktracker-cse-desktop.log
>
>
> I have been trying word count example distributed with Hadoop 0.16.3.
> It works fine on single machine mode. But the moment i add an extra slave reduce phase stalls.
> Went through some of the post like http://www.mail-archive.com/hadoop-user@lucene.apache.org/msg01688.html
> but to no avail.
> Can you please give some pointers
> Environment
> JDK 6.0
> Ubuntu
> I Get following message in my logs SLAVE
> .
> 2008-05-07 23:37:27,860 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:37:33,862 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:37:39,864 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:37:42,866 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:37:48,868 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:37:54,870 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:00,872 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:03,873 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:09,875 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:15,876 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:18,878 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:24,880 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:30,882 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:33,883 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:18,898 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:24,900 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:30,902 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:33,903 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:39,905 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:45,907 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:48,908 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:54,910 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:00,912 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:03,913 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:09,915 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:15,917 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:18,919 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:24,921 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:27,120 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction: task_200805080929_0001_m_000001_1
> 2008-05-07 23:40:28,705 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_m_000001_1 1.0% hdfs://master:54310/user/hadoop/d3:337381+337381
> 2008-05-07 23:40:28,708 INFO org.apache.hadoop.mapred.TaskTracker: Task task_200805080929_0001_m_000001_1 is done.
> 2008-05-07 23:40:30,923 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:36,925 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.33333334% reduce > copy (2 of 2 at 0.00 MB/s)
> 2008-05-07 23:40:37,558 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.8684772% reduce > reduce
> 2008-05-07 23:40:37,559 INFO org.apache.hadoop.mapred.TaskTracker: Task task_200805080929_0001_r_000000_0 is done.
> 2008-05-07 23:40:39,692 INFO org.apache.hadoop.mapred.TaskTracker: Received 'KillJobAction' for job: job_200805080929_0001
> 2008-05-07 23:40:39,692 INFO org.apache.hadoop.mapred.TaskRunner: task_200805080929_0001_m_000001_1 done; removing files.
> 2008-05-07 23:40:39,695 INFO org.apache.hadoop.mapred.TaskRunner: task_200805080929_0001_m_000000_0 done; removing files.
> 2008-05-07 23:40:39,698 INFO org.apache.hadoop.mapred.TaskRunner: task_200805080929_0001_r_000000_0 done; removing files.
> 2008-05-07 23:45:49,869 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction: task_200805080929_0002_m_000001_0
> And in MASTER(which is also a slave)
> 008-05-08 09:30:58,991 INFO org.apache.hadoop.mapred.JobInProgress: Task 'task_200805080929_0001_m_000000_0' has completed tip_200805080929_0001_m_000000 successfully.
> 2008-05-08 09:33:39,111 INFO org.apache.hadoop.mapred.JobInProgress: Failed fetch notification #1 for task task_200805080929_0001_m_000001_0
> 2008-05-08 09:36:14,303 INFO org.apache.hadoop.conf.Configuration: found resource webapps/static/jobconf.xsl at file:/home/hadoop/HADOOP/hadoop-0.16.3/webapps/static/jobconf.xsl
> 2008-05-08 09:38:36,511 INFO org.apache.hadoop.mapred.JobInProgress: Failed fetch notification #2 for task task_200805080929_0001_m_000001_0
> 2008-05-08 09:43:44,540 INFO org.apache.hadoop.mapred.JobInProgress: Failed fetch notification #3 for task task_200805080929_0001_m_000001_0
> 2008-05-08 09:43:44,540 INFO org.apache.hadoop.mapred.JobInProgress:* Too many fetch-failures for output of task: ta*sk_200805080929_0001_m_000001_0 ... killing it
> 2008-05-08 09:43:44,540 INFO org.apache.hadoop.mapred.TaskInProgress: Error from task_200805080929_0001_m_000001_0: Too many fetch-failures
> 2008-05-08 09:43:44,541 INFO org.apache.hadoop.mapred.JobInProgress: Choosing normal task tip_200805080929_0001_m_000001
> 2008-05-08 09:43:44,541 INFO org.apache.hadoop.mapred.JobTracker: Adding task 'task_200805080929_0001_m_000001_1' to tip tip_200805080929_0001_m_000001, for tracker 'tracker_mtech-desktop:localhost/127.0.0.1:39716'
> 2008-05-08 09:43:46,695 INFO org.apache.hadoop.mapred.TaskRunner: Saved output of task 'task_200805080929_0001_m_000001_1' to hdfs://master:54310/user/hadoop/d4
> 2008-05-08 09:43:46,695 INFO org.apache.hadoop.mapred.JobInProgress: Task 'task_200805080929_0001_m_000001_1' has completed tip_200805080929_0001_m_000001 successfully.
> 2008-05-08 09:43:46,757 INFO org.apache.hadoop.mapred.JobTracker: Removed completed task 'task_200805080929_0001_m_000001_0' from 'tracker_cse
> Can any one give some ideas as to what might be the problem.
> Configuration of cluster is as per (http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Multi-Node_Cluster%29) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HADOOP-3362) Reduce Task wont complete goes till 16% and halts

Posted by "Amit Kumar Singh (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-3362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amit Kumar Singh updated HADOOP-3362:
-------------------------------------

    Attachment: hadoop-hadoop-jobtracker-cse-desktop.log

> Reduce Task wont complete goes till 16% and halts
> -------------------------------------------------
>
>                 Key: HADOOP-3362
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3362
>             Project: Hadoop Core
>          Issue Type: Bug
>    Affects Versions: 0.16.3
>         Environment: Distributor ID: Ubuntu
> Description:    Ubuntu 7.10
> Release:        7.10
> Codename:       gutsy
> JDK 1.6
>            Reporter: Amit Kumar Singh
>         Attachments: hadoop-hadoop-datanode-cse-desktop.log, hadoop-hadoop-jobtracker-cse-desktop.log, hadoop-hadoop-namenode-cse-desktop.log
>
>
> I have been trying word count example distributed with Hadoop 0.16.3.
> It works fine on single machine mode. But the moment i add an extra slave reduce phase stalls.
> Went through some of the post like http://www.mail-archive.com/hadoop-user@lucene.apache.org/msg01688.html
> but to no avail.
> Can you please give some pointers
> Environment
> JDK 6.0
> Ubuntu
> I Get following message in my logs SLAVE
> .
> 2008-05-07 23:37:27,860 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:37:33,862 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:37:39,864 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:37:42,866 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:37:48,868 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:37:54,870 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:00,872 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:03,873 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:09,875 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:15,876 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:18,878 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:24,880 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:30,882 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:33,883 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:18,898 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:24,900 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:30,902 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:33,903 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:39,905 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:45,907 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:48,908 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:54,910 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:00,912 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:03,913 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:09,915 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:15,917 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:18,919 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:24,921 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:27,120 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction: task_200805080929_0001_m_000001_1
> 2008-05-07 23:40:28,705 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_m_000001_1 1.0% hdfs://master:54310/user/hadoop/d3:337381+337381
> 2008-05-07 23:40:28,708 INFO org.apache.hadoop.mapred.TaskTracker: Task task_200805080929_0001_m_000001_1 is done.
> 2008-05-07 23:40:30,923 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:36,925 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.33333334% reduce > copy (2 of 2 at 0.00 MB/s)
> 2008-05-07 23:40:37,558 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.8684772% reduce > reduce
> 2008-05-07 23:40:37,559 INFO org.apache.hadoop.mapred.TaskTracker: Task task_200805080929_0001_r_000000_0 is done.
> 2008-05-07 23:40:39,692 INFO org.apache.hadoop.mapred.TaskTracker: Received 'KillJobAction' for job: job_200805080929_0001
> 2008-05-07 23:40:39,692 INFO org.apache.hadoop.mapred.TaskRunner: task_200805080929_0001_m_000001_1 done; removing files.
> 2008-05-07 23:40:39,695 INFO org.apache.hadoop.mapred.TaskRunner: task_200805080929_0001_m_000000_0 done; removing files.
> 2008-05-07 23:40:39,698 INFO org.apache.hadoop.mapred.TaskRunner: task_200805080929_0001_r_000000_0 done; removing files.
> 2008-05-07 23:45:49,869 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction: task_200805080929_0002_m_000001_0
> And in MASTER(which is also a slave)
> 008-05-08 09:30:58,991 INFO org.apache.hadoop.mapred.JobInProgress: Task 'task_200805080929_0001_m_000000_0' has completed tip_200805080929_0001_m_000000 successfully.
> 2008-05-08 09:33:39,111 INFO org.apache.hadoop.mapred.JobInProgress: Failed fetch notification #1 for task task_200805080929_0001_m_000001_0
> 2008-05-08 09:36:14,303 INFO org.apache.hadoop.conf.Configuration: found resource webapps/static/jobconf.xsl at file:/home/hadoop/HADOOP/hadoop-0.16.3/webapps/static/jobconf.xsl
> 2008-05-08 09:38:36,511 INFO org.apache.hadoop.mapred.JobInProgress: Failed fetch notification #2 for task task_200805080929_0001_m_000001_0
> 2008-05-08 09:43:44,540 INFO org.apache.hadoop.mapred.JobInProgress: Failed fetch notification #3 for task task_200805080929_0001_m_000001_0
> 2008-05-08 09:43:44,540 INFO org.apache.hadoop.mapred.JobInProgress:* Too many fetch-failures for output of task: ta*sk_200805080929_0001_m_000001_0 ... killing it
> 2008-05-08 09:43:44,540 INFO org.apache.hadoop.mapred.TaskInProgress: Error from task_200805080929_0001_m_000001_0: Too many fetch-failures
> 2008-05-08 09:43:44,541 INFO org.apache.hadoop.mapred.JobInProgress: Choosing normal task tip_200805080929_0001_m_000001
> 2008-05-08 09:43:44,541 INFO org.apache.hadoop.mapred.JobTracker: Adding task 'task_200805080929_0001_m_000001_1' to tip tip_200805080929_0001_m_000001, for tracker 'tracker_mtech-desktop:localhost/127.0.0.1:39716'
> 2008-05-08 09:43:46,695 INFO org.apache.hadoop.mapred.TaskRunner: Saved output of task 'task_200805080929_0001_m_000001_1' to hdfs://master:54310/user/hadoop/d4
> 2008-05-08 09:43:46,695 INFO org.apache.hadoop.mapred.JobInProgress: Task 'task_200805080929_0001_m_000001_1' has completed tip_200805080929_0001_m_000001 successfully.
> 2008-05-08 09:43:46,757 INFO org.apache.hadoop.mapred.JobTracker: Removed completed task 'task_200805080929_0001_m_000001_0' from 'tracker_cse
> Can any one give some ideas as to what might be the problem.
> Configuration of cluster is as per (http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Multi-Node_Cluster%29) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HADOOP-3362) Reduce Task wont complete goes till 16% and halts

Posted by "Amit Kumar Singh (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-3362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amit Kumar Singh updated HADOOP-3362:
-------------------------------------

    Attachment: hadoop-hadoop-namenode-cse-desktop.log

> Reduce Task wont complete goes till 16% and halts
> -------------------------------------------------
>
>                 Key: HADOOP-3362
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3362
>             Project: Hadoop Core
>          Issue Type: Bug
>    Affects Versions: 0.16.3
>         Environment: Distributor ID: Ubuntu
> Description:    Ubuntu 7.10
> Release:        7.10
> Codename:       gutsy
> JDK 1.6
>            Reporter: Amit Kumar Singh
>         Attachments: hadoop-hadoop-datanode-cse-desktop.log, hadoop-hadoop-jobtracker-cse-desktop.log, hadoop-hadoop-namenode-cse-desktop.log
>
>
> I have been trying word count example distributed with Hadoop 0.16.3.
> It works fine on single machine mode. But the moment i add an extra slave reduce phase stalls.
> Went through some of the post like http://www.mail-archive.com/hadoop-user@lucene.apache.org/msg01688.html
> but to no avail.
> Can you please give some pointers
> Environment
> JDK 6.0
> Ubuntu
> I Get following message in my logs SLAVE
> .
> 2008-05-07 23:37:27,860 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:37:33,862 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:37:39,864 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:37:42,866 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:37:48,868 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:37:54,870 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:00,872 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:03,873 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:09,875 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:15,876 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:18,878 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:24,880 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:30,882 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:33,883 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:18,898 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:24,900 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:30,902 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:33,903 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:39,905 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:45,907 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:48,908 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:54,910 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:00,912 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:03,913 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:09,915 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:15,917 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:18,919 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:24,921 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:27,120 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction: task_200805080929_0001_m_000001_1
> 2008-05-07 23:40:28,705 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_m_000001_1 1.0% hdfs://master:54310/user/hadoop/d3:337381+337381
> 2008-05-07 23:40:28,708 INFO org.apache.hadoop.mapred.TaskTracker: Task task_200805080929_0001_m_000001_1 is done.
> 2008-05-07 23:40:30,923 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:36,925 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.33333334% reduce > copy (2 of 2 at 0.00 MB/s)
> 2008-05-07 23:40:37,558 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.8684772% reduce > reduce
> 2008-05-07 23:40:37,559 INFO org.apache.hadoop.mapred.TaskTracker: Task task_200805080929_0001_r_000000_0 is done.
> 2008-05-07 23:40:39,692 INFO org.apache.hadoop.mapred.TaskTracker: Received 'KillJobAction' for job: job_200805080929_0001
> 2008-05-07 23:40:39,692 INFO org.apache.hadoop.mapred.TaskRunner: task_200805080929_0001_m_000001_1 done; removing files.
> 2008-05-07 23:40:39,695 INFO org.apache.hadoop.mapred.TaskRunner: task_200805080929_0001_m_000000_0 done; removing files.
> 2008-05-07 23:40:39,698 INFO org.apache.hadoop.mapred.TaskRunner: task_200805080929_0001_r_000000_0 done; removing files.
> 2008-05-07 23:45:49,869 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction: task_200805080929_0002_m_000001_0
> And in MASTER(which is also a slave)
> 008-05-08 09:30:58,991 INFO org.apache.hadoop.mapred.JobInProgress: Task 'task_200805080929_0001_m_000000_0' has completed tip_200805080929_0001_m_000000 successfully.
> 2008-05-08 09:33:39,111 INFO org.apache.hadoop.mapred.JobInProgress: Failed fetch notification #1 for task task_200805080929_0001_m_000001_0
> 2008-05-08 09:36:14,303 INFO org.apache.hadoop.conf.Configuration: found resource webapps/static/jobconf.xsl at file:/home/hadoop/HADOOP/hadoop-0.16.3/webapps/static/jobconf.xsl
> 2008-05-08 09:38:36,511 INFO org.apache.hadoop.mapred.JobInProgress: Failed fetch notification #2 for task task_200805080929_0001_m_000001_0
> 2008-05-08 09:43:44,540 INFO org.apache.hadoop.mapred.JobInProgress: Failed fetch notification #3 for task task_200805080929_0001_m_000001_0
> 2008-05-08 09:43:44,540 INFO org.apache.hadoop.mapred.JobInProgress:* Too many fetch-failures for output of task: ta*sk_200805080929_0001_m_000001_0 ... killing it
> 2008-05-08 09:43:44,540 INFO org.apache.hadoop.mapred.TaskInProgress: Error from task_200805080929_0001_m_000001_0: Too many fetch-failures
> 2008-05-08 09:43:44,541 INFO org.apache.hadoop.mapred.JobInProgress: Choosing normal task tip_200805080929_0001_m_000001
> 2008-05-08 09:43:44,541 INFO org.apache.hadoop.mapred.JobTracker: Adding task 'task_200805080929_0001_m_000001_1' to tip tip_200805080929_0001_m_000001, for tracker 'tracker_mtech-desktop:localhost/127.0.0.1:39716'
> 2008-05-08 09:43:46,695 INFO org.apache.hadoop.mapred.TaskRunner: Saved output of task 'task_200805080929_0001_m_000001_1' to hdfs://master:54310/user/hadoop/d4
> 2008-05-08 09:43:46,695 INFO org.apache.hadoop.mapred.JobInProgress: Task 'task_200805080929_0001_m_000001_1' has completed tip_200805080929_0001_m_000001 successfully.
> 2008-05-08 09:43:46,757 INFO org.apache.hadoop.mapred.JobTracker: Removed completed task 'task_200805080929_0001_m_000001_0' from 'tracker_cse
> Can any one give some ideas as to what might be the problem.
> Configuration of cluster is as per (http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Multi-Node_Cluster%29) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HADOOP-3362) Reduce Task wont complete goes till 16% and halts

Posted by "Amit Kumar Singh (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-3362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amit Kumar Singh updated HADOOP-3362:
-------------------------------------

    Attachment: hadoop-hadoop-tasktracker-cse-desktop.log

> Reduce Task wont complete goes till 16% and halts
> -------------------------------------------------
>
>                 Key: HADOOP-3362
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3362
>             Project: Hadoop Core
>          Issue Type: Bug
>    Affects Versions: 0.16.3
>         Environment: Distributor ID: Ubuntu
> Description:    Ubuntu 7.10
> Release:        7.10
> Codename:       gutsy
> JDK 1.6
>            Reporter: Amit Kumar Singh
>         Attachments: hadoop-hadoop-datanode-cse-desktop.log, hadoop-hadoop-jobtracker-cse-desktop.log, hadoop-hadoop-namenode-cse-desktop.log, hadoop-hadoop-secondarynamenode-cse-desktop.log, hadoop-hadoop-tasktracker-cse-desktop.log
>
>
> I have been trying word count example distributed with Hadoop 0.16.3.
> It works fine on single machine mode. But the moment i add an extra slave reduce phase stalls.
> Went through some of the post like http://www.mail-archive.com/hadoop-user@lucene.apache.org/msg01688.html
> but to no avail.
> Can you please give some pointers
> Environment
> JDK 6.0
> Ubuntu
> I Get following message in my logs SLAVE
> .
> 2008-05-07 23:37:27,860 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:37:33,862 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:37:39,864 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:37:42,866 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:37:48,868 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:37:54,870 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:00,872 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:03,873 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:09,875 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:15,876 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:18,878 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:24,880 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:30,882 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:33,883 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:18,898 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:24,900 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:30,902 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:33,903 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:39,905 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:45,907 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:48,908 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:54,910 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:00,912 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:03,913 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:09,915 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:15,917 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:18,919 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:24,921 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:27,120 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction: task_200805080929_0001_m_000001_1
> 2008-05-07 23:40:28,705 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_m_000001_1 1.0% hdfs://master:54310/user/hadoop/d3:337381+337381
> 2008-05-07 23:40:28,708 INFO org.apache.hadoop.mapred.TaskTracker: Task task_200805080929_0001_m_000001_1 is done.
> 2008-05-07 23:40:30,923 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:36,925 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.33333334% reduce > copy (2 of 2 at 0.00 MB/s)
> 2008-05-07 23:40:37,558 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.8684772% reduce > reduce
> 2008-05-07 23:40:37,559 INFO org.apache.hadoop.mapred.TaskTracker: Task task_200805080929_0001_r_000000_0 is done.
> 2008-05-07 23:40:39,692 INFO org.apache.hadoop.mapred.TaskTracker: Received 'KillJobAction' for job: job_200805080929_0001
> 2008-05-07 23:40:39,692 INFO org.apache.hadoop.mapred.TaskRunner: task_200805080929_0001_m_000001_1 done; removing files.
> 2008-05-07 23:40:39,695 INFO org.apache.hadoop.mapred.TaskRunner: task_200805080929_0001_m_000000_0 done; removing files.
> 2008-05-07 23:40:39,698 INFO org.apache.hadoop.mapred.TaskRunner: task_200805080929_0001_r_000000_0 done; removing files.
> 2008-05-07 23:45:49,869 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction: task_200805080929_0002_m_000001_0
> And in MASTER(which is also a slave)
> 008-05-08 09:30:58,991 INFO org.apache.hadoop.mapred.JobInProgress: Task 'task_200805080929_0001_m_000000_0' has completed tip_200805080929_0001_m_000000 successfully.
> 2008-05-08 09:33:39,111 INFO org.apache.hadoop.mapred.JobInProgress: Failed fetch notification #1 for task task_200805080929_0001_m_000001_0
> 2008-05-08 09:36:14,303 INFO org.apache.hadoop.conf.Configuration: found resource webapps/static/jobconf.xsl at file:/home/hadoop/HADOOP/hadoop-0.16.3/webapps/static/jobconf.xsl
> 2008-05-08 09:38:36,511 INFO org.apache.hadoop.mapred.JobInProgress: Failed fetch notification #2 for task task_200805080929_0001_m_000001_0
> 2008-05-08 09:43:44,540 INFO org.apache.hadoop.mapred.JobInProgress: Failed fetch notification #3 for task task_200805080929_0001_m_000001_0
> 2008-05-08 09:43:44,540 INFO org.apache.hadoop.mapred.JobInProgress:* Too many fetch-failures for output of task: ta*sk_200805080929_0001_m_000001_0 ... killing it
> 2008-05-08 09:43:44,540 INFO org.apache.hadoop.mapred.TaskInProgress: Error from task_200805080929_0001_m_000001_0: Too many fetch-failures
> 2008-05-08 09:43:44,541 INFO org.apache.hadoop.mapred.JobInProgress: Choosing normal task tip_200805080929_0001_m_000001
> 2008-05-08 09:43:44,541 INFO org.apache.hadoop.mapred.JobTracker: Adding task 'task_200805080929_0001_m_000001_1' to tip tip_200805080929_0001_m_000001, for tracker 'tracker_mtech-desktop:localhost/127.0.0.1:39716'
> 2008-05-08 09:43:46,695 INFO org.apache.hadoop.mapred.TaskRunner: Saved output of task 'task_200805080929_0001_m_000001_1' to hdfs://master:54310/user/hadoop/d4
> 2008-05-08 09:43:46,695 INFO org.apache.hadoop.mapred.JobInProgress: Task 'task_200805080929_0001_m_000001_1' has completed tip_200805080929_0001_m_000001 successfully.
> 2008-05-08 09:43:46,757 INFO org.apache.hadoop.mapred.JobTracker: Removed completed task 'task_200805080929_0001_m_000001_0' from 'tracker_cse
> Can any one give some ideas as to what might be the problem.
> Configuration of cluster is as per (http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Multi-Node_Cluster%29) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HADOOP-3362) Reduce Task wont complete goes till 16% and halts

Posted by "Amit Kumar Singh (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-3362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amit Kumar Singh updated HADOOP-3362:
-------------------------------------

    Attachment: hadoop-hadoop-datanode-cse-desktop.log

> Reduce Task wont complete goes till 16% and halts
> -------------------------------------------------
>
>                 Key: HADOOP-3362
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3362
>             Project: Hadoop Core
>          Issue Type: Bug
>    Affects Versions: 0.16.3
>         Environment: Distributor ID: Ubuntu
> Description:    Ubuntu 7.10
> Release:        7.10
> Codename:       gutsy
> JDK 1.6
>            Reporter: Amit Kumar Singh
>         Attachments: hadoop-hadoop-datanode-cse-desktop.log
>
>
> I have been trying word count example distributed with Hadoop 0.16.3.
> It works fine on single machine mode. But the moment i add an extra slave reduce phase stalls.
> Went through some of the post like http://www.mail-archive.com/hadoop-user@lucene.apache.org/msg01688.html
> but to no avail.
> Can you please give some pointers
> Environment
> JDK 6.0
> Ubuntu
> I Get following message in my logs SLAVE
> .
> 2008-05-07 23:37:27,860 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:37:33,862 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:37:39,864 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:37:42,866 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:37:48,868 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:37:54,870 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:00,872 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:03,873 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:09,875 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:15,876 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:18,878 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:24,880 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:30,882 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:33,883 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:18,898 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:24,900 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:30,902 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:33,903 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:39,905 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:45,907 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:48,908 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:54,910 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:00,912 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:03,913 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:09,915 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:15,917 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:18,919 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:24,921 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:27,120 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction: task_200805080929_0001_m_000001_1
> 2008-05-07 23:40:28,705 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_m_000001_1 1.0% hdfs://master:54310/user/hadoop/d3:337381+337381
> 2008-05-07 23:40:28,708 INFO org.apache.hadoop.mapred.TaskTracker: Task task_200805080929_0001_m_000001_1 is done.
> 2008-05-07 23:40:30,923 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:36,925 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.33333334% reduce > copy (2 of 2 at 0.00 MB/s)
> 2008-05-07 23:40:37,558 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.8684772% reduce > reduce
> 2008-05-07 23:40:37,559 INFO org.apache.hadoop.mapred.TaskTracker: Task task_200805080929_0001_r_000000_0 is done.
> 2008-05-07 23:40:39,692 INFO org.apache.hadoop.mapred.TaskTracker: Received 'KillJobAction' for job: job_200805080929_0001
> 2008-05-07 23:40:39,692 INFO org.apache.hadoop.mapred.TaskRunner: task_200805080929_0001_m_000001_1 done; removing files.
> 2008-05-07 23:40:39,695 INFO org.apache.hadoop.mapred.TaskRunner: task_200805080929_0001_m_000000_0 done; removing files.
> 2008-05-07 23:40:39,698 INFO org.apache.hadoop.mapred.TaskRunner: task_200805080929_0001_r_000000_0 done; removing files.
> 2008-05-07 23:45:49,869 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction: task_200805080929_0002_m_000001_0
> And in MASTER(which is also a slave)
> 008-05-08 09:30:58,991 INFO org.apache.hadoop.mapred.JobInProgress: Task 'task_200805080929_0001_m_000000_0' has completed tip_200805080929_0001_m_000000 successfully.
> 2008-05-08 09:33:39,111 INFO org.apache.hadoop.mapred.JobInProgress: Failed fetch notification #1 for task task_200805080929_0001_m_000001_0
> 2008-05-08 09:36:14,303 INFO org.apache.hadoop.conf.Configuration: found resource webapps/static/jobconf.xsl at file:/home/hadoop/HADOOP/hadoop-0.16.3/webapps/static/jobconf.xsl
> 2008-05-08 09:38:36,511 INFO org.apache.hadoop.mapred.JobInProgress: Failed fetch notification #2 for task task_200805080929_0001_m_000001_0
> 2008-05-08 09:43:44,540 INFO org.apache.hadoop.mapred.JobInProgress: Failed fetch notification #3 for task task_200805080929_0001_m_000001_0
> 2008-05-08 09:43:44,540 INFO org.apache.hadoop.mapred.JobInProgress:* Too many fetch-failures for output of task: ta*sk_200805080929_0001_m_000001_0 ... killing it
> 2008-05-08 09:43:44,540 INFO org.apache.hadoop.mapred.TaskInProgress: Error from task_200805080929_0001_m_000001_0: Too many fetch-failures
> 2008-05-08 09:43:44,541 INFO org.apache.hadoop.mapred.JobInProgress: Choosing normal task tip_200805080929_0001_m_000001
> 2008-05-08 09:43:44,541 INFO org.apache.hadoop.mapred.JobTracker: Adding task 'task_200805080929_0001_m_000001_1' to tip tip_200805080929_0001_m_000001, for tracker 'tracker_mtech-desktop:localhost/127.0.0.1:39716'
> 2008-05-08 09:43:46,695 INFO org.apache.hadoop.mapred.TaskRunner: Saved output of task 'task_200805080929_0001_m_000001_1' to hdfs://master:54310/user/hadoop/d4
> 2008-05-08 09:43:46,695 INFO org.apache.hadoop.mapred.JobInProgress: Task 'task_200805080929_0001_m_000001_1' has completed tip_200805080929_0001_m_000001 successfully.
> 2008-05-08 09:43:46,757 INFO org.apache.hadoop.mapred.JobTracker: Removed completed task 'task_200805080929_0001_m_000001_0' from 'tracker_cse
> Can any one give some ideas as to what might be the problem.
> Configuration of cluster is as per (http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Multi-Node_Cluster%29) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HADOOP-3362) Reduce Task wont complete goes till 16% and halts

Posted by "Leon Mergen (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-3362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12618340#action_12618340 ] 

Leon Mergen commented on HADOOP-3362:
-------------------------------------

Hello,

Just to let you know: I'm having the same issues. As long as I have a semi-distributed 1 node setup, everything goes fine; it stalls with similar problems while Reduce'ing.

Did you manage to find the cause of the problem, and/or a fix/workaround?

> Reduce Task wont complete goes till 16% and halts
> -------------------------------------------------
>
>                 Key: HADOOP-3362
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3362
>             Project: Hadoop Core
>          Issue Type: Bug
>    Affects Versions: 0.16.3
>         Environment: Distributor ID: Ubuntu
> Description:    Ubuntu 7.10
> Release:        7.10
> Codename:       gutsy
> JDK 1.6
>            Reporter: Amit Kumar Singh
>         Attachments: hadoop-hadoop-datanode-cse-desktop.log, hadoop-hadoop-jobtracker-cse-desktop.log, hadoop-hadoop-namenode-cse-desktop.log, hadoop-hadoop-secondarynamenode-cse-desktop.log, hadoop-hadoop-tasktracker-cse-desktop.log
>
>
> I have been trying word count example distributed with Hadoop 0.16.3.
> It works fine on single machine mode. But the moment i add an extra slave reduce phase stalls.
> Went through some of the post like http://www.mail-archive.com/hadoop-user@lucene.apache.org/msg01688.html
> but to no avail.
> Can you please give some pointers
> Environment
> JDK 6.0
> Ubuntu
> I Get following message in my logs SLAVE
> .
> 2008-05-07 23:37:27,860 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:37:33,862 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:37:39,864 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:37:42,866 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:37:48,868 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:37:54,870 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:00,872 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:03,873 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:09,875 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:15,876 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:18,878 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:24,880 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:30,882 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:33,883 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:18,898 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:24,900 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:30,902 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:33,903 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:39,905 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:45,907 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:48,908 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:54,910 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:00,912 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:03,913 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:09,915 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:15,917 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:18,919 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:24,921 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:27,120 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction: task_200805080929_0001_m_000001_1
> 2008-05-07 23:40:28,705 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_m_000001_1 1.0% hdfs://master:54310/user/hadoop/d3:337381+337381
> 2008-05-07 23:40:28,708 INFO org.apache.hadoop.mapred.TaskTracker: Task task_200805080929_0001_m_000001_1 is done.
> 2008-05-07 23:40:30,923 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:36,925 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.33333334% reduce > copy (2 of 2 at 0.00 MB/s)
> 2008-05-07 23:40:37,558 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.8684772% reduce > reduce
> 2008-05-07 23:40:37,559 INFO org.apache.hadoop.mapred.TaskTracker: Task task_200805080929_0001_r_000000_0 is done.
> 2008-05-07 23:40:39,692 INFO org.apache.hadoop.mapred.TaskTracker: Received 'KillJobAction' for job: job_200805080929_0001
> 2008-05-07 23:40:39,692 INFO org.apache.hadoop.mapred.TaskRunner: task_200805080929_0001_m_000001_1 done; removing files.
> 2008-05-07 23:40:39,695 INFO org.apache.hadoop.mapred.TaskRunner: task_200805080929_0001_m_000000_0 done; removing files.
> 2008-05-07 23:40:39,698 INFO org.apache.hadoop.mapred.TaskRunner: task_200805080929_0001_r_000000_0 done; removing files.
> 2008-05-07 23:45:49,869 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction: task_200805080929_0002_m_000001_0
> And in MASTER(which is also a slave)
> 008-05-08 09:30:58,991 INFO org.apache.hadoop.mapred.JobInProgress: Task 'task_200805080929_0001_m_000000_0' has completed tip_200805080929_0001_m_000000 successfully.
> 2008-05-08 09:33:39,111 INFO org.apache.hadoop.mapred.JobInProgress: Failed fetch notification #1 for task task_200805080929_0001_m_000001_0
> 2008-05-08 09:36:14,303 INFO org.apache.hadoop.conf.Configuration: found resource webapps/static/jobconf.xsl at file:/home/hadoop/HADOOP/hadoop-0.16.3/webapps/static/jobconf.xsl
> 2008-05-08 09:38:36,511 INFO org.apache.hadoop.mapred.JobInProgress: Failed fetch notification #2 for task task_200805080929_0001_m_000001_0
> 2008-05-08 09:43:44,540 INFO org.apache.hadoop.mapred.JobInProgress: Failed fetch notification #3 for task task_200805080929_0001_m_000001_0
> 2008-05-08 09:43:44,540 INFO org.apache.hadoop.mapred.JobInProgress:* Too many fetch-failures for output of task: ta*sk_200805080929_0001_m_000001_0 ... killing it
> 2008-05-08 09:43:44,540 INFO org.apache.hadoop.mapred.TaskInProgress: Error from task_200805080929_0001_m_000001_0: Too many fetch-failures
> 2008-05-08 09:43:44,541 INFO org.apache.hadoop.mapred.JobInProgress: Choosing normal task tip_200805080929_0001_m_000001
> 2008-05-08 09:43:44,541 INFO org.apache.hadoop.mapred.JobTracker: Adding task 'task_200805080929_0001_m_000001_1' to tip tip_200805080929_0001_m_000001, for tracker 'tracker_mtech-desktop:localhost/127.0.0.1:39716'
> 2008-05-08 09:43:46,695 INFO org.apache.hadoop.mapred.TaskRunner: Saved output of task 'task_200805080929_0001_m_000001_1' to hdfs://master:54310/user/hadoop/d4
> 2008-05-08 09:43:46,695 INFO org.apache.hadoop.mapred.JobInProgress: Task 'task_200805080929_0001_m_000001_1' has completed tip_200805080929_0001_m_000001 successfully.
> 2008-05-08 09:43:46,757 INFO org.apache.hadoop.mapred.JobTracker: Removed completed task 'task_200805080929_0001_m_000001_0' from 'tracker_cse
> Can any one give some ideas as to what might be the problem.
> Configuration of cluster is as per (http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Multi-Node_Cluster%29) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Issue Comment Edited: (HADOOP-3362) Reduce Task wont complete goes till 16% and halts

Posted by "Leon Mergen (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-3362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12618340#action_12618340 ] 

solatis edited comment on HADOOP-3362 at 7/30/08 7:05 AM:
--------------------------------------------------------------

Hello,

Just to let you know: I'm having the same issues. As long as I have a semi-distributed 1 node setup, everything goes fine; it stalls with similar problems while Reduce'ing when I make a 2-node setup of it.

Did you manage to find the cause of the problem, and/or a fix/workaround?

      was (Author: solatis):
    Hello,

Just to let you know: I'm having the same issues. As long as I have a semi-distributed 1 node setup, everything goes fine; it stalls with similar problems while Reduce'ing.

Did you manage to find the cause of the problem, and/or a fix/workaround?
  
> Reduce Task wont complete goes till 16% and halts
> -------------------------------------------------
>
>                 Key: HADOOP-3362
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3362
>             Project: Hadoop Core
>          Issue Type: Bug
>    Affects Versions: 0.16.3
>         Environment: Distributor ID: Ubuntu
> Description:    Ubuntu 7.10
> Release:        7.10
> Codename:       gutsy
> JDK 1.6
>            Reporter: Amit Kumar Singh
>         Attachments: hadoop-hadoop-datanode-cse-desktop.log, hadoop-hadoop-jobtracker-cse-desktop.log, hadoop-hadoop-namenode-cse-desktop.log, hadoop-hadoop-secondarynamenode-cse-desktop.log, hadoop-hadoop-tasktracker-cse-desktop.log
>
>
> I have been trying word count example distributed with Hadoop 0.16.3.
> It works fine on single machine mode. But the moment i add an extra slave reduce phase stalls.
> Went through some of the post like http://www.mail-archive.com/hadoop-user@lucene.apache.org/msg01688.html
> but to no avail.
> Can you please give some pointers
> Environment
> JDK 6.0
> Ubuntu
> I Get following message in my logs SLAVE
> .
> 2008-05-07 23:37:27,860 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:37:33,862 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:37:39,864 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:37:42,866 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:37:48,868 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:37:54,870 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:00,872 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:03,873 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:09,875 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:15,876 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:18,878 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:24,880 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:30,882 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:38:33,883 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:18,898 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:24,900 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:30,902 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:33,903 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:39,905 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:45,907 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:48,908 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:39:54,910 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:00,912 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:03,913 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:09,915 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:15,917 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:18,919 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:24,921 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:27,120 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction: task_200805080929_0001_m_000001_1
> 2008-05-07 23:40:28,705 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_m_000001_1 1.0% hdfs://master:54310/user/hadoop/d3:337381+337381
> 2008-05-07 23:40:28,708 INFO org.apache.hadoop.mapred.TaskTracker: Task task_200805080929_0001_m_000001_1 is done.
> 2008-05-07 23:40:30,923 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.16666667% reduce > copy (1 of 2 at 0.03 MB/s) >
> 2008-05-07 23:40:36,925 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.33333334% reduce > copy (2 of 2 at 0.00 MB/s)
> 2008-05-07 23:40:37,558 INFO org.apache.hadoop.mapred.TaskTracker: task_200805080929_0001_r_000000_0 0.8684772% reduce > reduce
> 2008-05-07 23:40:37,559 INFO org.apache.hadoop.mapred.TaskTracker: Task task_200805080929_0001_r_000000_0 is done.
> 2008-05-07 23:40:39,692 INFO org.apache.hadoop.mapred.TaskTracker: Received 'KillJobAction' for job: job_200805080929_0001
> 2008-05-07 23:40:39,692 INFO org.apache.hadoop.mapred.TaskRunner: task_200805080929_0001_m_000001_1 done; removing files.
> 2008-05-07 23:40:39,695 INFO org.apache.hadoop.mapred.TaskRunner: task_200805080929_0001_m_000000_0 done; removing files.
> 2008-05-07 23:40:39,698 INFO org.apache.hadoop.mapred.TaskRunner: task_200805080929_0001_r_000000_0 done; removing files.
> 2008-05-07 23:45:49,869 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction: task_200805080929_0002_m_000001_0
> And in MASTER(which is also a slave)
> 008-05-08 09:30:58,991 INFO org.apache.hadoop.mapred.JobInProgress: Task 'task_200805080929_0001_m_000000_0' has completed tip_200805080929_0001_m_000000 successfully.
> 2008-05-08 09:33:39,111 INFO org.apache.hadoop.mapred.JobInProgress: Failed fetch notification #1 for task task_200805080929_0001_m_000001_0
> 2008-05-08 09:36:14,303 INFO org.apache.hadoop.conf.Configuration: found resource webapps/static/jobconf.xsl at file:/home/hadoop/HADOOP/hadoop-0.16.3/webapps/static/jobconf.xsl
> 2008-05-08 09:38:36,511 INFO org.apache.hadoop.mapred.JobInProgress: Failed fetch notification #2 for task task_200805080929_0001_m_000001_0
> 2008-05-08 09:43:44,540 INFO org.apache.hadoop.mapred.JobInProgress: Failed fetch notification #3 for task task_200805080929_0001_m_000001_0
> 2008-05-08 09:43:44,540 INFO org.apache.hadoop.mapred.JobInProgress:* Too many fetch-failures for output of task: ta*sk_200805080929_0001_m_000001_0 ... killing it
> 2008-05-08 09:43:44,540 INFO org.apache.hadoop.mapred.TaskInProgress: Error from task_200805080929_0001_m_000001_0: Too many fetch-failures
> 2008-05-08 09:43:44,541 INFO org.apache.hadoop.mapred.JobInProgress: Choosing normal task tip_200805080929_0001_m_000001
> 2008-05-08 09:43:44,541 INFO org.apache.hadoop.mapred.JobTracker: Adding task 'task_200805080929_0001_m_000001_1' to tip tip_200805080929_0001_m_000001, for tracker 'tracker_mtech-desktop:localhost/127.0.0.1:39716'
> 2008-05-08 09:43:46,695 INFO org.apache.hadoop.mapred.TaskRunner: Saved output of task 'task_200805080929_0001_m_000001_1' to hdfs://master:54310/user/hadoop/d4
> 2008-05-08 09:43:46,695 INFO org.apache.hadoop.mapred.JobInProgress: Task 'task_200805080929_0001_m_000001_1' has completed tip_200805080929_0001_m_000001 successfully.
> 2008-05-08 09:43:46,757 INFO org.apache.hadoop.mapred.JobTracker: Removed completed task 'task_200805080929_0001_m_000001_0' from 'tracker_cse
> Can any one give some ideas as to what might be the problem.
> Configuration of cluster is as per (http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Multi-Node_Cluster%29) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.