You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Arun C Murthy (JIRA)" <ji...@apache.org> on 2009/01/27 09:22:59 UTC
[jira] Created: (HADOOP-5130) TaskTracker seems to hold onto the
assigned task for a long while before launching it
TaskTracker seems to hold onto the assigned task for a long while before launching it
-------------------------------------------------------------------------------------
Key: HADOOP-5130
URL: https://issues.apache.org/jira/browse/HADOOP-5130
Project: Hadoop Core
Issue Type: Bug
Components: mapred
Affects Versions: 0.20.0
Reporter: Arun C Murthy
Priority: Critical
I saw atleast a couple of instances where the task assigned to the TaskTracker is launched several minutes after the receipt of the LaunchTaskAction:
{noformat}
2009-01-27 13:55:53,402 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_200901270818_0006_m_000602_0
2009-01-27 13:55:55,129 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_200901270818_0006_m_000602_0
2009-01-27 13:55:55,129 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_200901270818_0006_m_000602_0
2009-01-27 14:04:17,744 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_200901270818_0006_m_176495965 given task: attempt_200901270818_0006_m_000602_0
2009-01-27 14:04:24,020 INFO org.apache.hadoop.mapred.TaskTracker: attempt_200901270818_0006_m_000602_0 1.0%
2009-01-27 14:04:27,023 INFO org.apache.hadoop.mapred.TaskTracker: attempt_200901270818_0006_m_000602_0 1.0%
2009-01-27 14:04:30,026 INFO org.apache.hadoop.mapred.TaskTracker: attempt_200901270818_0006_m_000602_0 1.0%
2009-01-27 14:04:30,362 INFO org.apache.hadoop.mapred.TaskTracker: attempt_200901270818_0006_m_000602_0 1.0%
2009-01-27 14:04:30,362 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_200901270818_0006_m_000602_0 is done.
2009-01-27 14:04:30,362 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_200901270818_0006_m_000602_0 was 0
{noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5130) TaskTracker seems to hold onto the
assigned task for a long while before launching it
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12667631#action_12667631 ]
Arun C Murthy commented on HADOOP-5130:
---------------------------------------
To clarify - the task did eventually launch and succeed, just the latency is too high (order of several minutes).
> TaskTracker seems to hold onto the assigned task for a long while before launching it
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-5130
> URL: https://issues.apache.org/jira/browse/HADOOP-5130
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Arun C Murthy
> Priority: Critical
> Fix For: 0.20.0
>
>
> I saw atleast a couple of instances where the task assigned to the TaskTracker is launched several minutes after the receipt of the LaunchTaskAction:
> {noformat}
> 2009-01-27 13:55:53,402 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_200901270818_0006_m_000602_0
> 2009-01-27 13:55:55,129 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_200901270818_0006_m_000602_0
> 2009-01-27 13:55:55,129 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_200901270818_0006_m_000602_0
> 2009-01-27 14:04:17,744 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_200901270818_0006_m_176495965 given task: attempt_200901270818_0006_m_000602_0
> 2009-01-27 14:04:24,020 INFO org.apache.hadoop.mapred.TaskTracker: attempt_200901270818_0006_m_000602_0 1.0%
> 2009-01-27 14:04:27,023 INFO org.apache.hadoop.mapred.TaskTracker: attempt_200901270818_0006_m_000602_0 1.0%
> 2009-01-27 14:04:30,026 INFO org.apache.hadoop.mapred.TaskTracker: attempt_200901270818_0006_m_000602_0 1.0%
> 2009-01-27 14:04:30,362 INFO org.apache.hadoop.mapred.TaskTracker: attempt_200901270818_0006_m_000602_0 1.0%
> 2009-01-27 14:04:30,362 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_200901270818_0006_m_000602_0 is done.
> 2009-01-27 14:04:30,362 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_200901270818_0006_m_000602_0 was 0
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5130) TaskTracker seems to hold onto the
assigned task for a long while before launching it
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12667619#action_12667619 ]
Amareshwari Sriramadasu commented on HADOOP-5130:
-------------------------------------------------
I also saw this behavior many times.
And there are cases that these tasks got killed for not reporting progress for 10 mins.
> TaskTracker seems to hold onto the assigned task for a long while before launching it
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-5130
> URL: https://issues.apache.org/jira/browse/HADOOP-5130
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Arun C Murthy
> Priority: Critical
>
> I saw atleast a couple of instances where the task assigned to the TaskTracker is launched several minutes after the receipt of the LaunchTaskAction:
> {noformat}
> 2009-01-27 13:55:53,402 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_200901270818_0006_m_000602_0
> 2009-01-27 13:55:55,129 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_200901270818_0006_m_000602_0
> 2009-01-27 13:55:55,129 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_200901270818_0006_m_000602_0
> 2009-01-27 14:04:17,744 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_200901270818_0006_m_176495965 given task: attempt_200901270818_0006_m_000602_0
> 2009-01-27 14:04:24,020 INFO org.apache.hadoop.mapred.TaskTracker: attempt_200901270818_0006_m_000602_0 1.0%
> 2009-01-27 14:04:27,023 INFO org.apache.hadoop.mapred.TaskTracker: attempt_200901270818_0006_m_000602_0 1.0%
> 2009-01-27 14:04:30,026 INFO org.apache.hadoop.mapred.TaskTracker: attempt_200901270818_0006_m_000602_0 1.0%
> 2009-01-27 14:04:30,362 INFO org.apache.hadoop.mapred.TaskTracker: attempt_200901270818_0006_m_000602_0 1.0%
> 2009-01-27 14:04:30,362 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_200901270818_0006_m_000602_0 is done.
> 2009-01-27 14:04:30,362 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_200901270818_0006_m_000602_0 was 0
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-5130) TaskTracker seems to hold onto the
assigned task for a long while before launching it
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das resolved HADOOP-5130.
---------------------------------
Resolution: Won't Fix
Arun confirmed that this problem was due to the IPV4/V6 issue. This is discussed here - http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200812.mbox/%3CC56ED656.C28C%25runping@yahoo-inc.com%3E
Adding -Djava.net.preferIPv4Stack=true to mapred.child.java.opts fixes the problem.
> TaskTracker seems to hold onto the assigned task for a long while before launching it
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-5130
> URL: https://issues.apache.org/jira/browse/HADOOP-5130
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Arun C Murthy
> Priority: Blocker
> Fix For: 0.20.0
>
>
> I saw atleast a couple of instances where the task assigned to the TaskTracker is launched several minutes after the receipt of the LaunchTaskAction:
> {noformat}
> 2009-01-27 13:55:53,402 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_200901270818_0006_m_000602_0
> 2009-01-27 13:55:55,129 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_200901270818_0006_m_000602_0
> 2009-01-27 13:55:55,129 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_200901270818_0006_m_000602_0
> 2009-01-27 14:04:17,744 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_200901270818_0006_m_176495965 given task: attempt_200901270818_0006_m_000602_0
> 2009-01-27 14:04:24,020 INFO org.apache.hadoop.mapred.TaskTracker: attempt_200901270818_0006_m_000602_0 1.0%
> 2009-01-27 14:04:27,023 INFO org.apache.hadoop.mapred.TaskTracker: attempt_200901270818_0006_m_000602_0 1.0%
> 2009-01-27 14:04:30,026 INFO org.apache.hadoop.mapred.TaskTracker: attempt_200901270818_0006_m_000602_0 1.0%
> 2009-01-27 14:04:30,362 INFO org.apache.hadoop.mapred.TaskTracker: attempt_200901270818_0006_m_000602_0 1.0%
> 2009-01-27 14:04:30,362 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_200901270818_0006_m_000602_0 is done.
> 2009-01-27 14:04:30,362 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_200901270818_0006_m_000602_0 was 0
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5130) TaskTracker seems to hold onto the
assigned task for a long while before launching it
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-5130:
----------------------------------
Priority: Blocker (was: Critical)
Fix Version/s: 0.20.0
Promoting this to a blocker for 0.20.0.
> TaskTracker seems to hold onto the assigned task for a long while before launching it
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-5130
> URL: https://issues.apache.org/jira/browse/HADOOP-5130
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Arun C Murthy
> Priority: Blocker
> Fix For: 0.20.0
>
>
> I saw atleast a couple of instances where the task assigned to the TaskTracker is launched several minutes after the receipt of the LaunchTaskAction:
> {noformat}
> 2009-01-27 13:55:53,402 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_200901270818_0006_m_000602_0
> 2009-01-27 13:55:55,129 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_200901270818_0006_m_000602_0
> 2009-01-27 13:55:55,129 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_200901270818_0006_m_000602_0
> 2009-01-27 14:04:17,744 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_200901270818_0006_m_176495965 given task: attempt_200901270818_0006_m_000602_0
> 2009-01-27 14:04:24,020 INFO org.apache.hadoop.mapred.TaskTracker: attempt_200901270818_0006_m_000602_0 1.0%
> 2009-01-27 14:04:27,023 INFO org.apache.hadoop.mapred.TaskTracker: attempt_200901270818_0006_m_000602_0 1.0%
> 2009-01-27 14:04:30,026 INFO org.apache.hadoop.mapred.TaskTracker: attempt_200901270818_0006_m_000602_0 1.0%
> 2009-01-27 14:04:30,362 INFO org.apache.hadoop.mapred.TaskTracker: attempt_200901270818_0006_m_000602_0 1.0%
> 2009-01-27 14:04:30,362 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_200901270818_0006_m_000602_0 is done.
> 2009-01-27 14:04:30,362 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_200901270818_0006_m_000602_0 was 0
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.