You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Owen O'Malley (JIRA)" <ji...@apache.org> on 2006/05/01 20:46:46 UTC
[jira] Created: (HADOOP-181) task trackers should not restart for
having a late heartbeat
task trackers should not restart for having a late heartbeat
------------------------------------------------------------
Key: HADOOP-181
URL: http://issues.apache.org/jira/browse/HADOOP-181
Project: Hadoop
Type: Bug
Components: mapred
Versions: 0.1.1
Reporter: Owen O'Malley
Assigned to: Owen O'Malley
TaskTrackers should not close and restart themselves for having a late heartbeat. The JobTracker should just accept their current status.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-181) task trackers should not restart for
having a late heartbeat
Posted by "Sameer Paranjpye (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-181?page=comments#action_12427033 ]
Sameer Paranjpye commented on HADOOP-181:
-----------------------------------------
I feel that improved detection of tasktracker death is a separate issue, which needs addressing. At the same time, we need to try and not lose work if communication between a tasktracker and the jobtracker fails for some reason.
For instance, a tasktracker may appear lost to the jobtracker due to transient network problems. In such a case it would be ok for the jobtracker to mark the lost tasks as failed and reschedule them to other places.
If communication to the jobtracker is subsequently restored, while the job is still in progress, the jobtracker can
easily mark the lost and found tasks as succeeded. Multiple instances of a task should be handled by the speculative execution code. It seems like we could avoid losing a lot of work if we had such as mechanism in place.
> task trackers should not restart for having a late heartbeat
> ------------------------------------------------------------
>
> Key: HADOOP-181
> URL: http://issues.apache.org/jira/browse/HADOOP-181
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Fix For: 0.6.0
>
> Attachments: lost-heartbeat.patch
>
>
> TaskTrackers should not close and restart themselves for having a late heartbeat. The JobTracker should just accept their current status.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-181) task trackers should not restart for
having a late heartbeat
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-181?page=comments#action_12427954 ]
Doug Cutting commented on HADOOP-181:
-------------------------------------
> If a switch goes down for 15 minutes [ ... ]
We'll currently have a lot of other problems if a switch goes down for 15 minutes. All of the other tasks will probably fail because DFS will no longer have complete copies of files.
Is a switch going down for 15 minutes really a case we need to optimize? Is it acceptable to lose a few hours work on its hosts when a switch dies? When a switch fails, how long does it take to replace?
We can answer some of this fairly precisely. What is the MTBF for switches? How many switches would we have in a 10k-node system?
> task trackers should not restart for having a late heartbeat
> ------------------------------------------------------------
>
> Key: HADOOP-181
> URL: http://issues.apache.org/jira/browse/HADOOP-181
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Fix For: 0.6.0
>
> Attachments: lost-heartbeat.patch
>
>
> TaskTrackers should not close and restart themselves for having a late heartbeat. The JobTracker should just accept their current status.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] Commented: (HADOOP-181) task trackers should not restart for having a late heartbeat
Posted by Eric Baldeschwieler <er...@yahoo-inc.com>.
(RESEND, MENT TO ATTACH THE COMMENT BELOW TO THIS POSTING)
Why don't we include documenting this as part of the the "map-reduce
walk-through" sprint item?
-----
On reintegrating lost task trackers...
It does seem like we should do this to me, but we need to make sure
we reason through how this effects corner cases, what invariants the
system does maintain and so on.
I suggest we work this through, and then go forward with this patch
(modified if we find any corner cases) and post the reasoning, so we
can review it as this logic evolves. (And update any existing
documentation in this area of course...)
On Aug 10, 2006, at 12:38 PM, Devaraj Das (JIRA) wrote:
> [ http://issues.apache.org/jira/browse/HADOOP-181?
> page=comments#action_12427327 ]
>
> Devaraj Das commented on HADOOP-181:
> ------------------------------------
>
> Doug, does it make sense to do what is done in this patch only when
> speculative execution is on?
>
>> task trackers should not restart for having a late heartbeat
>> ------------------------------------------------------------
>>
>> Key: HADOOP-181
>> URL: http://issues.apache.org/jira/browse/HADOOP-181
>> Project: Hadoop
>> Issue Type: Bug
>> Components: mapred
>> Reporter: Owen O'Malley
>> Assigned To: Devaraj Das
>> Fix For: 0.6.0
>>
>> Attachments: lost-heartbeat.patch
>>
>>
>> TaskTrackers should not close and restart themselves for having a
>> late heartbeat. The JobTracker should just accept their current
>> status.
>
> --
> This message is automatically generated by JIRA.
> -
> If you think it was sent incorrectly contact one of the
> administrators: http://issues.apache.org/jira/secure/
> Administrators.jspa
> -
> For more information on JIRA, see: http://www.atlassian.com/
> software/jira
>
>
Re: [jira] Commented: (HADOOP-181) task trackers should not restart for having a late heartbeat
Posted by Eric Baldeschwieler <er...@yahoo-inc.com>.
Why don't we include documenting this as part of the the "map-reduce
walk-through" sprint item?
-----
Oh a whole thread can be had on this I'm sure!
Why does one turn off speculative execution? Presumably because a
MAP has unmanaged side-effects?
But... the framework still will rerun jobs if they complete and then
the node is lost, right? Won't this tickle exactly the same issues
that speculative execution raised anyway?
Doesn't this imply that disallowing speculative execution is
basically not the right mechanism to deal with the side-effect issue
and that this deserves a rethink?
On Aug 10, 2006, at 12:38 PM, Devaraj Das (JIRA) wrote:
> [ http://issues.apache.org/jira/browse/HADOOP-181?
> page=comments#action_12427327 ]
>
> Devaraj Das commented on HADOOP-181:
> ------------------------------------
>
> Doug, does it make sense to do what is done in this patch only when
> speculative execution is on?
>
>> task trackers should not restart for having a late heartbeat
>> ------------------------------------------------------------
>>
>> Key: HADOOP-181
>> URL: http://issues.apache.org/jira/browse/HADOOP-181
>> Project: Hadoop
>> Issue Type: Bug
>> Components: mapred
>> Reporter: Owen O'Malley
>> Assigned To: Devaraj Das
>> Fix For: 0.6.0
>>
>> Attachments: lost-heartbeat.patch
>>
>>
>> TaskTrackers should not close and restart themselves for having a
>> late heartbeat. The JobTracker should just accept their current
>> status.
>
> --
> This message is automatically generated by JIRA.
> -
> If you think it was sent incorrectly contact one of the
> administrators: http://issues.apache.org/jira/secure/
> Administrators.jspa
> -
> For more information on JIRA, see: http://www.atlassian.com/
> software/jira
>
>
[jira] Commented: (HADOOP-181) task trackers should not restart for
having a late heartbeat
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-181?page=comments#action_12427327 ]
Devaraj Das commented on HADOOP-181:
------------------------------------
Doug, does it make sense to do what is done in this patch only when speculative execution is on?
> task trackers should not restart for having a late heartbeat
> ------------------------------------------------------------
>
> Key: HADOOP-181
> URL: http://issues.apache.org/jira/browse/HADOOP-181
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Fix For: 0.6.0
>
> Attachments: lost-heartbeat.patch
>
>
> TaskTrackers should not close and restart themselves for having a late heartbeat. The JobTracker should just accept their current status.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-181) task trackers should not restart for
having a late heartbeat
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-181?page=all ]
Devaraj Das updated HADOOP-181:
-------------------------------
Status: Patch Available (was: Open)
Affects Version/s: (was: 0.1.1)
> task trackers should not restart for having a late heartbeat
> ------------------------------------------------------------
>
> Key: HADOOP-181
> URL: http://issues.apache.org/jira/browse/HADOOP-181
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Fix For: 0.6.0
>
> Attachments: lost-heartbeat.patch
>
>
> TaskTrackers should not close and restart themselves for having a late heartbeat. The JobTracker should just accept their current status.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-181) task trackers should not restart for
having a late heartbeat
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-181?page=comments#action_12426909 ]
Devaraj Das commented on HADOOP-181:
------------------------------------
I am open to a discussion on this but I thought that while we are discussing the possible solutions to the TaskTracker heartbeat problem in Hadoop-312 (I am actually testing a patch with increased queue length + retries), I think we can consider this patch as a solution to the problem raised in this Jira issue. Doesn't harm I think.
> task trackers should not restart for having a late heartbeat
> ------------------------------------------------------------
>
> Key: HADOOP-181
> URL: http://issues.apache.org/jira/browse/HADOOP-181
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Fix For: 0.6.0
>
> Attachments: lost-heartbeat.patch
>
>
> TaskTrackers should not close and restart themselves for having a late heartbeat. The JobTracker should just accept their current status.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-181) task trackers should not restart for
having a late heartbeat
Posted by "Sameer Paranjpye (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-181?page=all ]
Sameer Paranjpye updated HADOOP-181:
------------------------------------
Fix Version: 0.3
> task trackers should not restart for having a late heartbeat
> ------------------------------------------------------------
>
> Key: HADOOP-181
> URL: http://issues.apache.org/jira/browse/HADOOP-181
> Project: Hadoop
> Type: Bug
> Components: mapred
> Versions: 0.1.1
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Fix For: 0.3
>
> TaskTrackers should not close and restart themselves for having a late heartbeat. The JobTracker should just accept their current status.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-181) task trackers should not restart for
having a late heartbeat
Posted by "eric baldeschwieler (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-181?page=comments#action_12427052 ]
eric baldeschwieler commented on HADOOP-181:
--------------------------------------------
Would this have a substantial impact on the large sort benchmark tests our team runs?
> task trackers should not restart for having a late heartbeat
> ------------------------------------------------------------
>
> Key: HADOOP-181
> URL: http://issues.apache.org/jira/browse/HADOOP-181
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Fix For: 0.6.0
>
> Attachments: lost-heartbeat.patch
>
>
> TaskTrackers should not close and restart themselves for having a late heartbeat. The JobTracker should just accept their current status.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-181) task trackers should not restart for
having a late heartbeat
Posted by "Yoram Arnon (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-181?page=comments#action_12427126 ]
Yoram Arnon commented on HADOOP-181:
------------------------------------
Eric, we don't get many lost task trackers in our sort benchmark any more, so I don't expect a significant improvement.
In other jobs though, we do get the occasional lost task tracker. While every such case is a bug when nothing 'real' happens (HW error, network etc.) hunting every last one of them down is going to take quite a while.
Minimizing the damage caused by such a glitch would be welcome if it doesn't cause adverse side effects.
> task trackers should not restart for having a late heartbeat
> ------------------------------------------------------------
>
> Key: HADOOP-181
> URL: http://issues.apache.org/jira/browse/HADOOP-181
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Fix For: 0.6.0
>
> Attachments: lost-heartbeat.patch
>
>
> TaskTrackers should not close and restart themselves for having a late heartbeat. The JobTracker should just accept their current status.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-181) task trackers should not restart for
having a late heartbeat
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-181?page=all ]
Doug Cutting updated HADOOP-181:
--------------------------------
Status: Open (was: Patch Available)
I doubt this patch applies cleanly anymore, and there was doubt about whether this was the best approach...
> task trackers should not restart for having a late heartbeat
> ------------------------------------------------------------
>
> Key: HADOOP-181
> URL: http://issues.apache.org/jira/browse/HADOOP-181
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Attachments: lost-heartbeat.patch
>
>
> TaskTrackers should not close and restart themselves for having a late heartbeat. The JobTracker should just accept their current status.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-181) task trackers should not restart for
having a late heartbeat
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-181?page=comments#action_12427581 ]
Doug Cutting commented on HADOOP-181:
-------------------------------------
> Why does one turn off speculative execution?
In the case of the Nutch crawler, speculative execution is disabled to observe politeness. We do not want two tasks to attempt to fetch pages from a site at the same time.
This patch adds a fair amount of complexity, introducing a new state for tasks (presumed dead, but reanimateable). A new state is likely to add new failure modes.
Does anyone deny that this primarily addresses an issue that would go away if we could more reliably detect tasktracker death? Shouldn't we attempt to fix that first? Sameer raises the issue of "transient network problems". Are we actually seeing these? Even if these were to occur, the system would operate correctly as-is: this is an optimization. Is this a common-enough case that we can afford to optimize it?
> task trackers should not restart for having a late heartbeat
> ------------------------------------------------------------
>
> Key: HADOOP-181
> URL: http://issues.apache.org/jira/browse/HADOOP-181
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Fix For: 0.6.0
>
> Attachments: lost-heartbeat.patch
>
>
> TaskTrackers should not close and restart themselves for having a late heartbeat. The JobTracker should just accept their current status.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-181) task trackers should not restart for
having a late heartbeat
Posted by "eric baldeschwieler (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-181?page=comments#action_12427371 ]
eric baldeschwieler commented on HADOOP-181:
--------------------------------------------
Oh a whole thread can be had on this I'm sure!
Why does one turn off speculative execution? Presumably because a MAP has unmanaged side-effects?
But... the framework still will rerun jobs if they complete and then the node is lost, right? Won't this tickle exactly the same issues that speculative execution raised anyway?
Doesn't this imply that disallowing speculative execution is basically not the right mechanism to deal with the side-effect issue and that this deserves a rethink?
> task trackers should not restart for having a late heartbeat
> ------------------------------------------------------------
>
> Key: HADOOP-181
> URL: http://issues.apache.org/jira/browse/HADOOP-181
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Fix For: 0.6.0
>
> Attachments: lost-heartbeat.patch
>
>
> TaskTrackers should not close and restart themselves for having a late heartbeat. The JobTracker should just accept their current status.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-181) task trackers should not restart for
having a late heartbeat
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-181?page=all ]
Doug Cutting updated HADOOP-181:
--------------------------------
Fix Version: 0.5.0
(was: 0.4.0)
> task trackers should not restart for having a late heartbeat
> ------------------------------------------------------------
>
> Key: HADOOP-181
> URL: http://issues.apache.org/jira/browse/HADOOP-181
> Project: Hadoop
> Type: Bug
> Components: mapred
> Versions: 0.1.1
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Fix For: 0.5.0
>
> TaskTrackers should not close and restart themselves for having a late heartbeat. The JobTracker should just accept their current status.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-181) task trackers should not restart for
having a late heartbeat
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-181?page=comments#action_12426757 ]
Doug Cutting commented on HADOOP-181:
-------------------------------------
It strikes me that we should instead improve detection of task tracker death, rather than to try to revive things when we mistakenly decide one is dead. Thoughts?
> task trackers should not restart for having a late heartbeat
> ------------------------------------------------------------
>
> Key: HADOOP-181
> URL: http://issues.apache.org/jira/browse/HADOOP-181
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Fix For: 0.6.0
>
> Attachments: lost-heartbeat.patch
>
>
> TaskTrackers should not close and restart themselves for having a late heartbeat. The JobTracker should just accept their current status.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-181) task trackers should not restart for
having a late heartbeat
Posted by "eric baldeschwieler (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-181?page=comments#action_12427373 ]
eric baldeschwieler commented on HADOOP-181:
--------------------------------------------
On reintegrating lost task trackers...
It does seem like we should do this to me, but we need to make sure we reason through how this effects corner cases, what invariants the system does maintain and so on.
I suggest we work this through, and then go forward with this patch (modified if we find any corner cases) and post the reasoning, so we can review it as this logic evolves. (And update any existing documentation in this area of course...)
> task trackers should not restart for having a late heartbeat
> ------------------------------------------------------------
>
> Key: HADOOP-181
> URL: http://issues.apache.org/jira/browse/HADOOP-181
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Fix For: 0.6.0
>
> Attachments: lost-heartbeat.patch
>
>
> TaskTrackers should not close and restart themselves for having a late heartbeat. The JobTracker should just accept their current status.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-181) task trackers should not restart for
having a late heartbeat
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-181?page=all ]
Devaraj Das updated HADOOP-181:
-------------------------------
Attachment: lost-heartbeat.patch
Attached is the patch.
> task trackers should not restart for having a late heartbeat
> ------------------------------------------------------------
>
> Key: HADOOP-181
> URL: http://issues.apache.org/jira/browse/HADOOP-181
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.1.1
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Fix For: 0.6.0
>
> Attachments: lost-heartbeat.patch
>
>
> TaskTrackers should not close and restart themselves for having a late heartbeat. The JobTracker should just accept their current status.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-181) task trackers should not restart for
having a late heartbeat
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-181?page=comments#action_12427115 ]
Doug Cutting commented on HADOOP-181:
-------------------------------------
> improved detection of tasktracker death is a separate issue
It's certainly related. This issue deals with fixing things when tasktracker deaths are mis-detected. If we didn't misdetect so much, this would not be an issue.
> Multiple instances of a task should be handled by the speculative execution code.
What if speculative execution is disabled in the job? Then we'd get multiple instances of the task running at once, when the client explicitly requested that not happen.
I'm +0 on this patch. If others feel strongly that it's the best approach, I won't veto it. But I would prefer we address the root problem first, and then see if this is still an issue, before adding this new mechanism. Does that make sense, or am I missing something?
> task trackers should not restart for having a late heartbeat
> ------------------------------------------------------------
>
> Key: HADOOP-181
> URL: http://issues.apache.org/jira/browse/HADOOP-181
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Fix For: 0.6.0
>
> Attachments: lost-heartbeat.patch
>
>
> TaskTrackers should not close and restart themselves for having a late heartbeat. The JobTracker should just accept their current status.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-181) task trackers should not restart for
having a late heartbeat
Posted by "Sameer Paranjpye (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-181?page=all ]
Sameer Paranjpye updated HADOOP-181:
------------------------------------
Fix Version: 0.4
(was: 0.3)
> task trackers should not restart for having a late heartbeat
> ------------------------------------------------------------
>
> Key: HADOOP-181
> URL: http://issues.apache.org/jira/browse/HADOOP-181
> Project: Hadoop
> Type: Bug
> Components: mapred
> Versions: 0.1.1
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Fix For: 0.4
>
> TaskTrackers should not close and restart themselves for having a late heartbeat. The JobTracker should just accept their current status.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Assigned: (HADOOP-181) task trackers should not restart for
having a late heartbeat
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-181?page=all ]
Devaraj Das reassigned HADOOP-181:
----------------------------------
Assignee: Devaraj Das (was: Owen O'Malley)
> task trackers should not restart for having a late heartbeat
> ------------------------------------------------------------
>
> Key: HADOOP-181
> URL: http://issues.apache.org/jira/browse/HADOOP-181
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.1.1
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Fix For: 0.6.0
>
>
> TaskTrackers should not close and restart themselves for having a late heartbeat. The JobTracker should just accept their current status.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-181) task trackers should not restart for
having a late heartbeat
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-181?page=comments#action_12427680 ]
Owen O'Malley commented on HADOOP-181:
--------------------------------------
I agree that the original desire for this patch was born of the TaskTracker timeouts that shouldn't happen. Fixing those problems (and we _have_ fixed most of them over the last 4 months) should take precendence. However, that said, I think in the long term, we do want something like this patch. If a switch goes down for 15 minutes and then comes back up, it does not make sense to reshuffle, resort, and rerun a reduce that takes hours to run.
All map/reduce applications, even those with speculative execution turned off, must permit redundant copies of their tasks for precisely this reason. In this case, the JobTracker has decided a given task is dead, but hasn't been able to tell the responsible TaskTracker yet. Therefore it schedules another instance of the failed task on a different node. Therefore, they are going to run in parallel for a while.
I guess for now, let's sit on this patch and contemplate what the model should be for dealing with communication problems. We should also monitor this in real use and see how often task trackers are being lost and probably put some effort to determine at least whether it is the job tracker or the task tracker that is the cause of the delay.
> task trackers should not restart for having a late heartbeat
> ------------------------------------------------------------
>
> Key: HADOOP-181
> URL: http://issues.apache.org/jira/browse/HADOOP-181
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Fix For: 0.6.0
>
> Attachments: lost-heartbeat.patch
>
>
> TaskTrackers should not close and restart themselves for having a late heartbeat. The JobTracker should just accept their current status.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira