You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Mahadev konar (JIRA)" <ji...@apache.org> on 2006/07/05 20:09:29 UTC
[jira] Created: (HADOOP-343) In case of dead task tracker, the copy
mapouts try copying all mapoutputs from this tasktracker
In case of dead task tracker, the copy mapouts try copying all mapoutputs from this tasktracker
------------------------------------------------------------------------------------------------
Key: HADOOP-343
URL: http://issues.apache.org/jira/browse/HADOOP-343
Project: Hadoop
Type: Bug
Components: mapred
Versions: 0.5.0
Reporter: Mahadev konar
Assigned to: Mahadev konar
In case of a dead task tracker, the reduces which do not have the updated map out locations try copygin files from this node and since there are failures on copying, this leads to backoff and slowing down of the copy pahse.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-343) In case of dead task tracker, the copy
mapouts try copying all mapoutputs from this tasktracker
Posted by "Sameer Paranjpye (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-343?page=all ]
Sameer Paranjpye updated HADOOP-343:
------------------------------------
Status: Patch Available (was: In Progress)
Fix Version/s: 0.7.0
Affects Version/s: 0.6.2
(was: 0.5.0)
> In case of dead task tracker, the copy mapouts try copying all mapoutputs from this tasktracker
> -----------------------------------------------------------------------------------------------
>
> Key: HADOOP-343
> URL: http://issues.apache.org/jira/browse/HADOOP-343
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.6.2
> Reporter: Mahadev konar
> Assigned To: Sameer Paranjpye
> Fix For: 0.7.0
>
> Attachments: bugfix.patch, cache-purge.txt
>
>
> In case of a dead task tracker, the reduces which do not have the updated map out locations try copygin files from this node and since there are failures on copying, this leads to backoff and slowing down of the copy pahse.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-343) In case of dead task tracker, the copy
mapouts try copying all mapoutputs from this tasktracker
Posted by "Sameer Paranjpye (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-343?page=all ]
Sameer Paranjpye updated HADOOP-343:
------------------------------------
Fix Version/s: 0.6.0
> In case of dead task tracker, the copy mapouts try copying all mapoutputs from this tasktracker
> -----------------------------------------------------------------------------------------------
>
> Key: HADOOP-343
> URL: http://issues.apache.org/jira/browse/HADOOP-343
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.5.0
> Reporter: Mahadev konar
> Assigned To: Mahadev konar
> Fix For: 0.6.0
>
> Attachments: bugfix.patch
>
>
> In case of a dead task tracker, the reduces which do not have the updated map out locations try copygin files from this node and since there are failures on copying, this leads to backoff and slowing down of the copy pahse.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-343) In case of dead task tracker, the
copy mapouts try copying all mapoutputs from this tasktracker
Posted by "Sameer Paranjpye (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-343?page=comments#action_12439897 ]
Sameer Paranjpye commented on HADOOP-343:
-----------------------------------------
This patch addresses the concern raised. If a map output transfer from a particular tasktracker fails, other output locations from the tasktracker that are present in the cache are removed. This addresses the problem of repeated attempts and backoffs from a lost tasktracker, which is particularly bad towards the end of a shuffle. Copies can, of course, fail for other reasons, in these cases also output locations are removed. The cost of this removal is fairly low. This is because the number of output locations cached for a specific tasktracker is usually small (3-4), and removing these (multiple times even) results in a handful of extra polls of the jobtracker.
> In case of dead task tracker, the copy mapouts try copying all mapoutputs from this tasktracker
> -----------------------------------------------------------------------------------------------
>
> Key: HADOOP-343
> URL: http://issues.apache.org/jira/browse/HADOOP-343
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.6.2
> Reporter: Mahadev konar
> Assigned To: Sameer Paranjpye
> Fix For: 0.7.0
>
> Attachments: bugfix.patch, cache-purge.txt
>
>
> In case of a dead task tracker, the reduces which do not have the updated map out locations try copygin files from this node and since there are failures on copying, this leads to backoff and slowing down of the copy pahse.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Work started: (HADOOP-343) In case of dead task tracker, the
copy mapouts try copying all mapoutputs from this tasktracker
Posted by "Sameer Paranjpye (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-343?page=all ]
Work on HADOOP-343 started by Sameer Paranjpye.
> In case of dead task tracker, the copy mapouts try copying all mapoutputs from this tasktracker
> -----------------------------------------------------------------------------------------------
>
> Key: HADOOP-343
> URL: http://issues.apache.org/jira/browse/HADOOP-343
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.5.0
> Reporter: Mahadev konar
> Assigned To: Sameer Paranjpye
> Attachments: bugfix.patch
>
>
> In case of a dead task tracker, the reduces which do not have the updated map out locations try copygin files from this node and since there are failures on copying, this leads to backoff and slowing down of the copy pahse.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-343) In case of dead task tracker, the
copy mapouts try copying all mapoutputs from this tasktracker
Posted by "Sameer Paranjpye (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-343?page=comments#action_12428260 ]
Sameer Paranjpye commented on HADOOP-343:
-----------------------------------------
I think this needs further attention. The patch is probably out of date at this point, but the problem is real. I think this may also be responsible for the 'long pause' at the end of the shuffle.
If a tasktracker fails, it's map outputs are lost. However, the other tasktrackers are unaware of this. Towards the end of the shuffle they have all map output locations cached and they keep trying to pull data from the lost tasktracker, one file at a time. Every one of these file transfers fails, each failed transfer also causes the tasktrackers to back off from pulling their remaining outputs. The cumulative effect of all the backoffs is the long pause.
> In case of dead task tracker, the copy mapouts try copying all mapoutputs from this tasktracker
> -----------------------------------------------------------------------------------------------
>
> Key: HADOOP-343
> URL: http://issues.apache.org/jira/browse/HADOOP-343
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.5.0
> Reporter: Mahadev konar
> Assigned To: Mahadev konar
> Attachments: bugfix.patch
>
>
> In case of a dead task tracker, the reduces which do not have the updated map out locations try copygin files from this node and since there are failures on copying, this leads to backoff and slowing down of the copy pahse.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-343) In case of dead task tracker, the
copy mapouts try copying all mapoutputs from this tasktracker
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-343?page=comments#action_12419349 ]
Owen O'Malley commented on HADOOP-343:
--------------------------------------
I have concerns about this patch. It might have unintended consequences. In particular, if a node is slow you'll drop what you know about that node and put more load on the job tracker. This patch is certainly addressing a real problem, though.
> In case of dead task tracker, the copy mapouts try copying all mapoutputs from this tasktracker
> -----------------------------------------------------------------------------------------------
>
> Key: HADOOP-343
> URL: http://issues.apache.org/jira/browse/HADOOP-343
> Project: Hadoop
> Type: Bug
> Components: mapred
> Versions: 0.5.0
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Attachments: bugfix.patch
>
> In case of a dead task tracker, the reduces which do not have the updated map out locations try copygin files from this node and since there are failures on copying, this leads to backoff and slowing down of the copy pahse.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-343) In case of dead task tracker, the copy
mapouts try copying all mapoutputs from this tasktracker
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-343?page=all ]
Doug Cutting updated HADOOP-343:
--------------------------------
Status: Resolved (was: Patch Available)
Resolution: Fixed
I just committed this. Thanks, Sameer!
> In case of dead task tracker, the copy mapouts try copying all mapoutputs from this tasktracker
> -----------------------------------------------------------------------------------------------
>
> Key: HADOOP-343
> URL: http://issues.apache.org/jira/browse/HADOOP-343
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.6.2
> Reporter: Mahadev konar
> Assigned To: Sameer Paranjpye
> Fix For: 0.7.0
>
> Attachments: bugfix.patch, cache-purge.txt
>
>
> In case of a dead task tracker, the reduces which do not have the updated map out locations try copygin files from this node and since there are failures on copying, this leads to backoff and slowing down of the copy pahse.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Assigned: (HADOOP-343) In case of dead task tracker, the
copy mapouts try copying all mapoutputs from this tasktracker
Posted by "Sameer Paranjpye (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-343?page=all ]
Sameer Paranjpye reassigned HADOOP-343:
---------------------------------------
Assignee: Sameer Paranjpye (was: Mahadev konar)
> In case of dead task tracker, the copy mapouts try copying all mapoutputs from this tasktracker
> -----------------------------------------------------------------------------------------------
>
> Key: HADOOP-343
> URL: http://issues.apache.org/jira/browse/HADOOP-343
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.5.0
> Reporter: Mahadev konar
> Assigned To: Sameer Paranjpye
> Attachments: bugfix.patch
>
>
> In case of a dead task tracker, the reduces which do not have the updated map out locations try copygin files from this node and since there are failures on copying, this leads to backoff and slowing down of the copy pahse.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-343) In case of dead task tracker, the copy
mapouts try copying all mapoutputs from this tasktracker
Posted by "Sameer Paranjpye (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-343?page=all ]
Sameer Paranjpye updated HADOOP-343:
------------------------------------
Attachment: cache-purge.txt
> In case of dead task tracker, the copy mapouts try copying all mapoutputs from this tasktracker
> -----------------------------------------------------------------------------------------------
>
> Key: HADOOP-343
> URL: http://issues.apache.org/jira/browse/HADOOP-343
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.6.2
> Reporter: Mahadev konar
> Assigned To: Sameer Paranjpye
> Fix For: 0.7.0
>
> Attachments: bugfix.patch, cache-purge.txt
>
>
> In case of a dead task tracker, the reduces which do not have the updated map out locations try copygin files from this node and since there are failures on copying, this leads to backoff and slowing down of the copy pahse.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-343) In case of dead task tracker, the copy
mapouts try copying all mapoutputs from this tasktracker
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-343?page=all ]
Mahadev konar updated HADOOP-343:
---------------------------------
Attachment: bugfix.patch
This patch updates its mapoutput locations in case of any failures on copying from the task tracker. So, in case a copy failed from a task tracker, the map outputs corresponding to this node will be marked stale. The ReduceTask will ask for the mapout locations again for these stale map outputs. This patch also fixes a bug wherein the tasks keep polling the job tracker for map otputs "in a loop" without sleeping/waiting. With this fix the tasks will poll evry MIN_POLL_INTERVAL before querying the job tracker.
> In case of dead task tracker, the copy mapouts try copying all mapoutputs from this tasktracker
> -----------------------------------------------------------------------------------------------
>
> Key: HADOOP-343
> URL: http://issues.apache.org/jira/browse/HADOOP-343
> Project: Hadoop
> Type: Bug
> Components: mapred
> Versions: 0.5.0
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Attachments: bugfix.patch
>
> In case of a dead task tracker, the reduces which do not have the updated map out locations try copygin files from this node and since there are failures on copying, this leads to backoff and slowing down of the copy pahse.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira