You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Runping Qi (JIRA)" <ji...@apache.org> on 2007/11/09 00:26:50 UTC
[jira] Created: (HADOOP-2175) Blacklisted hosts may not be able to
serve map outputs
Blacklisted hosts may not be able to serve map outputs
------------------------------------------------------
Key: HADOOP-2175
URL: https://issues.apache.org/jira/browse/HADOOP-2175
Project: Hadoop
Issue Type: Bug
Components: mapred
Reporter: Runping Qi
After a node fails 4 mappers (tasks), it is added to blacklist thus it will no longer accept tasks.
But, it will continue serve the map outputs of any mappers that ran successfully there.
However, the node may not be able serve the map outputs either.
This will cause the reducers to mark the corresponding map outputs as from slow hosts,
but continue to try to get the map outputs from that node.
This may lead to waiting forever.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2175) Blacklisted hosts may not be able
to serve map outputs
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12541236 ]
Arun C Murthy commented on HADOOP-2175:
---------------------------------------
Runping, the work we did on HADOOP-1158 should take care of situations like this one, no?
If you are satisfied, I'll close this issue...
> Blacklisted hosts may not be able to serve map outputs
> ------------------------------------------------------
>
> Key: HADOOP-2175
> URL: https://issues.apache.org/jira/browse/HADOOP-2175
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Reporter: Runping Qi
> Assignee: Arun C Murthy
>
> After a node fails 4 mappers (tasks), it is added to blacklist thus it will no longer accept tasks.
> But, it will continue serve the map outputs of any mappers that ran successfully there.
> However, the node may not be able serve the map outputs either.
> This will cause the reducers to mark the corresponding map outputs as from slow hosts,
> but continue to try to get the map outputs from that node.
> This may lead to waiting forever.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2175) Blacklisted hosts may not be able
to serve map outputs
Posted by "Runping Qi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12541386 ]
Runping Qi commented on HADOOP-2175:
------------------------------------
The criteria for starting a speculative execution should also include
how much/little progress has been made in the past M minutes.
If a task is detected to be stalled, a speculative execution should be
started, no matter how close the original one is to completion.
> Blacklisted hosts may not be able to serve map outputs
> ------------------------------------------------------
>
> Key: HADOOP-2175
> URL: https://issues.apache.org/jira/browse/HADOOP-2175
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Reporter: Runping Qi
> Assignee: Arun C Murthy
>
> After a node fails 4 mappers (tasks), it is added to blacklist thus it will no longer accept tasks.
> But, it will continue serve the map outputs of any mappers that ran successfully there.
> However, the node may not be able serve the map outputs either.
> This will cause the reducers to mark the corresponding map outputs as from slow hosts,
> but continue to try to get the map outputs from that node.
> This may lead to waiting forever.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2175) Blacklisted hosts may not be able
to serve map outputs
Posted by "Runping Qi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12541240 ]
Runping Qi commented on HADOOP-2175:
------------------------------------
I was using hadoop 0.15.0.
is the patch for hadoop-1158 in 0.15.0?
if so, the problem is not solved.
> Blacklisted hosts may not be able to serve map outputs
> ------------------------------------------------------
>
> Key: HADOOP-2175
> URL: https://issues.apache.org/jira/browse/HADOOP-2175
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Reporter: Runping Qi
> Assignee: Arun C Murthy
>
> After a node fails 4 mappers (tasks), it is added to blacklist thus it will no longer accept tasks.
> But, it will continue serve the map outputs of any mappers that ran successfully there.
> However, the node may not be able serve the map outputs either.
> This will cause the reducers to mark the corresponding map outputs as from slow hosts,
> but continue to try to get the map outputs from that node.
> This may lead to waiting forever.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-2175) Blacklisted hosts may not be able to
serve map outputs
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy reassigned HADOOP-2175:
-------------------------------------
Assignee: Arun C Murthy
> Blacklisted hosts may not be able to serve map outputs
> ------------------------------------------------------
>
> Key: HADOOP-2175
> URL: https://issues.apache.org/jira/browse/HADOOP-2175
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Reporter: Runping Qi
> Assignee: Arun C Murthy
>
> After a node fails 4 mappers (tasks), it is added to blacklist thus it will no longer accept tasks.
> But, it will continue serve the map outputs of any mappers that ran successfully there.
> However, the node may not be able serve the map outputs either.
> This will cause the reducers to mark the corresponding map outputs as from slow hosts,
> but continue to try to get the map outputs from that node.
> This may lead to waiting forever.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2175) Blacklisted hosts may not be able
to serve map outputs
Posted by "Runping Qi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12541350 ]
Runping Qi commented on HADOOP-2175:
------------------------------------
then we need to try the patch for hadoop-1984.
Currently, a job may stall forever (I waited 30+ minutes, and eventually killed the job).
> Blacklisted hosts may not be able to serve map outputs
> ------------------------------------------------------
>
> Key: HADOOP-2175
> URL: https://issues.apache.org/jira/browse/HADOOP-2175
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Reporter: Runping Qi
> Assignee: Arun C Murthy
>
> After a node fails 4 mappers (tasks), it is added to blacklist thus it will no longer accept tasks.
> But, it will continue serve the map outputs of any mappers that ran successfully there.
> However, the node may not be able serve the map outputs either.
> This will cause the reducers to mark the corresponding map outputs as from slow hosts,
> but continue to try to get the map outputs from that node.
> This may lead to waiting forever.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2175) Blacklisted hosts may not be able
to serve map outputs
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12541243 ]
Arun C Murthy commented on HADOOP-2175:
---------------------------------------
Yes, HADOOP-1158 went into 0.15.0.
Unfortunately with the way the back-offs are structured (waiting period between re-tries) it takes a little while for the map-task on the blacklisted node to get killed. Could you wait for a bit and see if the map got killed?
The back-offs are fixed by HADOOP-1984 and they should help enormously here...
> Blacklisted hosts may not be able to serve map outputs
> ------------------------------------------------------
>
> Key: HADOOP-2175
> URL: https://issues.apache.org/jira/browse/HADOOP-2175
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Reporter: Runping Qi
> Assignee: Arun C Murthy
>
> After a node fails 4 mappers (tasks), it is added to blacklist thus it will no longer accept tasks.
> But, it will continue serve the map outputs of any mappers that ran successfully there.
> However, the node may not be able serve the map outputs either.
> This will cause the reducers to mark the corresponding map outputs as from slow hosts,
> but continue to try to get the map outputs from that node.
> This may lead to waiting forever.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.