You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-dev@hadoop.apache.org by "Alvin Chyan (JIRA)" <ji...@apache.org> on 2016/04/15 21:39:25 UTC
[jira] [Created] (MAPREDUCE-6679) on node failure, only restart
mappers whose output is not copied
Alvin Chyan created MAPREDUCE-6679:
--------------------------------------
Summary: on node failure, only restart mappers whose output is not copied
Key: MAPREDUCE-6679
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6679
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Affects Versions: 2.7.0
Reporter: Alvin Chyan
Priority: Minor
When we detect a bad node, we reschedule all succeeded map tasks on that node in JobImpl.actOnUnusableNode. Wouldn't we be able to get away with only rescheduling the map tasks that have not had their outputs copied to a reducer already?
One consideration could be that the reducer that fetched the mapper output is then killed itself. However, in testing, it seems that once a reducer has moved past the shuffle phase and is reducing, even if the mapper node fails, the mappers don't get rescheduled. The same mechanism that occurs then if a reducer dies can then be applied in this scenario.
This is helpful in general, but is especially beneficial in cloud environments that offer spot/preemptible instances. As long as reducers are running to continually fetch mapper outputs, the job can make progress as long as the preemptible instances stay up long enough for a map task to complete.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)