You are viewing a plain text version of this content. The canonical link for it is here.

Posted to mapreduce-issues@hadoop.apache.org by "Gerald Guo (JIRA)" <ji...@apache.org> on 2011/04/01 22:59:05 UTC

[jira] [Created] (MAPREDUCE-2419) Jobs stall forever if mapred.task.cache.levels is set to 0.

Jobs stall forever if mapred.task.cache.levels is set to 0.
-----------------------------------------------------------

                 Key: MAPREDUCE-2419
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2419
             Project: Hadoop Map/Reduce
          Issue Type: Task
          Components: jobtracker
    Affects Versions: 0.21.0, 0.20.1
            Reporter: Gerald Guo
            Priority: Minor


It seems that mapred.task.cache.levels is used by JobTracker to create task caches for nodes at various levels. This makes data-locality scheduling possible.
If I set mapred.task.cache.levels to 0 (to disable task caching) and use default network topology, then mapreduce job will stall forever. The reason is JobInProgress::findNewMapTask always returns -1. Field "nonRunningMapCache" is empty and field "nonLocalMaps" is also empty.  

I can think of two possible fixes. 
1) If mapred.task.cache.levels is set 0, Hadoop should fall back to some default caching strategy. E.g. put all tasks into JobInProgress::nonLocalMaps.
2) ignore values less than 1

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira