[jira] Updated: (HADOOP-299) maps from second jobs will not run until the first job finishes completely

["Owen O'Malley (JIRA)" <ji...@apache.org>]

     [ http://issues.apache.org/jira/browse/HADOOP-299?page=all ]

Owen O'Malley updated HADOOP-299:
---------------------------------

    Attachment: map-schedule.patch

This patch does a couple of things:
  1. It makes it more obvious that the task slot padding is only done on clusters bigger than 3 nodes.
  2. Remove the dead avgMaps and avgReduces variables.
  3. Change the default value of PAD_FRACTION from 0.1 to 0.01. (This variable is not in hadoop-default.xml, so the code controls the default value.)
  4. Fix a typo that was counting all of the running jobs' tasks instead of the the running jobs' running tasks. (totalMaps versus totalNeededMaps and totalReduces versus totalNeededReduces)

> maps from second jobs will not run until the first job finishes completely
> --------------------------------------------------------------------------
>
>          Key: HADOOP-299
>          URL: http://issues.apache.org/jira/browse/HADOOP-299
>      Project: Hadoop
>         Type: Bug

>   Components: mapred
>     Versions: 0.3.2
>     Reporter: Owen O'Malley
>     Assignee: Owen O'Malley
>      Fix For: 0.4.0
>  Attachments: map-schedule.patch
>
> Because of the logic in the JobTracker's pollForNewTask, second jobs will rarely start running maps until the first job finishes completely. The JobTracker leaves room to re-run failed maps from the first job and it reserves the total number of maps for the first job. Thus, if you have more maps in the first job than your cluster capacity, none of the second job maps will ever run.
> I propose setting the reserve to 1% of the first job's maps.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira