[jira] Created: (MAPREDUCE-2047) reduce overhead of findSpeculativeTask

["Joydeep Sen Sarma (JIRA)" <ji...@apache.org>]

reduce overhead of findSpeculativeTask
--------------------------------------

                 Key: MAPREDUCE-2047
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2047
             Project: Hadoop Map/Reduce
          Issue Type: Improvement
          Components: jobtracker
         Environment: hadoop-20 with HADOOP-2141
            Reporter: Joydeep Sen Sarma
            Assignee: Joydeep Sen Sarma


We are bottlenecked (in the JT) on the jobtracker lock and calls to findSpeculativeTask frequently show up as one of the top routines (by time) called holding this lock.

this routine calls canBeSpeculated() and hasRunOnMachine() for each task in a candidate job. Both these routines are reasonably expensive when invoked repeatedly  for thousands of tasks. The top candidates for speculation from a job only need to be refreshed periodically (and not once every heartbeat) - and we can can avoid most of these invocations this way.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.