[jira] Created: (MAPREDUCE-1903) Allow different slowTaskThreshold for mappers and reducers

["Scott Chen (JIRA)" <ji...@apache.org>]

Allow different slowTaskThreshold for mappers and reducers
----------------------------------------------------------

                 Key: MAPREDUCE-1903
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1903
             Project: Hadoop Map/Reduce
          Issue Type: Improvement
          Components: jobtracker
    Affects Versions: 0.22.0
            Reporter: Scott Chen
            Assignee: Scott Chen
             Fix For: 0.22.0


We have been running the new speculative logic in HADOOP-2141 done by Andy in our production cluster.
One thing that we observed is that there are significantly low number of speculative reducers.
We have seen usually 100 speculative mappers launched per minute but speculative reducers are usually less than 5.
But reducers are usually where we get complains about having speculative issues.

These two types of tasks has different properties and different needs.
It would be nice if we can configure the slow threshold separately to deal with them separately.

We can add the following config keys to allow setting them independently.
MRJobConfig.SPECULATIVE_MAP_SLOWTASK_THRESHOLD
MRJobConfig.SPECULATIVE_REDUCE_SLOWTASK_THRESHOLD

Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.