[jira] Created: (MAPREDUCE-1764) FairScheduler locality delay may put heavy pressure on Jobtracker

["Scott Chen (JIRA)" <ji...@apache.org>]

FairScheduler locality delay may put heavy pressure on Jobtracker
-----------------------------------------------------------------

                 Key: MAPREDUCE-1764
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1764
             Project: Hadoop Map/Reduce
          Issue Type: Bug
            Reporter: Scott Chen
            Assignee: Scott Chen


FairScheduler locality delay feature holds the scheduling of jobs until it gets good locality.
This greatly improves the locality of the tasks. Reduce the cost of traffic.

We have observed the following problem on FairScheduler locality delay:
We have some machines have older data and some newly added machines do not have important data.
When these machines send heartbeat, JT scans tasks to find jobs has the right locality.
Often time, these machines will scan all of the tasks of all the jobs and do not get any tasks.
Scanning all the tasks on the JT is very costly. This makes JT very slow.
And these machines often time do not get scheduled. This hurts the cluster utilization.

Any ideas?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.