You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Koji Noguchi (JIRA)" <ji...@apache.org> on 2012/08/16 00:22:38 UTC
[jira] [Commented] (MAPREDUCE-4499) Looking for speculative tasks
is very expensive in 1.x
[ https://issues.apache.org/jira/browse/MAPREDUCE-4499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435573#comment-13435573 ]
Koji Noguchi commented on MAPREDUCE-4499:
-----------------------------------------
Assuming the cost of tip.hasRunOnMachine is expensive, we can try reordering the if/else so that we call it less often.
>From JobInProgress.java
{noformat}
2196 if (!tip.hasRunOnMachine(ttStatus.getHost(),
2197 ttStatus.getTrackerName())) {
2198 if (tip.hasSpeculativeTask(currentTime, avgProgress)) {
2199 // In case of shared list we don't remove it. Since the TIP failed
2200 // on this tracker can be scheduled on some other tracker.
2201 if (shouldRemove) {
2202 iter.remove(); //this tracker is never going to run it again
2203 }
2204 return tip;
2205 }
2206 } else {
2207 // Check if this tip can be removed from the list.
2208 // If the list is shared then we should not remove.
2209 if (shouldRemove) {
2210 // This tracker will never speculate this tip
2211 iter.remove();
2212 }
2213 }
2214 }
{noformat}
Checking the action for each true&false.
{noformat}
tip.hasRun tip.hasSpeculative shouldRemove Action
F F F -
F F T -
F T F return tip
F T T iter.remove() & return tip;
T F F -
T F T iter.remove()
T T F -
T T T iter.remove()
{noformat}
Can we rewrite the above logic to
{noformat}
if (tip.hasSpeculative) {
if(shouldRemove){
iter.remove();
}
if(!tip.hasRun) {
return tip;
}
} else {
if (shouldRemove && tip.hasRun ){
iter.remove();
}
}
{noformat}
>From the jstack we see, I can tell that shouldRemove is often 'false' in our case. Depending on the value of tip.hasSpeculative, we may reduce the tip.hasRun calls with this rewrite.
(I don't know how often 'hasSpeculative' is true.)
> Looking for speculative tasks is very expensive in 1.x
> ------------------------------------------------------
>
> Key: MAPREDUCE-4499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4499
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, performance
> Affects Versions: 1.0.3
> Reporter: Nathan Roberts
>
> When there are lots of jobs and tasks active in a cluster, the process of figuring out whether or not to launch a speculative task becomes very expensive.
> I could be missing something but it certainly looks like on every heartbeat we could be scanning 10's of thousands of tasks looking for something which might need to be speculatively executed. In most cases, nothing gets chosen so we completely trashed our data cache and didn't even find a task to schedule, just to do it all over again on the next heartbeat.
> On busy jobtrackers, the following backtrace is very common:
> "IPC Server handler 32 on 50300" daemon prio=10 tid=0x00002ab36c74f800
> nid=0xb50 runnable [0x0000000045adb000]
> java.lang.Thread.State: RUNNABLE
> at java.util.TreeMap.valEquals(TreeMap.java:1182)
> at java.util.TreeMap.containsValue(TreeMap.java:227)
> at java.util.TreeMap$Values.contains(TreeMap.java:940)
> at
> org.apache.hadoop.mapred.TaskInProgress.hasRunOnMachine(TaskInProgress.java:1072)
> at
> org.apache.hadoop.mapred.JobInProgress.findSpeculativeTask(JobInProgress.java:2193)
> - locked <0x00002aaefde82338> (a
> org.apache.hadoop.mapred.JobInProgress)
> at
> org.apache.hadoop.mapred.JobInProgress.findNewMapTask(JobInProgress.java:2417)
> - locked <0x00002aaefde82338> (a
> org.apache.hadoop.mapred.JobInProgress)
> at
> org.apache.hadoop.mapred.JobInProgress.obtainNewNonLocalMapTask(JobInProgress.java:1432)
> - locked <0x00002aaefde82338> (a
> org.apache.hadoop.mapred.JobInProgress)
> at
> org.apache.hadoop.mapred.CapacityTaskScheduler$MapSchedulingMgr.obtainNewTask(CapacityTaskScheduler.java:525)
> at
> org.apache.hadoop.mapred.CapacityTaskScheduler$TaskSchedulingMgr.getTaskFromQueue(CapacityTaskScheduler.java:322)
> at
> org.apache.hadoop.mapred.CapacityTaskScheduler$TaskSchedulingMgr.assignTasks(CapacityTaskScheduler.java:419)
> at
> org.apache.hadoop.mapred.CapacityTaskScheduler$TaskSchedulingMgr.access$500(CapacityTaskScheduler.java:150)
> at
> org.apache.hadoop.mapred.CapacityTaskScheduler.addMapTasks(CapacityTaskScheduler.java:1075)
> at
> org.apache.hadoop.mapred.CapacityTaskScheduler.assignTasks(CapacityTaskScheduler.java:1044)
> - locked <0x00002aab6e27a4c8> (a
> org.apache.hadoop.mapred.CapacityTaskScheduler)
> at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:3398)
> - locked <0x00002aab6e191278> (a org.apache.hadoop.mapred.JobTracker)
> ...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira