You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Arun C Murthy (JIRA)" <ji...@apache.org> on 2009/07/07 18:54:14 UTC
[jira] Created: (MAPREDUCE-725)
CapacityScheduler.TaskSchedulingMgr.hasSpeculativeTask bypasses HADOOP-2141
CapacityScheduler.TaskSchedulingMgr.hasSpeculativeTask bypasses HADOOP-2141
---------------------------------------------------------------------------
Key: MAPREDUCE-725
URL: https://issues.apache.org/jira/browse/MAPREDUCE-725
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/capacity-sched
Reporter: Arun C Murthy
Priority: Critical
CapacityScheduler.TaskSchedulingMgr.hasSpeculativeTask has a duplicate of the old speculation code (pre HADOOP-2141) which needs to be fixed in-order for speculation to work correctly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-725)
CapacityScheduler.TaskSchedulingMgr.hasSpeculativeTask bypasses HADOOP-2141
Posted by "Matei Zaharia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728193#action_12728193 ]
Matei Zaharia commented on MAPREDUCE-725:
-----------------------------------------
Is it possible to add a hasSpeculativeTask method in JobInProgress rather than copying it out into the capacity scheduler again?
> CapacityScheduler.TaskSchedulingMgr.hasSpeculativeTask bypasses HADOOP-2141
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-725
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-725
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/capacity-sched
> Reporter: Arun C Murthy
> Priority: Critical
>
> CapacityScheduler.TaskSchedulingMgr.hasSpeculativeTask has a duplicate of the old speculation code (pre HADOOP-2141) which needs to be fixed in-order for speculation to work correctly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-725)
CapacityScheduler.TaskSchedulingMgr.hasSpeculativeTask bypasses HADOOP-2141
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729713#action_12729713 ]
rahul k singh commented on MAPREDUCE-725:
-----------------------------------------
Looked at the JobInProgress's findSpeculativeTask , it doesn't change any state of TaskInProgress. We can refactor this method in order to find out if jip has any speculative tasks.
We are using hasSpeculative to block a slot on the task tracker. As hasSpeculative is very dynamic in nature , there are chances that by the time tasktracker comes back for the task , we dont have any speculativeTasks to assign, this would mean we would waste some heartbeat and also would lead to under utilization.
This problem is already there with the capacity-scheduler though.
One way of reducing this is to have MIN_SPEC_CAP(in JobInProgress) kind of variable in capacity scheduler.
Any thoughts?
> CapacityScheduler.TaskSchedulingMgr.hasSpeculativeTask bypasses HADOOP-2141
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-725
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-725
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/capacity-sched
> Reporter: Arun C Murthy
> Priority: Critical
>
> CapacityScheduler.TaskSchedulingMgr.hasSpeculativeTask has a duplicate of the old speculation code (pre HADOOP-2141) which needs to be fixed in-order for speculation to work correctly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-725)
CapacityScheduler.TaskSchedulingMgr.hasSpeculativeTask bypasses HADOOP-2141
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728532#action_12728532 ]
Vinod K V commented on MAPREDUCE-725:
-------------------------------------
bq. the number of speculative tasks is very dynamic(depends on progress of other tasks, overall job progress etc.).
Not just that, as we've found out while working on MAPREDUCE-722, speculative tasks is also dependant on the TaskTracker in question - for e.g. a task can have a speculative attempt w.r.t one TT and none w.r.t another if it has already run on the later TT.
> CapacityScheduler.TaskSchedulingMgr.hasSpeculativeTask bypasses HADOOP-2141
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-725
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-725
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/capacity-sched
> Reporter: Arun C Murthy
> Priority: Critical
>
> CapacityScheduler.TaskSchedulingMgr.hasSpeculativeTask has a duplicate of the old speculation code (pre HADOOP-2141) which needs to be fixed in-order for speculation to work correctly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-725)
CapacityScheduler.TaskSchedulingMgr.hasSpeculativeTask bypasses HADOOP-2141
Posted by "Matei Zaharia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728350#action_12728350 ]
Matei Zaharia commented on MAPREDUCE-725:
-----------------------------------------
In the fair scheduler, my plan was to look at speculative task counts every 10-15 seconds, because while the count is dynamic, speculation decisions are made on the order of minutes anyway. Therefore, I'd be okay with a numSpeculative{Maps|Reduces} that is time-consuming to compute and may change fairly rapidly. However, if you don't think this should be part of the API, then just hasSpeculative{Map|Reduce} is fine. I'll try making just one speculative slot at a time and see if that causes problems.
> CapacityScheduler.TaskSchedulingMgr.hasSpeculativeTask bypasses HADOOP-2141
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-725
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-725
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/capacity-sched
> Reporter: Arun C Murthy
> Priority: Critical
>
> CapacityScheduler.TaskSchedulingMgr.hasSpeculativeTask has a duplicate of the old speculation code (pre HADOOP-2141) which needs to be fixed in-order for speculation to work correctly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-725)
CapacityScheduler.TaskSchedulingMgr.hasSpeculativeTask bypasses HADOOP-2141
Posted by "Matei Zaharia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728305#action_12728305 ]
Matei Zaharia commented on MAPREDUCE-725:
-----------------------------------------
In fact, it might be even better if there were methods called JobInProgress.numSpeculativeMaps and JobInProgress.numSpeculativeReduces that say how many speculative tasks the job wants to launch. Knowing this number is useful when you need to make high-memory slots for the speculative tasks, or when you need to preempt tasks from other jobs.
> CapacityScheduler.TaskSchedulingMgr.hasSpeculativeTask bypasses HADOOP-2141
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-725
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-725
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/capacity-sched
> Reporter: Arun C Murthy
> Priority: Critical
>
> CapacityScheduler.TaskSchedulingMgr.hasSpeculativeTask has a duplicate of the old speculation code (pre HADOOP-2141) which needs to be fixed in-order for speculation to work correctly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-725)
CapacityScheduler.TaskSchedulingMgr.hasSpeculativeTask bypasses HADOOP-2141
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729714#action_12729714 ]
rahul k singh commented on MAPREDUCE-725:
-----------------------------------------
Looked at the JobInProgress's findSpeculativeTask , it doesn't change any state of TaskInProgress. We can refactor this method in order to find out if jip has any speculative tasks.
We are using hasSpeculative to block a slot on the task tracker. As hasSpeculative is very dynamic in nature , there are chances that by the time tasktracker comes back for the task , we dont have any speculativeTasks to assign, this would mean we would waste some heartbeat and also would lead to under utilization.
This problem is already there with the capacity-scheduler though.
One way of reducing this is to have MIN_SPEC_CAP(in JobInProgress) kind of variable in capacity scheduler.
Any thoughts?
> CapacityScheduler.TaskSchedulingMgr.hasSpeculativeTask bypasses HADOOP-2141
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-725
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-725
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/capacity-sched
> Reporter: Arun C Murthy
> Priority: Critical
>
> CapacityScheduler.TaskSchedulingMgr.hasSpeculativeTask has a duplicate of the old speculation code (pre HADOOP-2141) which needs to be fixed in-order for speculation to work correctly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-725)
CapacityScheduler.TaskSchedulingMgr.hasSpeculativeTask bypasses HADOOP-2141
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728337#action_12728337 ]
Arun C Murthy commented on MAPREDUCE-725:
-----------------------------------------
I'm not sure we can make JobInProgress.numSpeculative{Maps|Reduces} stick... the number of speculative tasks is very dynamic(depends on progress of other tasks, overall job progress etc.).
Having said that, the fix for this issue *is* to move hasSpeculativeTask to JobInProgress.
> CapacityScheduler.TaskSchedulingMgr.hasSpeculativeTask bypasses HADOOP-2141
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-725
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-725
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/capacity-sched
> Reporter: Arun C Murthy
> Priority: Critical
>
> CapacityScheduler.TaskSchedulingMgr.hasSpeculativeTask has a duplicate of the old speculation code (pre HADOOP-2141) which needs to be fixed in-order for speculation to work correctly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.