You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Allen Wittenauer (JIRA)" <ji...@apache.org> on 2010/08/06 21:35:16 UTC
[jira] Created: (MAPREDUCE-1998) Size-based queuing for capacity
scheduler
Size-based queuing for capacity scheduler
-----------------------------------------
Key: MAPREDUCE-1998
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1998
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: contrib/capacity-sched
Reporter: Allen Wittenauer
On job submission, it would be useful if the capacity scheduler could pick a queue based on the # of maps and reduces. This way one could have queues based on job-size without users having to pick the queue prior to submission.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1998) Size-based queuing for capacity
scheduler
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896596#action_12896596 ]
Arun C Murthy commented on MAPREDUCE-1998:
------------------------------------------
An interesting way to look at things - this plays well with MAPREDUCE-1872 too.
> Size-based queuing for capacity scheduler
> -----------------------------------------
>
> Key: MAPREDUCE-1998
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1998
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: contrib/capacity-sched
> Reporter: Allen Wittenauer
>
> On job submission, it would be useful if the capacity scheduler could pick a queue based on the # of maps and reduces. This way one could have queues based on job-size without users having to pick the queue prior to submission.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1998) Size-based queuing for capacity
scheduler
Posted by "Allen Wittenauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896134#action_12896134 ]
Allen Wittenauer commented on MAPREDUCE-1998:
---------------------------------------------
So the way I imagine this working is:
admin configures 3 queues:
jobs with > 1000 tasks
jobs with <999 >100 tasks
jobs with < 99 tasks
user submits a job to default, which triggers sized-based queuing.
hadoop determines map tasks: lets say 100, and user requested 20 reduces. Total task count: 120.
JobTracker places job in the middle tier queue since there are 120 tasks.
> Size-based queuing for capacity scheduler
> -----------------------------------------
>
> Key: MAPREDUCE-1998
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1998
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: contrib/capacity-sched
> Reporter: Allen Wittenauer
>
> On job submission, it would be useful if the capacity scheduler could pick a queue based on the # of maps and reduces. This way one could have queues based on job-size without users having to pick the queue prior to submission.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-1998) Size-based queuing for capacity
scheduler
Posted by "Krishna Ramachandran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Krishna Ramachandran reassigned MAPREDUCE-1998:
-----------------------------------------------
Assignee: Krishna Ramachandran
> Size-based queuing for capacity scheduler
> -----------------------------------------
>
> Key: MAPREDUCE-1998
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1998
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: contrib/capacity-sched
> Reporter: Allen Wittenauer
> Assignee: Krishna Ramachandran
>
> On job submission, it would be useful if the capacity scheduler could pick a queue based on the # of maps and reduces. This way one could have queues based on job-size without users having to pick the queue prior to submission.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.