You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Mike Roark (JIRA)" <ji...@apache.org> on 2013/05/15 05:13:16 UTC
[jira] [Commented] (MAPREDUCE-3859) CapacityScheduler incorrectly
utilizes extra-resources of queue for high-memory jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-3859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13657840#comment-13657840 ]
Mike Roark commented on MAPREDUCE-3859:
---------------------------------------
I applied Sergey's patch from the first comment in my cdh3u5 cluster and had some improvement for jobs with 3 slots per task, but the numbers did not change for other sizes of tasks. Sergey, do you think I'm doing something wrong?
Same setup as above: https://issues.apache.org/jira/browse/MAPREDUCE-3859?focusedCommentId=13609117&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13609117
||mapred.reduce.tasks||slots per reducer||expected running tasks||running tasks without patch||using slots without patch||running tasks with patch||using slots with patch||
|30|1|30|30|30|30|30|
|30|2|30|{color:red}16{color}|32|16|32|
|30|3|20|{color:red}1{color}|3|{color:green}11{color}|33|
|30|4|10|{color:red}8{color}|32|8|32|
|30|5|10|{color:red}8{color}|40|8|40|
|30|6|10|{color:red}8{color}|48|8|48|
> CapacityScheduler incorrectly utilizes extra-resources of queue for high-memory jobs
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3859
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3859
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: capacity-sched
> Affects Versions: 1.0.0
> Environment: CDH3u1
> Reporter: Sergey Tryuber
> Assignee: Sergey Tryuber
> Attachments: test-to-fail.patch.txt
>
>
> Imagine, we have a queue A with capacity 10 slots and 20 as extra-capacity, jobs which use 3 map slots will never consume more than 9 slots, regardless how many free slots on a cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira