[jira] [Commented] (MAPREDUCE-3859) CapacityScheduler incorrectly utilizes extra-resources of queue for high-memory jobs

["Mike Roark (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609117#comment-13609117 ] 

Mike Roark commented on MAPREDUCE-3859:
---------------------------------------

Any updates on this bug? It is affecting us as well.. it has a pretty bad effect on cluster utilization for these kinds of jobs. I tested this locally to see if it was this issue, results below:

10 datanodes, each with 6 slots for reducers, 60 reducer slots total. No other jobs running.
Running jobs in a queue which has "Reduce tasks, Capacity: 4 slots, Maximum capacity: 60 slots"
Reducer sleeps for a while. This allows me to check steady state reducer slot allocation.

||mapred.reduce.tasks||slots per reducer||expected running reduce tasks||running reduce tasks||using slots||percent of capacity||notes||
|30|1|30|30|30|750| |
|30|2|30|{color:red}16{color}|32|800| |
|30|3|20|{color:red}1{color}|3|75| really bad |
|30|4|10|{color:red}8{color}|32|800| |
|30|5|10|{color:red}8{color}|40|1000| |
|30|6|10|{color:red}8{color}|48|1200| |
|30|7(err)|0|0|0|0| job hangs, but I expected this |
                
> CapacityScheduler incorrectly utilizes extra-resources of queue for high-memory jobs
> ------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-3859
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3859
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: capacity-sched
>    Affects Versions: 1.0.0
>         Environment: CDH3u1
>            Reporter: Sergey Tryuber
>         Attachments: test-to-fail.patch.txt
>
>
> Imagine, we have a queue A with capacity 10 slots and 20 as extra-capacity, jobs which use 3 map slots will never consume more than 9 slots, regardless how many free slots on a cluster.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira