[jira] [Commented] (YARN-1434) Single Job can affect fairshare of others

["Carlo Curino (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13829377#comment-13829377 ] 

Carlo Curino commented on YARN-1434:
------------------------------------

This has been observed while modifying the mapreduce AM behavior for other reasons. If the AM aggressively returns containers, it seems to be able to create the illusion to be under-capacity while wasting resources for everyone. A second job running in a separate queue (which was supposed to receive 50% of the cluster resources) was starved (only getting about 30% of the resources). This should be confirmed independently as the environment we observed this in had too much going on (i.e., this might be a false positive). 

If confirmed, this might be quite bad, as a single malevolent AM could affect the cluster utilization possibly by a lot.
  
[~sandyr], [~acmurthy]  thoughts?

> Single Job can affect fairshare of others
> -----------------------------------------
>
>                 Key: YARN-1434
>                 URL: https://issues.apache.org/jira/browse/YARN-1434
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: resourcemanager
>            Reporter: Carlo Curino
>            Priority: Minor
>
> A job receiving containers and deciding not to use them and yielding them back in the next heartbeat could significantly affect the amount of resources given to other jobs. 
> This is because by yielding containers back the job appears always to be under-capacity (more than others) so it is picked to be the next to receive containers.
> Observed by Robert Grandl, to be independently confirmed.



--
This message was sent by Atlassian JIRA
(v6.1#6144)