[jira] [Commented] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions

["Sandy Ryza (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13955960#comment-13955960 ] 

Sandy Ryza commented on YARN-596:
---------------------------------

Thanks.  The patch is looking almost done.  I like that you replaced the O(n log n) sort call in preemptContainer with an O(n) iteration.   Just a few more nits:

{code}
+      LOG.debug("Queue " + getName() + " is going to preempt a container " +
+          "from its childQueues.");
{code}
This doesn't make sense in FSLeafQueue, which can't have child queues.

{code}
+    // Let the selected queue to preempt
+    if (candidateQueue != null) {
+      toBePreempted = candidateQueue.preemptContainer();
+    }
{code}
Did you mean "Let the selected queue choose which of its containers to preempt"?

For preemptContainerPreCheck, it would be good to take multiple resources into account (using the DefaultResourceCalculator will only apply to memory).  Resources.fitsIn(getResourceUsage(), getFairShare) can be used to determine whether a Schedulable is safe from preemption.

Lastly, can you add a test that makes sure that containers from apps that are higher over their fair share get preempted first, even when containers from other apps that are over their fair share have lower priorities?

> In fair scheduler, intra-application container priorities affect inter-application preemption decisions
> -------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-596
>                 URL: https://issues.apache.org/jira/browse/YARN-596
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: scheduler
>    Affects Versions: 2.0.3-alpha
>            Reporter: Sandy Ryza
>            Assignee: Sandy Ryza
>         Attachments: YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch
>
>
> In the fair scheduler, containers are chosen for preemption in the following way:
> All containers for all apps that are in queues that are over their fair share are put in a list.
> The list is sorted in order of the priority that the container was requested in.
> This means that an application can shield itself from preemption by requesting it's containers at higher priorities, which doesn't really make sense.
> Also, an application that is not over its fair share, but that is in a queue that is over it's fair share is just as likely to have containers preempted as an application that is over its fair share.



--
This message was sent by Atlassian JIRA
(v6.2#6252)