[jira] [Commented] (YARN-8020) when DRF is used, preemption does not trigger due to incorrect idealAssigned

["Eric Payne (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-8020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397262#comment-16397262 ] 

Eric Payne commented on YARN-8020:
----------------------------------

[~kyungwan nam], on what version of YARN are you seeing this problem? My experience with DRF is different than is described above. I have investigated this on both 2.8 and 3.2 snapshot builds.

We are using the DRF calculator in large preemptable queues with various sizes of containers using both large memory or large vcores or both. Cross-queue preemption seems to be working well in general. I do see a corner case, but first I want to address your above comments.

bq. as a result, idealAssigned will be <Memory:-81GB, VCores:19>, which does not trigger preemption.
If one of the elements in the idealAssigned Resource is 0 or less than 0, preemption will not occur. This is so that preemption won't bring the queue too far below its guarantee for one of the elements. Having said that, it will preempt to a large extent even if it brings one of the elements below its guarantee, but if one of them goes to 0 or below in the idealAssigned Resource, it will stop preempting.

bq. avail: <Memory:181GB, Vcores:1>
Cross-queue preemption will not preempt if there are available resources in the cluster or queue. It depends on how many resources are being requested by the other queue, but even with 1 available vcore, preemption may choose not to preempt in this case as well.

Now on to my corner case.

I do not see a problem using DRF if the containers in the preemptable queue have a larger Resource element and the containers in the asking queue have smaller Resource elements. For example, it seems to work fine if Resources in the preemptable queue is using <Memory:1GB, VCores:10> containers and the asking queue is using smaller containers, for example <Memory:1GB, Vcores:3> containers.

The place where it seems to get stuck is when the containers in the preemptable queue are using one or more smaller Resource elements than the containers in the asking queue. For example, it will sometimes not preempt if the preemptable queue has containers using <Memory:0.5GB, VCores:1> and the asking queue queue has containers using <Memory:0.5GB, VCores:2>.

Even in the latter case, preemption will sometimes still occur, depending on the ratio of the sizes of each element to the ones in the ohter queue.

It would be helpful if you can provide a more detailed use case to describe exactly what you are seeing so I can try to reproduce it.

> when DRF is used, preemption does not trigger due to incorrect idealAssigned
> ----------------------------------------------------------------------------
>
>                 Key: YARN-8020
>                 URL: https://issues.apache.org/jira/browse/YARN-8020
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: kyungwan nam
>            Priority: Major
>
> I’ve met that Inter Queue Preemption does not work.
> It happens when DRF is used and submitting application with a large number of vcores.
> IMHO, idealAssigned can be set incorrectly by following code.
> {code}
> // This function "accepts" all the resources it can (pending) and return
> // the unused ones
> Resource offer(Resource avail, ResourceCalculator rc,
>     Resource clusterResource, boolean considersReservedResource) {
>   Resource absMaxCapIdealAssignedDelta = Resources.componentwiseMax(
>       Resources.subtract(getMax(), idealAssigned),
>       Resource.newInstance(0, 0));
>   // accepted = min{avail,
>   //               max - assigned,
>   //               current + pending - assigned,
>   //               # Make sure a queue will not get more than max of its
>   //               # used/guaranteed, this is to make sure preemption won't
>   //               # happen if all active queues are beyond their guaranteed
>   //               # This is for leaf queue only.
>   //               max(guaranteed, used) - assigned}
>   // remain = avail - accepted
>   Resource accepted = Resources.min(rc, clusterResource,
>       absMaxCapIdealAssignedDelta,
>       Resources.min(rc, clusterResource, avail, Resources
>           /*
>            * When we're using FifoPreemptionSelector (considerReservedResource
>            * = false).
>            *
>            * We should deduct reserved resource from pending to avoid excessive
>            * preemption:
>            *
>            * For example, if an under-utilized queue has used = reserved = 20.
>            * Preemption policy will try to preempt 20 containers (which is not
>            * satisfied) from different hosts.
>            *
>            * In FifoPreemptionSelector, there's no guarantee that preempted
>            * resource can be used by pending request, so policy will preempt
>            * resources repeatly.
>            */
>           .subtract(Resources.add(getUsed(),
>               (considersReservedResource ? pending : pendingDeductReserved)),
>               idealAssigned)));
> {code}
> let’s say,
> * cluster resource : <Memory:200GB, VCores:20>
> * idealAssigned(assigned): <Memory:100GB, VCores:10>
> * avail: <Memory:181GB, Vcores:1>
> * current: <Memory:19GB, Vcores:19>
> * pending: <Memory:0, Vcores:0>
> current + pending - assigned: <Memory:-181GB, Vcores:9>
> min ( avail, (current + pending - assigned) ) : <Memory:-181GB, Vcores:9>
> accepted: <Memory:-181GB, Vcores:9>
> as a result, idealAssigned will be <Memory:-81GB, VCores:19>, which does not trigger preemption.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org