You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Eric Payne (Jira)" <ji...@apache.org> on 2021/09/30 21:23:00 UTC
[jira] [Commented] (YARN-10938) Support reservation scheduling
enabled switch for capacity scheduler
[ https://issues.apache.org/jira/browse/YARN-10938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17423018#comment-17423018 ]
Eric Payne commented on YARN-10938:
-----------------------------------
[~Weihao Zheng]
Reservations are somewhat integral to the behavior of the Capacity Scheduler. The Capacity Scheduler is based on FIFO order, and reservations preserves the submission order of applications.
If you turn off reservation scheduling, an app that needs larger containers will be starved by those with smaller containers. For example,
- Cluster has 3 nodes of 10GB each.
- Cluster has 1 Queue.
- App1 requests 27 1.5GB containers, so each node has 9GB used.
- App2 launches and requests a 3GB AM container to start running.
- App3 needs 500 1GB containers.
Without reservations, App2 is never scheduled until App3 is finished, even though App2 got there first. This is because as soon as any 1.5GB container is released from App1, App3 jumps in and takes at least 1GB container on that node.
With reservations enabled, App2 can reserve on one of the nodes even though the total 3GB is not on the same node. The 3GB gets charged to the queue and when any of the 3 nodes has 3GB or more free, re-reservation kicks in and App2 gets the container. This preserves the order of submission.
> Support reservation scheduling enabled switch for capacity scheduler
> --------------------------------------------------------------------
>
> Key: YARN-10938
> URL: https://issues.apache.org/jira/browse/YARN-10938
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacity scheduler
> Reporter: Weihao Zheng
> Assignee: Weihao Zheng
> Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Currently the CapacityScheduler uses reservations in order to handle requests for large containers and the fact there might not currently be enough space available on a single host. But this algorithm is not suitable for small cluster which only have very limited resources. So we can add a switch property in capacity scheduler's configuration to avoid reservation scheduling in these use cases.
> CHANGES:
> Add {{"yarn.scheduler.capacity.reservation.enabled"}} in capacity scheduler's configuration.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org