[jira] [Commented] (YARN-1769) CapacityScheduler: Improve reservations

["Jason Lowe (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-1769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13960541#comment-13960541 ] 

Jason Lowe commented on YARN-1769:
----------------------------------

The patch no longer applies cleanly after YARN-1512.  Other comments on the patch:

- Nit: In LeafQueue.assignToQueue we could cache Resource.add(usedResources, required) in a local when we're computing potentialNewCapacity so we don't have to recompute it as part of the potentialNewWithoutReservedCapacity computation
- LeafQueue.assignToQueue and LeafQueue.assignToUser don't seem to need the new priority argument, and therefore LeafQueue.checkLimitsToReserve wouldn't seem to need it either once those others are updated.
- Should FiCaSchedulerApp getAppToUnreserve really be called getNodeIdToUnreserve or geNodeToUnreserve, since it's returning a node ID rather than an app?
- In LeafQueue.findNodeToUnreserve, isn't it kinda bad if the app thinks it has reservations on the node but the scheduler doesn't know about it?  Wondering if the bookkeeping is messed up at that point therefore something a bit more than debug is an appropriate log level and if further fixup is needed.
- LeafQueue.findNodeToUnreserve is adjusting the headroom when it unreserves, but I don't see other unreservations doing a similar calculation.  Wondering if this fixup is something that should have been in completedContainer or needs to be done elsewhere?  I could easily be missing something here but asking just in case other unreservation situations also need to have the headroom fixed.
- LeafQueue.assignContainer uses the much more expensive scheduler.getConfiguration().getReservationContinueLook() when it should be able to use the reservationsContinueLooking member instead.
- LeafQueue.getReservationContinueLooking should be package private
- Nit: LeafQueue.assignContainer has some reformatting of the log message after the "// Inform the node" comment which was clearer to read/maintain before since the label and the value were always on a line by themselves.  Same goes for the "Reserved container" log towards the end of the method.
- Ultra-Nit: ParentQueue.setupQueueConfig's log message should have the reservationsContinueLooking on the previous line to match the style of other label/value pairs in the log message.
- ParentQueue.getReservationContinueLooking should be package private.

> CapacityScheduler:  Improve reservations
> ----------------------------------------
>
>                 Key: YARN-1769
>                 URL: https://issues.apache.org/jira/browse/YARN-1769
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: capacityscheduler
>    Affects Versions: 2.3.0
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>         Attachments: YARN-1769.patch, YARN-1769.patch, YARN-1769.patch, YARN-1769.patch, YARN-1769.patch, YARN-1769.patch, YARN-1769.patch
>
>
> Currently the CapacityScheduler uses reservations in order to handle requests for large containers and the fact there might not currently be enough space available on a single host.
> The current algorithm for reservations is to reserve as many containers as currently required and then it will start to reserve more above that after a certain number of re-reservations (currently biased against larger containers).  Anytime it hits the limit of number reserved it stops looking at any other nodes. This results in potentially missing nodes that have enough space to fullfill the request.   
> The other place for improvement is currently reservations count against your queue capacity.  If you have reservations you could hit the various limits which would then stop you from looking further at that node.  
> The above 2 cases can cause an application requesting a larger container to take a long time to gets it resources.  
> We could improve upon both of those by simply continuing to look at incoming nodes to see if we could potentially swap out a reservation for an actual allocation. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)