[jira] [Commented] (YARN-45) Scheduler feedback to AM to release containers

["Carlo Curino (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-45?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13648986#comment-13648986 ] 

Carlo Curino commented on YARN-45:
----------------------------------

Based on all the feedback here, including discussions with [~acmurthy], [~vinodkv], [~bikassaha], [~hitesh], [~sseth], and [~tucu00], we propose the following message be added to the
{{AllocateResponse}} (pseudo):

{noformat}
PreemptionMessage {
  StrictPreemptionContract {
    Set<PreemptionContainer> containers
  } strict;
  PreemptionContract {
    Set<PreemptionContainer> containers
    List<PreemptionResourceRequest> resources
  } contract;
} message
{noformat}

This has some advantages over the previous design:
# By adding {{PreemptionContainer}} and {{PreemptionResourceRequest}} (wrappers of {{ContainerId}} and {{ResourceRequest}} respectively) we can add attributes to each item later on, without breaking the protocol (e.g., [~sandyr]'s earlier suggestion of time).
# By separating strict and non-strict contracts, the RM can pull back specific containers or give the AM flexibility in satisfying the contract. It also allows the RM to simultaneously and unambiguously include requests with both constraints
# By including the list of containers in the {{PreemptionContract}} together with the resources, AMs have a slightly more restricted search space when compared to "match all the resources that _might_ be killed, determine preferences among them". Thus, simpler AMs can mostly ignore the interpretation of {{ResourceRequest}} and just follow the RM hint.

We're updating YARN-567, YARN-568, and YARN-569 to accommodate these changes, in addition to the rest of the downstream patches.
                
> Scheduler feedback to AM to release containers
> ----------------------------------------------
>
>                 Key: YARN-45
>                 URL: https://issues.apache.org/jira/browse/YARN-45
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>            Reporter: Chris Douglas
>            Assignee: Carlo Curino
>         Attachments: YARN-45.patch, YARN-45.patch, YARN-45.patch, YARN-45.patch, YARN-45.patch, YARN-45_summary_of_alternatives.pdf
>
>
> The ResourceManager strikes a balance between cluster utilization and strict enforcement of resource invariants in the cluster. Individual allocations of containers must be reclaimed- or reserved- to restore the global invariants when cluster load shifts. In some cases, the ApplicationMaster can respond to fluctuations in resource availability without losing the work already completed by that task (MAPREDUCE-4584). Supplying it with this information would be helpful for overall cluster utilization [1]. To this end, we want to establish a protocol for the RM to ask the AM to release containers.
> [1] http://research.yahoo.com/files/yl-2012-003.pdf

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira