[jira] [Commented] (YARN-5325) Stateless ARMRMProxy policies implementation

["Carlo Curino (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15380918#comment-15380918 ] 

Carlo Curino commented on YARN-5325:
------------------------------------

The first patch provides 2 implementations for the {{ARMRMProxyFederationPolicy}}:
 # a simple broadcast policy that replicate the user asks to all sub-clusters
 # a sophisticated policy ({{LocalityMulticastAMRMProxyFederationPolicy}} that: 
  ## Directs node are rack local asks to sub-clusters that own the node and the rack. (in some deployments we stripe across racks, so the racks are not resolved to one subcluster and are thus treated as an ANY)
 ## Directs ANY that correspond to node/rack local requests to the corresponding sub-clusters by "splitting" the ANY according to how many containers are requested (with locality) to each sub-clusters.
 ## Directs ANY that are not associated with a localized request to subclusters according to a linear combination of policy weights and headroom information. An {{alpha}} parameter is used to choose how much headroom plays a role here. 
 ## Directs ANY requesting zero containers (user canceling their asks) to all sub-clusters we interacted with so far. 
## This policy minimizes the work done by RMs by ensuring that the sum of the request we send out is close to the user ask (might exceed by at most |sub-clusters| due to rounding to the closest integral number of containers).
 
The tests verify much of this behavior (though more should be done in testing). 

Per discussions with [~subru] we should probably work on optimizing the policy by factoring out sum-of-weight recomputations, etc... 

> Stateless ARMRMProxy policies implementation
> --------------------------------------------
>
>                 Key: YARN-5325
>                 URL: https://issues.apache.org/jira/browse/YARN-5325
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager, resourcemanager
>            Reporter: Carlo Curino
>            Assignee: Carlo Curino
>         Attachments: YARN-5325.01.patch
>
>
> This JIRA tracks policies in the AMRMProxy that decide how to forward ResourceRequests, without maintaining substantial state across decissions (e.g., broadcast).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org