[jira] [Comment Edited] (YARN-5323) Policies APIs (for Router and AMRMProxy policies)

["Carlo Curino (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-5323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15433409#comment-15433409 ] 

Carlo Curino edited comment on YARN-5323 at 8/23/16 7:05 PM:
-------------------------------------------------------------

[~wangda] thanks for reviewing.

Answering your questions:
 # The AMRMProxyFederationPolicy is used in the AMRMProxyService to determine where to send the ResourceRequests, while the RouterFederationPolicy is used by the router to determine where to send the AM. This patch is a bit of a "leaf" patch, other patches will use this code, so it make sense you don't get much context from this patch alone. 
 # We could remove it, but the key intuition we have is that the two AMRMProxyPolicy and RouterPolicy, while completely independent (run on different nodes and do different things), need to work in collaboration with each other to achieve a consistent system behavior. We are thus trying to use the code structure (e.g., FederationPolicy) as a way to tie them together. This should help administrators to avoid some of the obvious mistakes (e.g., combining a load-balancing RouterPolicy with a fault-tolerance focused AMRMProxyPolicy)---if you want you can, but have to implement a class and configure the system to use it, which should make you think harder about what you are doing. Also we are using this writer + getter api as a way to handle the entire lifecycle of ser/deser of the policy configuration through the store. By construction we want the configuration to be rather opaque (expandibility), but we need to keep the lifecycle management sane, this is our attempt. 
 # Works for me, though I don't think is the common way for other "*Context" objects around YARN.
 # The Router routes more than the APPs, all client-to-rm protocols go through there... Most of the decisions of where to route are not policy based, as they are deterministic (e.g., if you send a kill should go where the app was running). I think the policy name should say Router to indicate that is used by the router. Also in longer term I can see more policy behaviors to be added to the router, as we become more sophisticated and operations that are fully manual today (capacity allocation to subclusters) get automated. 

I hope this helps to clarify your concerns.


was (Author: curino):
[~wangda] thanks for reviewing.

Answering your questions:
 # The AMRMProxyFederationPolicy is used in the AMRMProxyService to determine where to send the ResourceRequests, while the RouterFederationPolicy is used by the router to determine where to send the AM. This patch is a bit of a "leaf" patch, other patches will use this code, so it make sense you don't get much context from this patch alone. 
 # We could remove it, but the key intuition we have is that the two AMRMProxyPolicy and RouterPolicy, while completely independent (run on different nodes and do different things), need to work in collaboration with each other to achieve a consistent system behavior. We are thus trying to use the code structure (e.g., FederationPolicy) as a way to tie them together. This should help administrators to avoid some of the obvious mistakes (e.g., combining a load-balancing RouterPolicy with a fault-tolerance focused AMRMProxyPolicy)---if you want you can, but have to implement a class and configure the system to use it, which should make you think harder about what you are doing. Also we are using this writer + getter api as a way to handle the entire lifecycle of ser/deser of the policy configuration through the store. By construction we want the configuration to be rather opaque (expandibility), but we need to keep the lifecycle management sane, this is our attempt. 
 # Works for me, though I don't think is the common way for other "*Context" objects around YARN.
 # The Router routes more than the APPs, all client-to-rm protocols go through there... Most of the decisions of where to route are not policy based, as they are deterministic (e.g., if you send a kill should go where the app was running). I think the policy name should say Router to indicate that is used by the router. Also in longer term I can see more policy behaviors to be added to the router, as we become more sophisticated and operations that are fully manual today (capacity allocation to subclusters) get automated. 

I hope this help clarify your concerns.

> Policies APIs (for Router and AMRMProxy policies)
> -------------------------------------------------
>
>                 Key: YARN-5323
>                 URL: https://issues.apache.org/jira/browse/YARN-5323
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager, resourcemanager
>    Affects Versions: YARN-2915
>            Reporter: Carlo Curino
>            Assignee: Carlo Curino
>         Attachments: YARN-5323-YARN-2915.05.patch, YARN-5323.01.patch, YARN-5323.02.patch, YARN-5323.03.patch, YARN-5323.04.patch
>
>
> This JIRA tracks APIs for the policies that will guide the Router and AMRMProxy decisions on where to fwd the jobs submission/query requests as well as ResourceRequests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org