[jira] [Commented] (YARN-4425) Pluggable sharing policy for Partition Node Label resources

["Wangda Tan (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-4425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15231431#comment-15231431 ] 

Wangda Tan commented on YARN-4425:
----------------------------------

[~Naganarasimha], [~xinxianyin],
Read the doc and took a very high level look at the patch, sorry for the huge delays. Some thoughts:

The new added policy seems like a backdoor of scheduler's capacity management: when under "NON_EXCLUSIVE" mode, scheduler completely depends on configured policy to decide who can get next resources.

And we need to make API of policy with more clear semantic: In existing scheduler, resource will be allocated on requested partition, and only request.partition = "" can get chance to allocate on other partitions when non-exclusive criteria is met.
In the new API, resource could be allocated on any partition regardless of requested partition. (depends on different policy implementation). Which will be conflict to our existing APIs for node partitions.

To me, sharing resource between partitions itself is not a clear API:
You can say partition A has total resource = 100G, partition B has total resource = 200G. But you cannot say: "under some conditions, partition A can use idle resources from partition B" -- because partition is not the entity which will consume resources.

Instead, it will be more clear to me to say:
1) Queue's shares of partitions could be dynamically adjusted, OR
2) Node's partition could be dynamically update


> Pluggable sharing policy for Partition Node Label resources
> -----------------------------------------------------------
>
>                 Key: YARN-4425
>                 URL: https://issues.apache.org/jira/browse/YARN-4425
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacity scheduler, resourcemanager
>            Reporter: Naganarasimha G R
>            Assignee: Naganarasimha G R
>         Attachments: ResourceSharingPolicyForNodeLabelsPartitions-V1.pdf, ResourceSharingPolicyForNodeLabelsPartitions-V2.pdf, YARN-4425.20160105-1.patch
>
>
> As part of support for sharing NonExclusive Node Label partitions in YARN-3214, NonExclusive partitions are shared only to Default Partitions and also have fixed rule when apps in default partitions makes use of resources of any NonExclusive partitions.
> There are many scenarios where in we require pluggable policy like MutliTenant, Hierarchical etc.. where in each partition can determine when they want to share the resources to other paritions and when other partitions wants to use resources from others
> More details in the attached document.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)