[jira] [Commented] (YARN-4902) [Umbrella] Generalized and unified scheduling-strategies in YARN

["Wangda Tan (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-4902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15409750#comment-15409750 ] 

Wangda Tan commented on YARN-4902:
----------------------------------

Thanks for sharing this, [~kkaranasos]/[~pg1712@imperial.ac.uk],

I took a quick look at the design doc and POC patch.

*For the design doc:*
1) From the requirement's perspective, I didn't see new things, please remind me if I missed anything
- The cardinality constraints is placement_set with maximum_concurrency constraint: see {{(4.3.3) Placement Strategy}} in my design doc.
- The dynamic tags for node properties (like hardware configs, etc.), is node constraints.
- The dynamic tags for application, is it as same as allocation tags? Using node label manager or not is a implementation decision to me
2) I'm not sure how LRA planner will look like, should it be a separate scheduler running in parallel? I didn't see your patch uses that approach.

*For the patch:*
3) It might be better to implement complex scheduling logics like affinity-between-apps and cardinality in a global scheduling way. (YARN-5139)

4) Will this patch support anti-affinity / affinity between apps? I uploaded my latest POC patch to YARN-1042, it supports affinity/anti-affinity for inter/intra apps. We can easily extend it to support intra/inter resource request within the app.

5) Major logic of this patch depends on node label manager dynamic tag changes. First of all, I'm not sure if NLM works efficiently when node label changes rapidly (we could update label on node when allocate / release every container). And I'm not sure how you plan to avoid malicious application add labels. For example if a distributed shell application claims it is a "hbase master" just for fun, how to enforce cardinality logics like "only put 10 HBase masters in the rack"?

*Suggestions*
- Could you take a look at global scheduling patch which I attached to YARN-5139 to see if it is possible to build new features added in your patch on top of the global scheduling framework? And also please share your thoughts about what's your overall feedbacks to the global scheduling framework like efficiency, extensibility, etc.
- It will be better to design Java API for this ticket, both of our poc patches (this one and the one I attached to YARN-1042) don't have a solid API definition. It is very important to define API first, could you help with API definition works?

> [Umbrella] Generalized and unified scheduling-strategies in YARN
> ----------------------------------------------------------------
>
>                 Key: YARN-4902
>                 URL: https://issues.apache.org/jira/browse/YARN-4902
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Wangda Tan
>         Attachments: Generalized and unified scheduling-strategies in YARN -v0.pdf, LRA-scheduling-design.v0.pdf, YARN-5468.prototype.patch
>
>
> Apache Hadoop YARN's ResourceRequest mechanism is the core part of the YARN's scheduling API for applications to use. The ResourceRequest mechanism is a powerful API for applications (specifically ApplicationMasters) to indicate to YARN what size of containers are needed, and where in the cluster etc.
> However a host of new feature requirements are making the API increasingly more and more complex and difficult to understand by users and making it very complicated to implement within the code-base.
> This JIRA aims to generalize and unify all such scheduling-strategies in YARN.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org