[jira] [Comment Edited] (YARN-7972) Support inter-app placement constraints for allocation tags by application ID

["Weiwei Yang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-7972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16384418#comment-16384418 ] 

Weiwei Yang edited comment on YARN-7972 at 3/3/18 1:48 AM:
-----------------------------------------------------------

Hi [~asuresh]

Thanks for your comments, I introduced the Evaluable interface is to separate 1) parsing phase and 2) eval phase, before a namespace can be consumed by the allocation tags manager. This is because during phase 1, we are not able to get the exact scope of a namespace, as the parser has no concept of what is the current application, other applications or all applications etc. So this will be handled in phase 2. I think this makes code more readable.

{quote}
then we must ensure evaluate is always required to be called
{quote}
Makes sense. To achieve this, I can try to add a new class to represent the eval target, e.g {{TargetApplications}}
{code}
TargetApplications ta = new TargetApplications(currentAppId, allAppIds);
// this way we will get rid of if-else code like you pointed out, and move the logic to the eval method itself
namespace.evaluate(ta);
{code}
If this looks good to you, I will submit a new patch with this change.

{quote}
I assume later when we have NOT_SELF and multiple app scopes, then the size of set would be all other apps and each specific app respectively. right ?
{quote}
Correct. The idea is the eval phase will make sure a namespace is interpreted to a specific scope, AKA a set of app IDs. NOT_SELF, ALL, APP-LABEL will all need to implement its own eval method.

{quote}
Wouldn't creating that set at parse time be very in-efficient ?
{quote}
I am not sure if I get your point correctly, we are not creating a new set every time, we only need to pass current app ID and all app IDs (from allocation tags manager) to a namespace so it can be evaluated to a specific scope according to different namespace types. Would that be a concern?

Thanks

 


was (Author: cheersyang):
Hi [~asuresh]

Thanks for your comments, I introduced the Evaluable interface is to separate 1) parsing phase and 2) eval phase, before a namespace can be consumed by the allocation tags manager. This is because during phase 1, we are not able to get the exact scope of a namespace during parsing phase, as the parser has no concept of what is the current application, other applications or all applications etc. So this will be handled in phase 2. I think this makes code more readable.

{quote}
then we must ensure evaluate is always required to be called
{quote}
Makes sense. To achieve this, I can try to add a new class to represent the eval target, e.g {{TargetApplications}}
{code}
TargetApplications ta = new TargetApplications(currentAppId, allAppIds);
// this way we will get rid of if-else code like you pointed out, and move the logic to the eval method itself
namespace.evaluate(ta);
{code}
If this looks good to you, I will submit a new patch with this change.

{quote}
I assume later when we have NOT_SELF and multiple app scopes, then the size of set would be all other apps and each specific app respectively. right ?
{quote}
Correct. The idea is the eval phase will make sure a namespace is interpreted to a specific scope, AKA a set of app IDs. NOT_SELF, ALL, APP-LABEL will all need to implement its own eval method.

{quote}
Wouldn't creating that set at parse time be very in-efficient ?
{quote}
I am not sure if I get your point correctly, we are not creating a new set every time, we only need to pass current app ID and all app IDs (from allocation tags manager) to a namespace so it can be evaluated to a specific scope according to different namespace types. Would that be a concern?

Thanks

 

> Support inter-app placement constraints for allocation tags by application ID
> -----------------------------------------------------------------------------
>
>                 Key: YARN-7972
>                 URL: https://issues.apache.org/jira/browse/YARN-7972
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Weiwei Yang
>            Assignee: Weiwei Yang
>            Priority: Major
>         Attachments: YARN-7972.001.patch, YARN-7972.002.patch, YARN-7972.003.patch, YARN-7972.004.patch, YARN-7972.005.patch, YARN-7972.006.patch
>
>
> Per discussion in [this comment|https://issues.apache.org/jira/browse/YARN-6599focusedCommentId=16319662&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16319662] in  YARN-6599, we need to support inter-app PC for allocation tags.
> This will help to do better placement when dealing with potential competing resource applications, e.g don't place two tensorflow workers from two different applications on one same node.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org