You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Konstantinos Karanasos (JIRA)" <ji...@apache.org> on 2016/12/08 01:30:59 UTC
[jira] [Comment Edited] (YARN-3409) Add constraint node labels
[ https://issues.apache.org/jira/browse/YARN-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15730729#comment-15730729 ]
Konstantinos Karanasos edited comment on YARN-3409 at 12/8/16 1:30 AM:
-----------------------------------------------------------------------
Hey guys, apologies for the late reply.
Here are my thoughts...
bq. Add a new field for constraint expression, and also for affnity/anti-affinity (Per suggested by Kostas). This should have minimum impact to existing features. But after this, the "nodeLabelExpression becomes a little ambiguous, we may need to deprecate existing nodeLabelExpression.
Agreed with that, with one clarification: do you mean having an extra affinity/anti-affinity constraint expression or use the same constraint expression? Probably we will need a separate one.
bq. Extend existing NodeLabel object to support node constraint, we only need two additional field to support node constraint. 1) isNodeConstraint 2) Value (For example, we can have a constraint named jdk-verion, and value could be 6/7/8).
I followed your discussion on this and on evaluating the constraints. I also had an offline discussion with [~chris.douglas].
I will suggest to have an even simpler approach than the one Wangda proposed.
I believe we should have a first version with just boolean expressions, that is, simply request whether a label exists or not (possibly including negation of boolean expressions).
In other words, I suggest to have neither a constraint type nor a value.
Let's have a first simple version of (boolean) labels that works. In a future iteration of this, we can add attributes (i.e., with values) instead of labels.
Having simple labels allows us to bypass the problem of constraint types. Like Wangda says, constraint types are not really solving the problem of comparing values, given that people will right their values in different formats. You can also give a look at YARN-44676 for an efficient boolean expression matcher.
For example, using simple labels, one node can be annotated with label "Java6". Then a task that requires at least Java 5 can request for a node with "Java5 || Java6". I think that with our current use cases, this will be sufficient.
Let me know what you think.
was (Author: kkaranasos):
Hey guys, apologies for the late reply.
Here are my thoughts...
bq. Add a new field for constraint expression, and also for affnity/anti-affinity (Per suggested by Kostas). This should have minimum impact to existing features. But after this, the "nodeLabelExpression becomes a little ambiguous, we may need to deprecate existing nodeLabelExpression.
Agreed with that, with one clarification: do you mean having an extra affinity/anti-affinity constraint expression or use the same constraint expression? Probably we will need a separate one.
bq. Extend existing NodeLabel object to support node constraint, we only need two additional field to support node constraint. 1) isNodeConstraint 2) Value (For example, we can have a constraint named jdk-verion, and value could be 6/7/8).
I followed your discussion on this and on evaluating the constraints. I also had an offline discussion with [~chris.douglas].
I will suggest to have an even simpler approach than the one Wangda proposed.
I believe we should have a first version with just boolean expressions, that is, simply request whether a label exists or not (possibly including negation of boolean expressions).
In other words, I suggest to have neither a constraint type nor a value.
Let's have a first simple version of (boolean) labels that works. In a future iteration of this, we can add attributes (i.e., with values) instead of labels.
Having simple labels allows us to bypass the problem of constraint types. Like Wangda says, constraint types are not really solving the problem of comparing values, given that people will right their values in different formats. You can also give a look at YARN-4467 for an efficient boolean expression matcher.
For example, using simple labels, one node can be annotated with label "Java6". Then a task that requires at least Java 5 can request for a node with "Java5 || Java6". I think that with our current use cases, this will be sufficient.
Let me know what you think.
> Add constraint node labels
> --------------------------
>
> Key: YARN-3409
> URL: https://issues.apache.org/jira/browse/YARN-3409
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, capacityscheduler, client
> Reporter: Wangda Tan
> Assignee: Naganarasimha G R
> Attachments: Constraint-Node-Labels-Requirements-Design-doc_v1.pdf, YARN-3409.WIP.001.patch
>
>
> Specify only one label for each node (IAW, partition a cluster) is a way to determinate how resources of a special set of nodes could be shared by a group of entities (like teams, departments, etc.). Partitions of a cluster has following characteristics:
> - Cluster divided to several disjoint sub clusters.
> - ACL/priority can apply on partition (Only market team / marke team has priority to use the partition).
> - Percentage of capacities can apply on partition (Market team has 40% minimum capacity and Dev team has 60% of minimum capacity of the partition).
> Constraints are orthogonal to partition, they’re describing attributes of node’s hardware/software just for affinity. Some example of constraints:
> - glibc version
> - JDK version
> - Type of CPU (x86_64/i686)
> - Type of OS (windows, linux, etc.)
> With this, application can be able to ask for resource has (glibc.version >= 2.20 && JDK.version >= 8u20 && x86_64).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org