You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Weiwei Yang (JIRA)" <ji...@apache.org> on 2017/12/22 02:04:00 UTC
[jira] [Comment Edited] (YARN-3409) Support Node Attribute
functionality
[ https://issues.apache.org/jira/browse/YARN-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300867#comment-16300867 ]
Weiwei Yang edited comment on YARN-3409 at 12/22/17 2:03 AM:
-------------------------------------------------------------
Hi [~Naganarasimha], [~leftnoteasy]
I went through the design doc, there is one thing I want to make sure it will be addressed,
Lets assume we have added attributes for a group of nodes, e.g NUM_OF_DISKS=<num_of_disk_on_this_node>. Then there are lots of requests coming without specifying any attribute, and assume they are allocated to all cluster nodes; then there is a request asking for {{NUM_OF_DISKS > 3}}, there is only 1 node satisfy this requirement but its resource is already used up by other containers. What will happen here? Will you preempt containers on this node to make room for such request?
The problem here is we need a way to specify the scheduling policy of an expression, e.g "hard" or "soft". When an expression is specified as "hard", that means yarn needs to preempt containers to satisfy the requirement as long as it is possible; "soft" means best-effort to offer such nodes. Default can be hard. Make sense?
was (Author: cheersyang):
Hi [~Naganarasimha], [~leftnoteasy]
I went through the design doc, there is one thing I want to make sure it will be addressed,
Lets assume we have added attributes for a group of nodes, e.g NUM_OF_DISKS=${num_of_disk_on_this_node}. Then there are lots of requests coming without specifying any attribute, and assume they are allocated to all cluster nodes; then there is a request asking for NUM_OF_DISKS > 3, there is only 1 node satisfy this requirement but its resource is already used up by other containers. What will happen here? Will you preempt containers on this node to make room for such request?
The problem here is we need a way to specify the scheduling policy of an expression, e.g "hard" or "soft". When an expression is specified as "hard", that means yarn needs to preempt containers to satisfy the requirement as long as it is possible; "soft" means best-effort to offer such nodes. Default can be hard. Make sense?
> Support Node Attribute functionality
> ------------------------------------
>
> Key: YARN-3409
> URL: https://issues.apache.org/jira/browse/YARN-3409
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: api, client, RM
> Reporter: Wangda Tan
> Assignee: Naganarasimha G R
> Attachments: 3409-apiChanges_v2.pdf (4).pdf, Constraint-Node-Labels-Requirements-Design-doc_v1.pdf, YARN-3409.WIP.001.patch
>
>
> Specify only one label for each node (IAW, partition a cluster) is a way to determinate how resources of a special set of nodes could be shared by a group of entities (like teams, departments, etc.). Partitions of a cluster has following characteristics:
> - Cluster divided to several disjoint sub clusters.
> - ACL/priority can apply on partition (Only market team / marke team has priority to use the partition).
> - Percentage of capacities can apply on partition (Market team has 40% minimum capacity and Dev team has 60% of minimum capacity of the partition).
> Attributes are orthogonal to partition, they’re describing features of node’s hardware/software just for affinity. Some example of attributes:
> - glibc version
> - JDK version
> - Type of CPU (x86_64/i686)
> - Type of OS (windows, linux, etc.)
> With this, application can be able to ask for resource has (glibc.version >= 2.20 && JDK.version >= 8u20 && x86_64).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org