You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by as...@apache.org on 2017/09/12 20:41:42 UTC
[27/50] [abbrv] hadoop git commit: YARN-7056. Document Resource
Profiles feature. (Sunil G via wangda)
YARN-7056. Document Resource Profiles feature. (Sunil G via wangda)
Change-Id: I5f5d52cdf7aba4ac09684aab573e6916dbad183e
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/6b88cd19
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/6b88cd19
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/6b88cd19
Branch: refs/heads/YARN-5972
Commit: 6b88cd1901a65b977fed759e322dcd75a2cd33b6
Parents: 8050954
Author: Wangda Tan <wa...@apache.org>
Authored: Thu Sep 7 10:07:35 2017 -0700
Committer: Wangda Tan <wa...@apache.org>
Committed: Tue Sep 12 09:19:12 2017 -0700
----------------------------------------------------------------------
hadoop-project/src/site/site.xml | 1 +
.../src/site/markdown/ResourceProfiles.md | 116 +++++++++++++++++++
2 files changed, 117 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/6b88cd19/hadoop-project/src/site/site.xml
----------------------------------------------------------------------
diff --git a/hadoop-project/src/site/site.xml b/hadoop-project/src/site/site.xml
index e7af227..4685e2a 100644
--- a/hadoop-project/src/site/site.xml
+++ b/hadoop-project/src/site/site.xml
@@ -139,6 +139,7 @@
<item name="Graceful Decommission" href="hadoop-yarn/hadoop-yarn-site/GracefulDecommission.html"/>
<item name="Opportunistic Containers" href="hadoop-yarn/hadoop-yarn-site/OpportunisticContainers.html"/>
<item name="YARN Federation" href="hadoop-yarn/hadoop-yarn-site/Federation.html"/>
+ <item name="Resource Profiles" href="hadoop-yarn/hadoop-yarn-site/ResourceProfiles.html"/>
</menu>
<menu name="YARN REST APIs" inherit="top">
http://git-wip-us.apache.org/repos/asf/hadoop/blob/6b88cd19/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceProfiles.md
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceProfiles.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceProfiles.md
new file mode 100644
index 0000000..d422ab6
--- /dev/null
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceProfiles.md
@@ -0,0 +1,116 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+Hadoop: YARN Resource Profiles
+==============================
+
+Overview
+--------
+Resource profiles support in YARN helps to extend the YARN resource model to a more flexible model which makes it easier to add new countable resourcetypes. This solution also helps the users to submit jobs with ease to specify the resources they need.
+
+Resource model of YARN
+-----------------------
+Resource Manager will load a new configuration file named `resource-types.xml` to determine the set of resource types for which scheduling is enabled. Sample XML will look like below.
+
+```xml
+<configuration>
+ <property>
+ <name>yarn.resource-types</name>
+ <value>resource1, resource2</value>
+ </property>
+
+ <property>
+ <name>yarn.resource-types.resource1.units</name>
+ <value>G</value>
+ </property>
+</configuration>
+```
+
+Similarly, a new configuration file `node-resources.xml` will also be loaded by Node Manager where the resource capabilities of a node can be specified.
+
+```xml
+<configuration>
+ <property>
+ <name>yarn.nodemanager.resource-type.resource1</name>
+ <value>5G</value>
+ </property>
+
+ <property>
+ <name>yarn.nodemanager.resource-type.resource2</name>
+ <value>2m</value>
+ </property>
+
+</configuration>
+```
+
+Node Manager will use these custom resource types and will register it's capability to Resource Manager.
+
+Resource Profiles
+-----------------
+As per the updated resource model of YARN, specifying and configuring additional resource types for container allocation request will become more cumbersome. With the support of resource profiles, such requests could be placed by using simple profile names which has defined meaning from YARN.
+
+Admin could specify `resource-profiles.json` which indicates various profiles supported.
+
+```json
+{
+ "minimum": {
+ "memory-mb" : 1024,
+ "vcores" : 1
+ },
+ "default" : {
+ "memory-mb" : 2048,
+ "vcores" : 2
+ },
+ "maximum" : {
+ "memory-mb": 4096,
+ "vcores" : 4
+ },
+ "http" : {
+ "memory-mb" : 2048,
+ "vcores" : 2
+ }
+}
+```
+This indicates that profile named "minimum", "default", "maximum" and "http" is supported. Clients could easily use this name instead of specifying each resource details.
+
+Configurations
+-------------
+
+*In `yarn-site.xml`*
+
+| Configuration Property | Description |
+|:---- |:---- |
+| `yarn.resourcemanager.resource-profiles.enabled` | *(Required)* In the server side it indicates whether resource profiles support is enabled or not. Defaults to `false`. |
+
+Please note that, `resource-types.xml` and `node-resources.xml` file also need to be placed in conf directory if new resources are to be added to YARN.
+
+*In `resource-types.xml`*
+
+| Configuration Property | Value | Description |
+|:---- |:---- |:---- |
+| `yarn.resource-types` | resource1 | Custom resource |
+| `yarn.resource-types.resource1.units` | G | Default unit for resource1 type |
+
+*In `node-resources.xml`*
+
+| Configuration Property | Value | Description |
+|:---- |:---- |:---- |
+| `yarn.nodemanager.resource-type.resource1` | 5G | Resource capability for resource named 'resource1'. |
+
+Use it
+--------
+Distributed Shell is supporting this feature and user could specify a resource profile name which YARN is supporting while running a DS application.
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org