You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-commits@hadoop.apache.org by sa...@apache.org on 2013/08/26 21:59:04 UTC
svn commit: r1517668 - in /hadoop/common/branches/branch-2.1-beta/hadoop-yarn-project: CHANGES.txt hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm

Author: sandy
Date: Mon Aug 26 19:59:04 2013
New Revision: 1517668

URL: http://svn.apache.org/r1517668
Log:
YARN-1093. Corrections to Fair Scheduler documentation (Wing Yew Poon via Sandy Ryza)

Modified:
    hadoop/common/branches/branch-2.1-beta/hadoop-yarn-project/CHANGES.txt
    hadoop/common/branches/branch-2.1-beta/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm

Modified: hadoop/common/branches/branch-2.1-beta/hadoop-yarn-project/CHANGES.txt
URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-2.1-beta/hadoop-yarn-project/CHANGES.txt?rev=1517668&r1=1517667&r2=1517668&view=diff
==============================================================================
--- hadoop/common/branches/branch-2.1-beta/hadoop-yarn-project/CHANGES.txt (original)
+++ hadoop/common/branches/branch-2.1-beta/hadoop-yarn-project/CHANGES.txt Mon Aug 26 19:59:04 2013
@@ -13,6 +13,9 @@ Release 2.1.1-beta - UNRELEASED
     YARN-1074. Cleaned up YARN CLI application list to only display running
     applications by default. (Xuan Gong via vinodkv)
     
+    YARN-1093. Corrections to Fair Scheduler documentation (Wing Yew Poon via
+    Sandy Ryza)
+
   OPTIMIZATIONS
 
   BUG FIXES

Modified: hadoop/common/branches/branch-2.1-beta/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm
URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-2.1-beta/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm?rev=1517668&r1=1517667&r2=1517668&view=diff
==============================================================================
--- hadoop/common/branches/branch-2.1-beta/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm (original)
+++ hadoop/common/branches/branch-2.1-beta/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm Mon Aug 26 19:59:04 2013
@@ -47,7 +47,7 @@ Hadoop MapReduce Next Generation - Fair 
 
   The scheduler organizes apps further into "queues", and shares resources
   fairly between these queues. By default, all users share a single queue,
-  called âdefaultâ. If an app specifically lists a queue in a container resource
+  named âdefaultâ. If an app specifically lists a queue in a container resource
   request, the request is submitted to that queue. It is also possible to assign
   queues based on the user name included with the request through
   configuration. Within each queue, a scheduling policy is used to share
@@ -85,7 +85,7 @@ Hadoop MapReduce Next Generation - Fair 
   their parents in the fair scheduler configuration file.
   
   A queue's name starts with the names of its parents, with periods as
-  separators. So a queue named "queue1" under the root named, would be referred
+  separators. So a queue named "queue1" under the root queue, would be referred
   to as "root.queue1", and a queue named "queue2" under a queue named "parent1"
   would be referred to as "root.parent1.queue2". When referring to queues, the
   root part of the name is optional, so queue1 could be referred to as just
@@ -118,22 +118,23 @@ Hadoop MapReduce Next Generation - Fair 
 
   Customizing the Fair Scheduler typically involves altering two files. First, 
   scheduler-wide options can be set by adding configuration properties in the 
-  fair-scheduler.xml file in your existing configuration directory. Second, in 
+  yarn-site.xml file in your existing configuration directory. Second, in 
   most cases users will want to create a manifest file listing which queues 
   exist and their respective weights and capacities. The location of this file 
-  is flexible - but it must be declared in fair-scheduler.xml. 
+  is flexible - but it must be declared in yarn-site.xml.
 
  * <<<yarn.scheduler.fair.allocation.file>>>
 
    * Path to allocation file. An allocation file is an XML manifest describing
      queues and their properties, in addition to certain policy defaults. This file
      must be in XML format as described in the next section.
+     Defaults to fair-scheduler.xml in configuration directory.
 
  * <<<yarn.scheduler.fair.user-as-default-queue>>>
 
     * Whether to use the username associated with the allocation as the default 
       queue name, in the event that a queue name is not specified. If this is set 
-      to "false" or unset, all jobs have a shared default queue, called "default".
+      to "false" or unset, all jobs have a shared default queue, named "default".
       Defaults to true.
 
  * <<<yarn.scheduler.fair.preemption>>>
@@ -178,14 +179,15 @@ Hadoop MapReduce Next Generation - Fair 
 
 Allocation file format
 
-  The allocation file must be in XML format. The format contains four types of
+  The allocation file must be in XML format. The format contains five types of
   elements:
 
  * <<Queue elements>>, which represent queues. Each may contain the following
      properties:
 
    * minResources: minimum resources the queue is entitled to, in the form
-     "X mb, Y vcores". If a queue's minimum share is not satisfied, it will be
+     "X mb, Y vcores". For the single-resource fairness policy, the vcores
+     value is ignored. If a queue's minimum share is not satisfied, it will be
      offered available resources before any other queue under the same parent.
      Under the single-resource fairness policy, a queue
      is considered unsatisfied if its memory usage is below its minimum memory
@@ -199,7 +201,8 @@ Allocation file format
      may be using those resources.
 
    * maxResources: maximum resources a queue is allowed, in the form
-     "X mb, Y vcores". A queue will never be assigned a container that would
+     "X mb, Y vcores". For the single-resource fairness policy, the vcores
+     value is ignored. A queue will never be assigned a container that would
      put its aggregate usage over this limit.
 
    * maxRunningApps: limit the number of apps from the queue to run at once
@@ -234,19 +237,23 @@ Allocation file format
    its fair share before it will try to preempt containers to take resources from
    other queues.
 
+ * <<A defaultQueueSchedulingPolicy element>>, which sets the default scheduling 
+   policy for queues; overriden by the schedulingPolicy element in each queue
+   if specified. Defaults to "fair".
+
   An example allocation file is given here:
 
 ---
 <?xml version="1.0"?>
 <allocations>
   <queue name="sample_queue">
-    <minResources>10000 mb</minResources>
-    <maxResources>90000 mb</maxResources>
+    <minResources>10000 mb,0vcores</minResources>
+    <maxResources>90000 mb,0vcores</maxResources>
     <maxRunningApps>50</maxRunningApps>
     <weight>2.0</weight>
     <schedulingPolicy>fair</schedulingPolicy>
     <queue name="sample_sub_queue">
-      <minResources>5000 mb</minResources>
+      <minResources>5000 mb,0vcores</minResources>
     </queue>
   </queue>
   <user name="sample_user">