You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by om...@apache.org on 2008/10/20 18:56:18 UTC
svn commit: r706338 [4/4] - in /hadoop/core/trunk: ./ docs/
src/contrib/capacity-scheduler/ src/docs/src/documentation/content/xdocs/
Modified: hadoop/core/trunk/docs/native_libraries.html
URL: http://svn.apache.org/viewvc/hadoop/core/trunk/docs/native_libraries.html?rev=706338&r1=706337&r2=706338&view=diff
==============================================================================
--- hadoop/core/trunk/docs/native_libraries.html (original)
+++ hadoop/core/trunk/docs/native_libraries.html Mon Oct 20 09:56:17 2008
@@ -153,6 +153,9 @@
<a href="hod.html">Hadoop On Demand</a>
</div>
<div class="menuitem">
+<a href="capacity_scheduler.html">Capacity Scheduler</a>
+</div>
+<div class="menuitem">
<a href="api/index.html">API Docs</a>
</div>
<div class="menuitem">
Modified: hadoop/core/trunk/docs/quickstart.html
URL: http://svn.apache.org/viewvc/hadoop/core/trunk/docs/quickstart.html?rev=706338&r1=706337&r2=706338&view=diff
==============================================================================
--- hadoop/core/trunk/docs/quickstart.html (original)
+++ hadoop/core/trunk/docs/quickstart.html Mon Oct 20 09:56:17 2008
@@ -153,6 +153,9 @@
<a href="hod.html">Hadoop On Demand</a>
</div>
<div class="menuitem">
+<a href="capacity_scheduler.html">Capacity Scheduler</a>
+</div>
+<div class="menuitem">
<a href="api/index.html">API Docs</a>
</div>
<div class="menuitem">
Modified: hadoop/core/trunk/docs/streaming.html
URL: http://svn.apache.org/viewvc/hadoop/core/trunk/docs/streaming.html?rev=706338&r1=706337&r2=706338&view=diff
==============================================================================
--- hadoop/core/trunk/docs/streaming.html (original)
+++ hadoop/core/trunk/docs/streaming.html Mon Oct 20 09:56:17 2008
@@ -156,6 +156,9 @@
<a href="hod.html">Hadoop On Demand</a>
</div>
<div class="menuitem">
+<a href="capacity_scheduler.html">Capacity Scheduler</a>
+</div>
+<div class="menuitem">
<a href="api/index.html">API Docs</a>
</div>
<div class="menuitem">
Modified: hadoop/core/trunk/src/contrib/capacity-scheduler/README
URL: http://svn.apache.org/viewvc/hadoop/core/trunk/src/contrib/capacity-scheduler/README?rev=706338&r1=706337&r2=706338&view=diff
==============================================================================
--- hadoop/core/trunk/src/contrib/capacity-scheduler/README (original)
+++ hadoop/core/trunk/src/contrib/capacity-scheduler/README Mon Oct 20 09:56:17 2008
@@ -10,126 +10,7 @@
This package implements a scheduler for Map-Reduce jobs, called Capacity
Task Scheduler (or just Capacity Scheduler), which provides a way to share
-large clusters. The scheduler provides the following features (which are
-described in detail in HADOOP-3421):
+large clusters.
-* Support for queues, where a job is submitted to a queue.
-* Queues are guaranteed a fraction of the capacity of the grid (their
- 'guaranteed capacity') in the sense that a certain capacity of resources
- will be at their disposal. All jobs submitted to the queues of an Org will
- have access to the capacity guaranteed to the Org.
-* Free resources can be allocated to any queue beyond its guaranteed capacity.
- These excess allocated resources can be reclaimed and made available to
- another queue in order to meet its capacity guarantee.
-* The scheduler guarantees that excess resources taken from a queue will be
- restored to it within N minutes of its need for them.
-* Queues optionally support job priorities (disabled by default).
-* Within a queue, jobs with higher priority will have access to the queue's
- resources before jobs with lower priority. However, once a job is running, it
- will not be preempted for a higher priority job.
-* In order to prevent one or more users from monopolizing its resources, each
- queue enforces a limit on the percentage of resources allocated to a user at
- any given time, if there is competition for them.
-* Support for memory-intensive jobs, wherein a job can optionally specify
- higher memory-requirements than the default, and the tasks of the job will
- only be run on TaskTrackers that have enough memory to spare.
-
-Whenever a TaskTracker is free, the Capacity Scheduler first picks a queue
-that needs to reclaim any resources the earliest. If no such queue is found,
-it then picks a queue which has most free space (whose ratio of # of running
-slots to guaranteed capacity is the lowest).
-
---------------------------------------------------------------------------------
-
-BUILDING:
-
-In HADOOP_HOME, run ant package to build Hadoop and its contrib packages.
-
---------------------------------------------------------------------------------
-
-INSTALLING:
-
-To run the capacity scheduler in your Hadoop installation, you need to put it
-on the CLASSPATH. The easiest way is to copy the
-hadoop-*-capacity-scheduler.jar from
-HADOOP_HOME/build/contrib/capacity-scheduler to HADOOP_HOME/lib. Alternatively
-you can modify HADOOP_CLASSPATH to include this jar, in conf/hadoop-env.sh.
-
-You will also need to set the following property in the Hadoop config file
-(conf/hadoop-site.xml) to have Hadoop use the capacity scheduler:
-
-<property>
- <name>mapred.jobtracker.taskScheduler</name>
- <value>org.apache.hadoop.mapred.CapacityTaskScheduler</value>
-</property>
-
---------------------------------------------------------------------------------
-
-CONFIGURATION:
-
-The following properties can be set in hadoop-site.xml to configure the
-scheduler:
-
-mapred.capacity-scheduler.reclaimCapacity.interval:
- The capacity scheduler checks, every 'interval' seconds, whether any
- capacity needs to be reclaimed. The default value is 5 seconds.
-
-The scheduling information for queues is maintained in a configuration file
-called 'capacity-scheduler.xml'. Note that the queue names are set in
-hadoop-site.xml. capacity-scheduler.xml sets the scheduling properties
-for each queue. See that file for configuration details, but the following
-are the configuration options for each queue:
-
-mapred.capacity-scheduler.queue.<queue-name>.guaranteed-capacity
- Percentage of the number of slots in the cluster that are
- guaranteed to be available for jobs in this queue.
- The sum of guaranteed capacities for all queues should be less than or
- equal 100.
-
-mapred.capacity-scheduler.queue.<queue-name>.reclaim-time-limit
- The amount of time, in seconds, before which resources distributed to other
- queues will be reclaimed.
-
-mapred.capacity-scheduler.queue.<queue-name>.supports-priority
- If true, priorities of jobs will be taken into account in scheduling
- decisions.
-
-mapred.capacity-scheduler.queue.<queue-name>.minimum-user-limit-percent
- Each queue enforces a limit on the percentage of resources
- allocated to a user at any given time, if there is competition for them.
- This user limit can vary between a minimum and maximum value. The former
- depends on the number of users who have submitted jobs, and the latter is
- set to this property value. For example, suppose the value of this
- property is 25. If two users have submitted jobs to a queue, no single
- user can use more than 50% of the queue resources. If a third user submits
- a job, no single user can use more than 33% of the queue resources. With 4
- or more users, no user can use more than 25% of the queue's resources. A
- value of 100 implies no user limits are imposed.
-
-
---------------------------------------------------------------------------------
-
-IMPLEMENTATION:
-
-When a TaskTracker is free, the capacity scheduler does the following (note
-that many of these steps can be, and will be, enhanced over time to provide
-better algorithms):
-1. Decide whether to giev it a Map or Reduce task, depending on how many tasks
-the TT is already running of that type, with respect to the maximum taks it
-can run.
-2. The scheduler then picks a queue. Queues that need to reclaim capacity
-sooner, come before queues that don't. For queues that don't, they're ordered
-by a ratio of (# of running tasks)/Guaranteed capacity, which indicates how
-much 'free space' the queue has, or how much it is over capacity.
-3. A job is picked in the queue based on its state (running jobs are picked
-first), its priority (if the queue supports priorities) or its submission
-time, and whether the job's user is under or over limit.
-4. A task is picked from the job in the same way it always has.
-
-Periodically, a thread checks each queue to see if it needs to reclaim any
-capacity. Queues that are running below capacity and that have tasks waiting,
-need to reclaim capacity within a certain perdiod of time. If a queue hasn't
-received enough tasks in a certain amount of time, tasks will be killed from
-queues that are running over capacity.
-
---------------------------------------------------------------------------------
+The functionality of this scheduler is described in the Forrest documentation
+under src\docs\src\documentation\content\xdocs\capacity_scheduler.xml.
Added: hadoop/core/trunk/src/docs/src/documentation/content/xdocs/capacity_scheduler.xml
URL: http://svn.apache.org/viewvc/hadoop/core/trunk/src/docs/src/documentation/content/xdocs/capacity_scheduler.xml?rev=706338&view=auto
==============================================================================
--- hadoop/core/trunk/src/docs/src/documentation/content/xdocs/capacity_scheduler.xml (added)
+++ hadoop/core/trunk/src/docs/src/documentation/content/xdocs/capacity_scheduler.xml Mon Oct 20 09:56:17 2008
@@ -0,0 +1,256 @@
+<?xml version="1.0"?>
+<!--
+ Copyright 2002-2004 The Apache Software Foundation
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<!DOCTYPE document PUBLIC "-//APACHE//DTD Documentation V2.0//EN" "http://forrest.apache.org/dtd/document-v20.dtd">
+
+<document>
+
+ <header>
+ <title>Capacity Scheduler</title>
+ </header>
+
+ <body>
+
+ <section>
+ <title>Purpose</title>
+
+ <p>This document describes the Capacity Scheduler, a pluggable Map/Reduce scheduler for Hadoop which provides a way to share large clusters.</p>
+ </section>
+
+ <section>
+ <title>Features</title>
+
+ <p>The Capacity Scheduler supports the following features:</p>
+ <ul>
+ <li>
+ Support for multiple queues, where a job is submitted to a queue.
+ </li>
+ <li>
+ Queues are guaranteed a fraction of the capacity of the grid (their
+ 'guaranteed capacity') in the sense that a certain capacity of
+ resources will be at their disposal. All jobs submitted to the a
+ queue will have access to the capacity guaranteed to the queue.
+ </li>
+ <li>
+ Free resources can be allocated to any queue beyond its guaranteed
+ capacity. These excess allocated resources can be reclaimed and made
+ available to another queue in order to meet its capacity guarantee.
+ </li>
+ <li>
+ The scheduler guarantees that excess resources taken from a queue
+ will be restored to it within N minutes of its need for them.
+ </li>
+ <li>
+ Queues optionally support job priorities (disabled by default).
+ </li>
+ <li>
+ Within a queue, jobs with higher priority will have access to the
+ queue's resources before jobs with lower priority. However, once a
+ job is running, it will not be preempted for a higher priority job.
+ </li>
+ <li>
+ In order to prevent one or more users from monopolizing its
+ resources, each queue enforces a limit on the percentage of
+ resources allocated to a user at any given time, if there is
+ competition for them.
+ </li>
+ <li>
+ Support for memory-intensive jobs, wherein a job can optionally
+ specify higher memory-requirements than the default, and the tasks
+ of the job will only be run on TaskTrackers that have enough memory
+ to spare.
+ </li>
+ </ul>
+ </section>
+
+ <section>
+ <title>Picking a task to run</title>
+
+ <p>Note that many of these steps can be, and will be, enhanced over time
+ to provide better algorithms.</p>
+
+ <p>Whenever a TaskTracker is free, the Capacity Scheduler first picks a
+ queue that needs to reclaim any resources the earliest (this is a queue
+ whose resources were temporarily being used by some other queue and now
+ needs access to those resources). If no such queue is found, it then picks
+ a queue which has most free space (whose ratio of # of running slots to
+ guaranteed capacity is the lowest).</p>
+
+ <p>Once a queue is selected, the scheduler picks a job in the queue. Jobs
+ are sorted based on when they're submitted and their priorities (if the
+ queue supports priorities). Jobs are considered in order, and a job is
+ selected if its user is within the user-quota for the queue, i.e., the
+ user is not already using queue resources above his/her limit. The
+ scheduler also makes sure that there is enough free memory in the
+ TaskTracker to tun the job's task, in case the job has special memory
+ requirements.</p>
+
+ <p>Once a job is selected, the scheduler picks a task to run. This logic
+ to pick a task remains unchanged from earlier versions.</p>
+
+ </section>
+
+ <section>
+ <title>Reclaiming capacity</title>
+
+ <p>Periodically, the scheduler determines:</p>
+ <ul>
+ <li>
+ if a queue needs to reclaim capacity. This happens when a queue has
+ at least one task pending and part of its guaranteed capacity is
+ being used by some other queue. If this happens, the scheduler notes
+ the amount of resources it needs to reclaim for this queue within a
+ specified period of time (the reclaim time).
+ </li>
+ <li>
+ if a queue has not received all the resources it needed to reclaim,
+ and its reclaim time is about to expire. In this case, the scheduler
+ needs to kill tasks from queues running over capacity. This it does
+ by killing the tasks that started the latest.
+ </li>
+ </ul>
+
+ </section>
+
+ <section>
+ <title>Installation</title>
+
+ <p>The capacity scheduler is available as a JAR file in the Hadoop
+ tarball under the <em>contrib/capacity-scheduler</em> directory. The name of
+ the JAR file would be on the lines of hadoop-*-capacity-scheduler.jar.</p>
+ <p>You can also build the scheduler from source by executing
+ <em>ant package</em>, in which case it would be available under
+ <em>build/contrib/capacity-scheduler</em>.</p>
+ <p>To run the capacity scheduler in your Hadoop installation, you need
+ to put it on the <em>CLASSPATH</em>. The easiest way is to copy the
+ <code>hadoop-*-capacity-scheduler.jar</code> from
+ to <code>HADOOP_HOME/lib</code>. Alternatively, you can modify
+ <em>HADOOP_CLASSPATH</em> to include this jar, in
+ <code>conf/hadoop-env.sh</code>.</p>
+ </section>
+
+ <section>
+ <title>Configuration</title>
+
+ <section>
+ <title>Using the capacity scheduler</title>
+ <p>
+ To make the Hadoop framework use the capacity scheduler, set up
+ the following property in the site configuration:</p>
+ <table>
+ <tr>
+ <td>Property</td>
+ <td>Value</td>
+ </tr>
+ <tr>
+ <td>mapred.jobtracker.taskScheduler</td>
+ <td>org.apache.hadoop.mapred.CapacityTaskScheduler</td>
+ </tr>
+ </table>
+ </section>
+
+ <section>
+ <title>Setting up queues</title>
+ <p>
+ You can define multiple queues to which users can submit jobs with
+ the capacity scheduler. To define multiple queues, you should edit
+ the site configuration for Hadoop and modify the
+ <em>mapred.queue.names</em> property.
+ </p>
+ <p>
+ You can also configure ACLs for controlling which users or groups
+ have access to the queues.
+ </p>
+ <p>
+ For more details, refer to
+ <a href="cluster_setup.html#Configuring+the+Hadoop+Daemons">Cluster
+ Setup</a> documentation.
+ </p>
+ </section>
+
+ <section>
+ <title>Configuring properties for queues</title>
+
+ <p>The capacity scheduler can be configured with several properties
+ for each queue that control the behavior of the scheduler. This
+ configuration is in the <em>conf/capacity-scheduler.xml</em>. By
+ default, the configuration is set up for one queue, named
+ <em>default</em>.</p>
+ <p>To specify a property for a queue that is defined in the site
+ configuration, you should use the property name as
+ <em>mapred.capacity-scheduler.queue.<queue-name>.<property-name></em>.
+ </p>
+ <p>For example, to define the property <em>guaranteed-capacity</em>
+ for queue named <em>research</em>, you should specify the property
+ name as
+ <em>mapred.capacity-scheduler.queue.research.guaranteed-capacity</em>.
+ </p>
+
+ <p>The properties defined for queues and their descriptions are
+ listed in the table below:</p>
+
+ <table>
+ <tr><th>Name</th><th>Description</th></tr>
+ <tr><td>mapred.capacity-scheduler.queue.<queue-name>.guaranteed-capacity</td>
+ <td>Percentage of the number of slots in the cluster that are
+ guaranteed to be available for jobs in this queue.
+ The sum of guaranteed capacities for all queues should be less
+ than or equal 100.</td>
+ </tr>
+ <tr><td>mapred.capacity-scheduler.queue.<queue-name>.reclaim-time-limit</td>
+ <td>The amount of time, in seconds, before which resources
+ distributed to other queues will be reclaimed.</td>
+ </tr>
+ <tr><td>mapred.capacity-scheduler.queue.<queue-name>.supports-priority</td>
+ <td>If true, priorities of jobs will be taken into account in scheduling
+ decisions.</td>
+ </tr>
+ <tr><td>mapred.capacity-scheduler.queue.<queue-name>.minimum-user-limit-percent</td>
+ <td>Each queue enforces a limit on the percentage of resources
+ allocated to a user at any given time, if there is competition
+ for them. This user limit can vary between a minimum and maximum
+ value. The former depends on the number of users who have submitted
+ jobs, and the latter is set to this property value. For example,
+ suppose the value of this property is 25. If two users have
+ submitted jobs to a queue, no single user can use more than 50%
+ of the queue resources. If a third user submits a job, no single
+ user can use more than 33% of the queue resources. With 4 or more
+ users, no user can use more than 25% of the queue's resources. A
+ value of 100 implies no user limits are imposed.</td>
+ </tr>
+ </table>
+ </section>
+
+ <section>
+ <title>Reviewing the configuration of the capacity scheduler</title>
+ <p>
+ Once the installation and configuration is completed, you can review
+ it after starting the Map/Reduce cluster from the admin UI.
+ </p>
+ <ul>
+ <li>Start the Map/Reduce cluster as usual.</li>
+ <li>Open the JobTracker web UI.</li>
+ <li>The queues you have configured should be listed under the <em>Scheduling
+ Information</em> section of the page.</li>
+ <li>The properties for the queues should be visible in the <em>Scheduling
+ Information</em> column against each queue.</li>
+ </ul>
+ </section>
+ </section>
+ </body>
+
+</document>
Modified: hadoop/core/trunk/src/docs/src/documentation/content/xdocs/cluster_setup.xml
URL: http://svn.apache.org/viewvc/hadoop/core/trunk/src/docs/src/documentation/content/xdocs/cluster_setup.xml?rev=706338&r1=706337&r2=706338&view=diff
==============================================================================
--- hadoop/core/trunk/src/docs/src/documentation/content/xdocs/cluster_setup.xml (original)
+++ hadoop/core/trunk/src/docs/src/documentation/content/xdocs/cluster_setup.xml Mon Oct 20 09:56:17 2008
@@ -244,6 +244,65 @@
TaskTrackers.
</td>
</tr>
+ <tr>
+ <td>mapred.queue.names</td>
+ <td>Comma separated list of queues to which jobs can be submitted.</td>
+ <td>
+ The Map/Reduce system always supports atleast one queue
+ with the name as <em>default</em>. Hence, this parameter's
+ value should always contain the string <em>default</em>.
+ Some job schedulers supported in Hadoop, like the
+ <a href="capacity_scheduler.html">Capacity
+ Scheduler</a>, support multiple queues. If such a scheduler is
+ being used, the list of configured queue names must be
+ specified here. Once queues are defined, users can submit
+ jobs to a queue using the property name
+ <em>mapred.job.queue.name</em> in the job configuration.
+ There could be a separate
+ configuration file for configuring properties of these
+ queues that is managed by the scheduler.
+ Refer to the documentation of the scheduler for information on
+ the same.
+ </td>
+ </tr>
+ <tr>
+ <td>mapred.acls.enabled</td>
+ <td>Specifies whether ACLs are supported for controlling job
+ submission and administration</td>
+ <td>
+ If <em>true</em>, ACLs would be checked while submitting
+ and administering jobs. ACLs can be specified using the
+ configuration parameters of the form
+ <em>mapred.queue.queue-name.acl-name</em>, defined below.
+ </td>
+ </tr>
+ <tr>
+ <td>mapred.queue.<em>queue-name</em>.acl-submit-job</td>
+ <td>List of users and groups that can submit jobs to the
+ specified <em>queue-name</em>.</td>
+ <td>
+ The list of users and groups are both comma separated
+ list of names. The two lists are separated by a blank.
+ Example: <em>user1,user2 group1,group2</em>.
+ If you wish to define only a list of groups, provide
+ a blank at the beginning of the value.
+ </td>
+ </tr>
+ <tr>
+ <td>mapred.queue.<em>queue-name</em>.acl-administer-job</td>
+ <td>List of users and groups that can change the priority
+ or kill jobs that have been submitted to the
+ specified <em>queue-name</em>.</td>
+ <td>
+ The list of users and groups are both comma separated
+ list of names. The two lists are separated by a blank.
+ Example: <em>user1,user2 group1,group2</em>.
+ If you wish to define only a list of groups, provide
+ a blank at the beginning of the value. Note that an
+ owner of a job can always change the priority or kill
+ his/her own job, irrespective of the ACLs.
+ </td>
+ </tr>
</table>
<p>Typically all the above parameters are marked as
Modified: hadoop/core/trunk/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml
URL: http://svn.apache.org/viewvc/hadoop/core/trunk/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml?rev=706338&r1=706337&r2=706338&view=diff
==============================================================================
--- hadoop/core/trunk/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml (original)
+++ hadoop/core/trunk/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml Mon Oct 20 09:56:17 2008
@@ -1703,6 +1703,23 @@
<title>Other Useful Features</title>
<section>
+ <title>Submitting Jobs to a Queue</title>
+ <p>Some job schedulers supported in Hadoop, like the
+ <a href="capacity_scheduler.html">Capacity
+ Scheduler</a>, support multiple queues. If such a scheduler is
+ being used, users can submit jobs to one of the queues
+ administrators would have defined in the
+ <em>mapred.queue.names</em> property of the Hadoop site
+ configuration. The queue name can be specified through the
+ <em>mapred.job.queue.name</em> property, or through the
+ <a href="ext:api/org/apache/hadoop/mapred/jobconf/setqueuename">setQueueName(String)</a>
+ API. Note that administrators may choose to define ACLs
+ that control which queues a job can be submitted to by a
+ given user. In that case, if the job is not submitted
+ to one of the queues where the user has access,
+ the job would be rejected.</p>
+ </section>
+ <section>
<title>Counters</title>
<p><code>Counters</code> represent global counters, defined either by
Modified: hadoop/core/trunk/src/docs/src/documentation/content/xdocs/site.xml
URL: http://svn.apache.org/viewvc/hadoop/core/trunk/src/docs/src/documentation/content/xdocs/site.xml?rev=706338&r1=706337&r2=706338&view=diff
==============================================================================
--- hadoop/core/trunk/src/docs/src/documentation/content/xdocs/site.xml (original)
+++ hadoop/core/trunk/src/docs/src/documentation/content/xdocs/site.xml Mon Oct 20 09:56:17 2008
@@ -52,6 +52,7 @@
<hod-admin-guide href="hod_admin_guide.html"/>
<hod-config-guide href="hod_config_guide.html"/>
</hod>
+ <capacity_scheduler label="Capacity Scheduler" href="capacity_scheduler.html"/>
<api label="API Docs" href="ext:api/index" />
<jdiff label="API Changes" href="ext:jdiff/changes" />
<wiki label="Wiki" href="ext:wiki" />
@@ -182,6 +183,7 @@
<setprofiletaskrange href="#setProfileTaskRange(boolean,%20java.lang.String)" />
<setprofileparams href="#setProfileParams(java.lang.String)" />
<setnumtaskstoexecuteperjvm href="#setNumTasksToExecutePerJvm(int)" />
+ <setqueuename href="#setQueueName(java.lang.String)" />
<getjoblocaldir href="#getJobLocalDir()" />
<getjar href="#getJar()" />
</jobconf>