You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Robert Chansler (JIRA)" <ji...@apache.org> on 2009/10/08 06:17:31 UTC
[jira] Updated: (HADOOP-4768) Dynamic Priority Scheduler that
allows queue shares to be controlled dynamically by a currency
[ https://issues.apache.org/jira/browse/HADOOP-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler updated HADOOP-4768:
------------------------------------
Release Note: New contribution Dynamic Scheduler implements dynamic priorities with a currency model. Usage instructions are in the Jira item. (was: This package implements dynamic priority scheduling for MapReduce jobs.
Overview
--------
The purpose of this scheduler is to allow users to increase and decrease
their queue priorities continuosly to meet the requirements of their
current workloads. The scheduler is aware of the current demand and makes
it more expensive to boost the priority under peak usage times. Thus
users who move their workload to low usage times are rewarded with
discounts. Priorities can only be boosted within a limited quota.
All users are given a quota or a budget which is deducted periodically
in configurable accounting intervals. How much of the budget is
deducted is determined by a per-user spending rate, which may
be modified at any time directly by the user. The cluster slots
share allocated to a particular user is computed as that users
spending rate over the sum of all spending rates in the same accounting
period.
Configuration
-------------
This scheduler comprises two components, an accounting or resource allocation part that
manages and bills for queue shares, and a scheduler that
enforces the queue shares in the form of map and reduce slots of running jobs.
Hadoop Configuration (e.g. hadoop-site.xml):
mapred.jobtracker.taskScheduler
This needs to be set to
org.apache.hadoop.mapred.DynamicPriorityScheduler
to use the dynamic scheduler.
Scheduler Configuration:
mapred.dynamic-scheduler.scheduler
The Java path of the MapReduce scheduler that should
enforce the allocated shares.
Has been tested with (which is the default):
org.apache.hadoop.mapred.PriorityScheduler
mapred.priority-scheduler.acl-file
Full path of ACL with syntax:
<user> <role> <secret key>
separated by line feeds
mapred.dynamic-scheduler.budget-file
The full OS path of the file from which the
budgets are read and stored. The syntax of this file is:
<queue name> <budget> <spending rate>
separated by newlines where budget can be specified
as a Java float. The file should not be edited
directly, if the server is running, but through the
servlet API to ensure proper synchronization.
mapred.dynamic-scheduler.alloc-interval
Allocation interval, when the scheduler rereads the
spending rates and recalculates the cluster shares.
Specified as seconds between allocations.
Default is 20 seconds.
)
Editorial pass over all release notes prior to publication of 0.21.
Usage instructions:
Overview
--------
The purpose of this scheduler is to allow users to increase and decrease
their queue priorities continuosly to meet the requirements of their
current workloads. The scheduler is aware of the current demand and makes
it more expensive to boost the priority under peak usage times. Thus
users who move their workload to low usage times are rewarded with
discounts. Priorities can only be boosted within a limited quota.
All users are given a quota or a budget which is deducted periodically
in configurable accounting intervals. How much of the budget is
deducted is determined by a per-user spending rate, which may
be modified at any time directly by the user. The cluster slots
share allocated to a particular user is computed as that users
spending rate over the sum of all spending rates in the same accounting
period.
Configuration
-------------
This scheduler comprises two components, an accounting or resource allocation part that
manages and bills for queue shares, and a scheduler that
enforces the queue shares in the form of map and reduce slots of running jobs.
Hadoop Configuration (e.g. hadoop-site.xml):
mapred.jobtracker.taskScheduler
This needs to be set to
org.apache.hadoop.mapred.Editorial pass over all release notes prior to publication of 0.21.
to use the dynamic scheduler.
Scheduler Configuration:
mapred.dynamic-scheduler.scheduler
The Java path of the MapReduce scheduler that should
enforce the allocated shares.
Has been tested with (which is the default):
org.apache.hadoop.mapred.PriorityScheduler
mapred.priority-scheduler.acl-file
Full path of ACL with syntax:
<user> <role> <secret key>
separated by line feeds
mapred.dynamic-scheduler.budget-file
The full OS path of the file from which the
budgets are read and stored. The syntax of this file is:
<queue name> <budget> <spending rate>
separated by newlines where budget can be specified
as a Java float. The file should not be edited
directly, if the server is running, but through the
servlet API to ensure proper synchronization.
mapred.dynamic-scheduler.alloc-interval
Allocation interval, when the scheduler rereads the
spending rates and recalculates the cluster shares.
Specified as seconds between allocations.
Default is 20 seconds.
> Dynamic Priority Scheduler that allows queue shares to be controlled dynamically by a currency
> ----------------------------------------------------------------------------------------------
>
> Key: HADOOP-4768
> URL: https://issues.apache.org/jira/browse/HADOOP-4768
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.21.0
> Reporter: Thomas Sandholm
> Assignee: Thomas Sandholm
> Fix For: 0.21.0
>
> Attachments: HADOOP-4768-2.patch, HADOOP-4768-3.patch, HADOOP-4768-4.patch, HADOOP-4768-5.patch, HADOOP-4768-6.patch, HADOOP-4768-capacity-scheduler.patch, HADOOP-4768-dynamic-scheduler.patch, HADOOP-4768-fairshare.patch, HADOOP-4768.patch
>
>
> Dynamic (economic) priority scheduler based on work presented at the Hadoop User Group meeting in Santa Clara in September and the HadoopCamp in New Orleans in November 2008.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.