You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by su...@apache.org on 2017/10/25 22:34:13 UTC

[1/2] hadoop git commit: YARN-4687. Document Reservation ACLs (Contributed by Sean Po)

Repository: hadoop
Updated Branches:
  refs/heads/branch-2 fa86fdc3c -> 74aee8673


YARN-4687. Document Reservation ACLs (Contributed by Sean Po)

(cherry picked from commit 2e1d0ff4e901b8313c8d71869735b94ed8bc40a0)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/774a5753
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/774a5753
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/774a5753

Branch: refs/heads/branch-2
Commit: 774a57536393f37100a6399f1fb206c649bc28bf
Parents: fa86fdc
Author: Arun Suresh <as...@apache.org>
Authored: Thu Mar 24 10:40:51 2016 -0700
Committer: Subru Krishnan <su...@apache.org>
Committed: Wed Oct 25 15:29:32 2017 -0700

----------------------------------------------------------------------
 .../hadoop-yarn-site/src/site/markdown/FairScheduler.md        | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/hadoop/blob/774a5753/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md
index db3e1da..941bfa2 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md
@@ -189,12 +189,16 @@ The allocation file must be in XML format. The format contains five types of ele
 
 ###Queue Access Control Lists
 
-Queue Access Control Lists (ACLs) allow administrators to control who may take actions on particular queues. They are configured with the aclSubmitApps and aclAdministerApps properties, which can be set per queue. Currently the only supported administrative action is killing an application. Anybody who may administer a queue may also submit applications to it. These properties take values in a format like "user1,user2 group1,group2" or " group1,group2". An action on a queue will be permitted if its user or group is in the ACL of that queue or in the ACL of any of that queue's ancestors. So if queue2 is inside queue1, and user1 is in queue1's ACL, and user2 is in queue2's ACL, then both users may submit to queue2.
+Queue Access Control Lists (ACLs) allow administrators to control who may take actions on particular queues. They are configured with the aclSubmitApps and aclAdministerApps properties, which can be set per queue. Currently the only supported administrative action is killing an application. An administrator may also submit applications to it. These properties take values in a format like "user1,user2 group1,group2" or " group1,group2". Actions on a queue are permitted if the user/group is a member of the queue ACL or a member of the queue ACL of any of that queue's ancestors. So if queue2 is inside queue1, and user1 is in queue1's ACL, and user2 is in queue2's ACL, then both users may submit to queue2.
 
 **Note:** The delimiter is a space character. To specify only ACL groups, begin the value with a space character.
 
 The root queue's ACLs are "\*" by default which, because ACLs are passed down, means that everybody may submit to and kill applications from every queue. To start restricting access, change the root queue's ACLs to something other than "\*".
 
+###Reservation Access Control Lists
+
+Reservation Access Control Lists (ACLs) allow administrators to control who may take reservation actions on particular queues. They are configured with the aclAdministerReservations, aclListReservations, and the aclSubmitReservations properties, which can be set per queue. Currently the supported administrative actions are updating and deleting reservations. An administrator may also submit and list *all* reservations on the queue. These properties take values in a format like "user1,user2 group1,group2" or " group1,group2". Actions on a queue are permitted if the user/group is a member of the reservation ACL. Note that any user can update, delete, or list their own reservations. If reservation ACLs are enabled but not defined, everyone will have access.
+
 ##Administration
 
 The fair scheduler provides support for administration at runtime through a few mechanisms:


---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org


[2/2] hadoop git commit: YARN-4827. Document configuration of ReservationSystem for FairScheduler. (Yufei Gu via Subru).

Posted by su...@apache.org.
YARN-4827. Document configuration of ReservationSystem for FairScheduler. (Yufei Gu via Subru).

(cherry picked from commit 3fae675383489129b3ca3c66683a1215d0c6edf0)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/74aee867
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/74aee867
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/74aee867

Branch: refs/heads/branch-2
Commit: 74aee8673f047414a38fc4f5b0570a90465550ec
Parents: 774a575
Author: Subru Krishnan <su...@apache.org>
Authored: Wed Oct 25 15:07:50 2017 -0700
Committer: Subru Krishnan <su...@apache.org>
Committed: Wed Oct 25 15:32:03 2017 -0700

----------------------------------------------------------------------
 .../reservation/CapacityOverTimePolicy.java     |  3 ---
 .../src/site/markdown/FairScheduler.md          | 24 ++++++++++++++++++++
 .../src/site/markdown/ReservationSystem.md      |  2 +-
 3 files changed, 25 insertions(+), 4 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/hadoop/blob/74aee867/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/CapacityOverTimePolicy.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/CapacityOverTimePolicy.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/CapacityOverTimePolicy.java
index acd5774..1f3f9bc 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/CapacityOverTimePolicy.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/CapacityOverTimePolicy.java
@@ -56,9 +56,6 @@ public class CapacityOverTimePolicy extends NoOverCommitPolicy {
   private float maxInst;
   private float maxAvg;
 
-  // For now this is CapacityScheduler specific, but given a hierarchy in the
-  // configuration structure of the schedulers (e.g., SchedulerConfiguration)
-  // it should be easy to remove this limitation
   @Override
   public void init(String reservationQueuePath,
       ReservationSchedulerConfiguration conf) {

http://git-wip-us.apache.org/repos/asf/hadoop/blob/74aee867/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md
index 941bfa2..e5cb31b 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md
@@ -107,6 +107,8 @@ The allocation file must be in XML format. The format contains five types of ele
 
     * **allowPreemptionFrom**: determines whether the scheduler is allowed to preempt resources from the queue. The default is true. If a queue has this property set to false, this property will apply recursively to all child queues.
 
+    * **reservation**: indicates to the `ReservationSystem` that the queue's resources is available for users to reserve. This only applies for leaf queues. A leaf queue is not reservable if this property isn't configured.
+
 * **User elements**: which represent settings governing the behavior of individual users. They can contain a single property: maxRunningApps, a limit on the number of running apps for a particular user.
 
 * **A userMaxAppsDefault element**: which sets the default running app limit for any users whose limit is not otherwise specified.
@@ -125,6 +127,12 @@ The allocation file must be in XML format. The format contains five types of ele
 
 * **A defaultQueueSchedulingPolicy element**: which sets the default scheduling policy for queues; overridden by the schedulingPolicy element in each queue if specified. Defaults to "fair".
 
+* **A reservation-agent element**: which sets the class name of the implementation of the `ReservationAgent`, which attempts to place the user's reservation request in the `Plan`. The default value is `org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.AlignedPlannerWithGreedy`.
+
+* **A reservation-policy element**: which sets the class name of the implementation of the `SharingPolicy`, which validates if the new reservation doesn't violate any invariants. The default value is `org.apache.hadoop.yarn.server.resourcemanager.reservation.CapacityOverTimePolicy`.
+
+* **A reservation-planner element**: which sets the class name of the implementation of the `Planner`, which is invoked if the `Plan` capacity fall below (due to scheduled maintenance or node failuers) the user reserved resources. The default value is `org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.SimpleCapacityReplanner`, which scans the `Plan` and greedily removes reservations in reversed order of acceptance (LIFO) till the reserved resources are within the `Plan` capacity.
+
 * **A queuePlacementPolicy element**: which contains a list of rule elements that tell the scheduler how to place incoming apps into queues. Rules are applied in the order that they are listed. Rules may take arguments. All rules accept the "create" argument, which indicates whether the rule can create a new queue. "Create" defaults to true; if set to false and the rule would place the app in a queue that is not configured in the allocations file, we continue on to the next rule. The last rule must be one that can never issue a continue. Valid rules are:
 
     * **specified**: the app is placed into the queue it requested. If the app requested no queue, i.e. it specified "default", we continue. If the app requested a queue name starting or ending with period, i.e. names like ".q1" or "q1." will be rejected.
@@ -157,6 +165,9 @@ The allocation file must be in XML format. The format contains five types of ele
       <aclSubmitApps>charlie</aclSubmitApps>
       <minResources>5000 mb,0vcores</minResources>
     </queue>
+    <queue name="sample_reservable_queue">
+      <reservation></reservation>
+    </queue>
   </queue>
 
   <queueMaxAMShareDefault>0.5</queueMaxAMShareDefault>
@@ -199,6 +210,19 @@ The root queue's ACLs are "\*" by default which, because ACLs are passed down, m
 
 Reservation Access Control Lists (ACLs) allow administrators to control who may take reservation actions on particular queues. They are configured with the aclAdministerReservations, aclListReservations, and the aclSubmitReservations properties, which can be set per queue. Currently the supported administrative actions are updating and deleting reservations. An administrator may also submit and list *all* reservations on the queue. These properties take values in a format like "user1,user2 group1,group2" or " group1,group2". Actions on a queue are permitted if the user/group is a member of the reservation ACL. Note that any user can update, delete, or list their own reservations. If reservation ACLs are enabled but not defined, everyone will have access.
 
+### Configuring `ReservationSystem`
+
+The Fair Scheduler supports the **ReservationSystem** which allows users to reserve resources ahead of time. The application can request reserved resources at runtime by specifying the `reservationId` during submission. The following configuration parameters can be configured in yarn-site.xml for `ReservationSystem`.
+
+| Property | Description |
+|:---- |:---- |
+| `yarn.resourcemanager.reservation-system.enable` | *Mandatory* parameter: to enable the `ReservationSystem` in the **ResourceManager**. Boolean value expected. The default value is *false*, i.e. `ReservationSystem` is not enabled by default. |
+| `yarn.resourcemanager.reservation-system.class` | *Optional* parameter: the class name of the `ReservationSystem`. The default value is picked based on the configured Scheduler, i.e. if `FairScheduler` is configured, then it is `FairReservationSystem`. |
+| `yarn.resourcemanager.reservation-system.plan.follower` | *Optional* parameter: the class name of the `PlanFollower` that runs on a timer, and synchronizes the `FairScheduler` with the `Plan` and vice versa. The default value is picked based on the configured Scheduler, i.e. if `FairScheduler` is configured, then it is `FairSchedulerPlanFollower`. |
+| `yarn.resourcemanager.reservation-system.planfollower.time-step` | *Optional* parameter: the frequency in milliseconds of the `PlanFollower` timer. Long value expected. The default value is *1000*. |
+
+The `ReservationSystem` is integrated with the Fair Scheduler queue hierarchy and can be configured for and only for leaf queues. The detailed instructions are in section `Allocation file format`.
+
 ##Administration
 
 The fair scheduler provides support for administration at runtime through a few mechanisms:

http://git-wip-us.apache.org/repos/asf/hadoop/blob/74aee867/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ReservationSystem.md
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ReservationSystem.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ReservationSystem.md
index cd4ae54..ac3269d 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ReservationSystem.md
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ReservationSystem.md
@@ -60,4 +60,4 @@ With reference to the figure above, a typical reservation proceeds as follows:
 Configuring the Reservation System
 ----------------------------------
 
-Configuring the `ReservationSystem` is simple. Currently we have added support for *reservations* in both `CapacityScheduler` and `FairScheduler`. You can mark any **leaf queue** in the **capacity-scheduler.xml** or **fair-scheduler.xml** as available for "reservations" (see [CapacityScheduler](./CapacityScheduler.html#Configuring_ReservationSystem_with_CapacityScheduler) and the [FairScheduler](./FairScheduler.html) for details). Then the capacity/fair share within that queue can be used for making reservations. Jobs can still be submitted to the *reservable queue* without a reservation, in which case they will be run in best-effort mode in whatever capacity is left over by the jobs running within active reservations.
+Configuring the `ReservationSystem` is simple. Currently we have added support for *reservations* in both `CapacityScheduler` and `FairScheduler`. You can mark any **leaf queue** in the **capacity-scheduler.xml** or **fair-scheduler.xml** as available for "reservations" (see [CapacityScheduler](./CapacityScheduler.html#Configuring_ReservationSystem_with_CapacityScheduler) and the [FairScheduler](./FairScheduler.html#Configuring_ReservationSystem) for details). Then the capacity/fair share within that queue can be used for making reservations. Jobs can still be submitted to the *reservable queue* without a reservation, in which case they will be run in best-effort mode in whatever capacity is left over by the jobs running within active reservations.


---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org