You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by jh...@apache.org on 2019/08/05 17:04:49 UTC
[hadoop] branch branch-2 updated (9aae72a -> 19af228)
This is an automated email from the ASF dual-hosted git repository.
jhung pushed a change to branch branch-2
in repository https://gitbox.apache.org/repos/asf/hadoop.git.
from 9aae72a YARN-8045. Reduce log output from container status calls. Contributed by Craig Condit
new 75fc058 YARN-8896. Limit the maximum number of container assignments per heartbeat. (Zhankun Tang via wangda)
new 4e5ba31 YARN-8915. Update the doc about the default value of maximum-container-assignments for capacity scheduler. Contributed by Zhankun Tang.
new 19af228 YARN-8944. TestContainerAllocation.testUserLimitAllocationMultipleContainers failure after YARN-8896. Contributed by Wilfred Spiegelenburg.
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../scheduler/capacity/CapacitySchedulerConfiguration.java | 5 ++++-
.../resourcemanager/scheduler/capacity/TestContainerAllocation.java | 4 ++++
.../hadoop-yarn-site/src/site/markdown/CapacityScheduler.md | 2 +-
3 files changed, 9 insertions(+), 2 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[hadoop] 03/03: YARN-8944.
TestContainerAllocation.testUserLimitAllocationMultipleContainers failure
after YARN-8896. Contributed by Wilfred Spiegelenburg.
Posted by jh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jhung pushed a commit to branch branch-2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 19af2289571b68d1d474462cd06fb87450732e16
Author: Weiwei Yang <ww...@apache.org>
AuthorDate: Mon Oct 29 11:53:10 2018 +0800
YARN-8944. TestContainerAllocation.testUserLimitAllocationMultipleContainers failure after YARN-8896. Contributed by Wilfred Spiegelenburg.
(cherry picked from commit 1d90a0dd23c5d1bd52d04c303506806d0dc61cd2)
(cherry picked from commit 70efe253f369f579f05d833d45e6a1bae3e41dfe)
(cherry picked from commit cd8e9fa80b2903935ca9c337631e60161a6053b3)
---
.../resourcemanager/scheduler/capacity/TestContainerAllocation.java | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerAllocation.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerAllocation.java
index c1fcda8..5164738 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerAllocation.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerAllocation.java
@@ -67,6 +67,7 @@ import org.junit.Before;
import org.junit.Test;
import static org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.MAXIMUM_ALLOCATION_MB;
+import static org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.MAX_ASSIGN_PER_HEARTBEAT;
public class TestContainerAllocation {
@@ -902,6 +903,9 @@ public class TestContainerAllocation {
CapacitySchedulerConfiguration newConf =
(CapacitySchedulerConfiguration) TestUtils
.getConfigurationWithMultipleQueues(conf);
+ // make sure an unlimited number of containers can be assigned,
+ // overriding the default of 100 after YARN-8896
+ newConf.set(MAX_ASSIGN_PER_HEARTBEAT, "-1");
newConf.setUserLimit("root.c", 50);
MockRM rm1 = new MockRM(newConf);
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[hadoop] 02/03: YARN-8915. Update the doc about the default value
of maximum-container-assignments for capacity scheduler. Contributed by
Zhankun Tang.
Posted by jh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jhung pushed a commit to branch branch-2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 4e5ba315c6e09a1cb73db40d8a580cfa47743d44
Author: Weiwei Yang <ww...@apache.org>
AuthorDate: Mon Oct 29 12:04:32 2018 +0800
YARN-8915. Update the doc about the default value of maximum-container-assignments for capacity scheduler. Contributed by Zhankun Tang.
(cherry picked from commit 78ea897b6701c82799a6231103ca7ee1f8b4efde)
(cherry picked from commit 7dd8eafe337a33689ff6f2f947ad152e06dbc55f)
(cherry picked from commit d095325234fe8fa8d2513c2f03c03347ef5f3606)
---
.../hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
index 05be21e..fd7140e 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
@@ -290,7 +290,7 @@ The `ReservationSystem` is integrated with the `CapacityScheduler` queue hierach
| Property | Description |
|:---- |:---- |
| `yarn.scheduler.capacity.per-node-heartbeat.multiple-assignments-enabled` | Whether to allow multiple container assignments in one NodeManager heartbeat. Defaults to true. |
-| `yarn.scheduler.capacity.per-node-heartbeat.maximum-container-assignments` | If `multiple-assignments-enabled` is true, the maximum amount of containers that can be assigned in one NodeManager heartbeat. Defaults to -1, which sets no limit. |
+| `yarn.scheduler.capacity.per-node-heartbeat.maximum-container-assignments` | If `multiple-assignments-enabled` is true, the maximum amount of containers that can be assigned in one NodeManager heartbeat. Default value is 100, which limits the maximum number of container assignments per heartbeat to 100. Set this value to -1 will disable this limit. |
| `yarn.scheduler.capacity.per-node-heartbeat.maximum-offswitch-assignments` | If `multiple-assignments-enabled` is true, the maximum amount of off-switch containers that can be assigned in one NodeManager heartbeat. Defaults to 1, which represents only one off-switch allocation allowed in one heartbeat. |
###Reviewing the configuration of the CapacityScheduler
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[hadoop] 01/03: YARN-8896. Limit the maximum number of container
assignments per heartbeat. (Zhankun Tang via wangda)
Posted by jh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jhung pushed a commit to branch branch-2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 75fc058290aee7361d2a75b6dbbb9505a7ec3c0a
Author: Wangda Tan <wa...@apache.org>
AuthorDate: Thu Oct 18 10:58:21 2018 -0700
YARN-8896. Limit the maximum number of container assignments per heartbeat. (Zhankun Tang via wangda)
Change-Id: I6e72f8362bd7f5c2a844cb9e3c4732492314e9f1
(cherry picked from commit 780be14f07df2a3ed6273b96ae857c278fd72718)
(cherry picked from commit 46baafedf13446f5c1b973d667f437bf7158d3c2)
(cherry picked from commit bf4da6aacee62be906e71eca6e33821f5ee79a2d)
---
.../scheduler/capacity/CapacitySchedulerConfiguration.java | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
index 1e22e0b..358a976 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
@@ -314,8 +314,11 @@ public class CapacitySchedulerConfiguration extends ReservationSchedulerConfigur
public static final String MAX_ASSIGN_PER_HEARTBEAT = PREFIX
+ "per-node-heartbeat.maximum-container-assignments";
+ /**
+ * Avoid potential risk that greedy assign multiple may involve
+ * */
@Private
- public static final int DEFAULT_MAX_ASSIGN_PER_HEARTBEAT = -1;
+ public static final int DEFAULT_MAX_ASSIGN_PER_HEARTBEAT = 100;
AppPriorityACLConfigurationParser priorityACLConfig = new AppPriorityACLConfigurationParser();
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org