You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by jh...@apache.org on 2019/08/05 17:02:55 UTC
[hadoop] branch branch-3.0 updated (01f76b4 -> 373e324)
This is an automated email from the ASF dual-hosted git repository.
jhung pushed a change to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/hadoop.git.
from 01f76b4 YARN-8045. Reduce log output from container status calls. Contributed by Craig Condit
new d18ec19 YARN-8896. Limit the maximum number of container assignments per heartbeat. (Zhankun Tang via wangda)
new ea345de YARN-6851. Capacity Scheduler: document configs for controlling # containers allowed to be allocated per node heartbeat.
new fd977a1 YARN-8915. Update the doc about the default value of maximum-container-assignments for capacity scheduler. Contributed by Zhankun Tang.
new 373e324 YARN-8944. TestContainerAllocation.testUserLimitAllocationMultipleContainers failure after YARN-8896. Contributed by Wilfred Spiegelenburg.
The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../scheduler/capacity/CapacitySchedulerConfiguration.java | 5 ++++-
.../scheduler/capacity/TestContainerAllocation.java | 4 ++++
.../hadoop-yarn-site/src/site/markdown/CapacityScheduler.md | 10 ++++++++++
3 files changed, 18 insertions(+), 1 deletion(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[hadoop] 01/04: YARN-8896. Limit the maximum number of container
assignments per heartbeat. (Zhankun Tang via wangda)
Posted by jh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jhung pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit d18ec19ed885dded487d97eaa25d43ab7540ecbc
Author: Wangda Tan <wa...@apache.org>
AuthorDate: Thu Oct 18 10:58:21 2018 -0700
YARN-8896. Limit the maximum number of container assignments per heartbeat. (Zhankun Tang via wangda)
Change-Id: I6e72f8362bd7f5c2a844cb9e3c4732492314e9f1
(cherry picked from commit 780be14f07df2a3ed6273b96ae857c278fd72718)
(cherry picked from commit 46baafedf13446f5c1b973d667f437bf7158d3c2)
---
.../scheduler/capacity/CapacitySchedulerConfiguration.java | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
index 631f1f8..1754a97 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
@@ -328,8 +328,11 @@ public class CapacitySchedulerConfiguration extends ReservationSchedulerConfigur
public static final String MAX_ASSIGN_PER_HEARTBEAT = PREFIX
+ "per-node-heartbeat.maximum-container-assignments";
+ /**
+ * Avoid potential risk that greedy assign multiple may involve
+ * */
@Private
- public static final int DEFAULT_MAX_ASSIGN_PER_HEARTBEAT = -1;
+ public static final int DEFAULT_MAX_ASSIGN_PER_HEARTBEAT = 100;
AppPriorityACLConfigurationParser priorityACLConfig = new AppPriorityACLConfigurationParser();
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[hadoop] 04/04: YARN-8944.
TestContainerAllocation.testUserLimitAllocationMultipleContainers failure
after YARN-8896. Contributed by Wilfred Spiegelenburg.
Posted by jh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jhung pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 373e32480b81a885cf28b8efa53244155022b98a
Author: Weiwei Yang <ww...@apache.org>
AuthorDate: Mon Oct 29 11:53:10 2018 +0800
YARN-8944. TestContainerAllocation.testUserLimitAllocationMultipleContainers failure after YARN-8896. Contributed by Wilfred Spiegelenburg.
(cherry picked from commit 1d90a0dd23c5d1bd52d04c303506806d0dc61cd2)
(cherry picked from commit 70efe253f369f579f05d833d45e6a1bae3e41dfe)
---
.../resourcemanager/scheduler/capacity/TestContainerAllocation.java | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerAllocation.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerAllocation.java
index c1fcda8..5164738 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerAllocation.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerAllocation.java
@@ -67,6 +67,7 @@ import org.junit.Before;
import org.junit.Test;
import static org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.MAXIMUM_ALLOCATION_MB;
+import static org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.MAX_ASSIGN_PER_HEARTBEAT;
public class TestContainerAllocation {
@@ -902,6 +903,9 @@ public class TestContainerAllocation {
CapacitySchedulerConfiguration newConf =
(CapacitySchedulerConfiguration) TestUtils
.getConfigurationWithMultipleQueues(conf);
+ // make sure an unlimited number of containers can be assigned,
+ // overriding the default of 100 after YARN-8896
+ newConf.set(MAX_ASSIGN_PER_HEARTBEAT, "-1");
newConf.setUserLimit("root.c", 50);
MockRM rm1 = new MockRM(newConf);
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[hadoop] 03/04: YARN-8915. Update the doc about the default value
of maximum-container-assignments for capacity scheduler. Contributed by
Zhankun Tang.
Posted by jh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jhung pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit fd977a138a1f8ee751b1ce03a2a275d52329b0ce
Author: Weiwei Yang <ww...@apache.org>
AuthorDate: Mon Oct 29 12:04:32 2018 +0800
YARN-8915. Update the doc about the default value of maximum-container-assignments for capacity scheduler. Contributed by Zhankun Tang.
(cherry picked from commit 78ea897b6701c82799a6231103ca7ee1f8b4efde)
(cherry picked from commit 7dd8eafe337a33689ff6f2f947ad152e06dbc55f)
---
.../hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
index 7bc08f0..1f0c436 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
@@ -295,7 +295,7 @@ Note, this feature should be disabled if YARN is deployed separately with the fi
| Property | Description |
|:---- |:---- |
| `yarn.scheduler.capacity.per-node-heartbeat.multiple-assignments-enabled` | Whether to allow multiple container assignments in one NodeManager heartbeat. Defaults to true. |
-| `yarn.scheduler.capacity.per-node-heartbeat.maximum-container-assignments` | If `multiple-assignments-enabled` is true, the maximum amount of containers that can be assigned in one NodeManager heartbeat. Defaults to -1, which sets no limit. |
+| `yarn.scheduler.capacity.per-node-heartbeat.maximum-container-assignments` | If `multiple-assignments-enabled` is true, the maximum amount of containers that can be assigned in one NodeManager heartbeat. Default value is 100, which limits the maximum number of container assignments per heartbeat to 100. Set this value to -1 will disable this limit. |
| `yarn.scheduler.capacity.per-node-heartbeat.maximum-offswitch-assignments` | If `multiple-assignments-enabled` is true, the maximum amount of off-switch containers that can be assigned in one NodeManager heartbeat. Defaults to 1, which represents only one off-switch allocation allowed in one heartbeat. |
###Reviewing the configuration of the CapacityScheduler
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[hadoop] 02/04: YARN-6851. Capacity Scheduler: document configs for
controlling # containers allowed to be allocated per node heartbeat.
Posted by jh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jhung pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit ea345de738e6432178e3c13400981c6cd5cb1bd0
Author: Wei Yan <we...@apache.org>
AuthorDate: Wed Nov 29 14:37:57 2017 -0800
YARN-6851. Capacity Scheduler: document configs for controlling # containers allowed to be allocated per node heartbeat.
(cherry picked from commit 333ef303ff0caf9adfd378652a8f966377901768)
---
.../hadoop-yarn-site/src/site/markdown/CapacityScheduler.md | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
index 9843879..7bc08f0 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
@@ -288,6 +288,16 @@ Capacity Scheduler leverages `Delay Scheduling` to honor task locality constrain
Note, this feature should be disabled if YARN is deployed separately with the file system, as locality is meaningless. This can be done by setting `yarn.scheduler.capacity.node-locality-delay` to `-1`, in this case, request's locality constraint is ignored.
+ * Container Allocation per NodeManager Heartbeat
+
+ The `CapacityScheduler` supports the following parameters to control how many containers can be allocated in each NodeManager heartbeat.
+
+| Property | Description |
+|:---- |:---- |
+| `yarn.scheduler.capacity.per-node-heartbeat.multiple-assignments-enabled` | Whether to allow multiple container assignments in one NodeManager heartbeat. Defaults to true. |
+| `yarn.scheduler.capacity.per-node-heartbeat.maximum-container-assignments` | If `multiple-assignments-enabled` is true, the maximum amount of containers that can be assigned in one NodeManager heartbeat. Defaults to -1, which sets no limit. |
+| `yarn.scheduler.capacity.per-node-heartbeat.maximum-offswitch-assignments` | If `multiple-assignments-enabled` is true, the maximum amount of off-switch containers that can be assigned in one NodeManager heartbeat. Defaults to 1, which represents only one off-switch allocation allowed in one heartbeat. |
+
###Reviewing the configuration of the CapacityScheduler
Once the installation and configuration is completed, you can review it after starting the YARN cluster from the web-ui.
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org