You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2021/11/11 19:24:29 UTC
[GitHub] [hadoop] afchung opened a new pull request #3646: YARN-11003. Make RMNode aware of all (OContainer inclusive) allocated resources
afchung opened a new pull request #3646:
URL: https://github.com/apache/hadoop/pull/3646
### Description of PR
* Adds a new method in RMNode: getAllocatedContainerResource, which
records how much resource is allocated to containers on a node
* Adds up RUNNING and NEW container resources in HB loop to keep track
of resources allocated for containers
### How was this patch tested?
* Unit test
* Deployment in a production cluster
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3646: YARN-11003. Make RMNode aware of all (OContainer inclusive) allocated resources
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3646:
URL: https://github.com/apache/hadoop/pull/3646#issuecomment-970682650
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 49s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 36s | | trunk passed |
| +1 :green_heart: | compile | 21m 51s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 20m 7s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 41s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 50s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 41s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 37s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 3s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 5s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 37s | | the patch passed |
| +1 :green_heart: | compile | 29m 17s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 29m 17s | | the patch passed |
| +1 :green_heart: | compile | 25m 24s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 25m 24s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 36s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3646/4/artifact/out/results-checkstyle-root.txt) | root: The patch generated 3 new + 87 unchanged - 2 fixed = 90 total (was 89) |
| +1 :green_heart: | mvnsite | 2m 2s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 48s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 38s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 59s | | the patch passed |
| +1 :green_heart: | shadedclient | 27m 21s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 114m 6s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 12m 29s | | hadoop-sls in the patch passed. |
| +1 :green_heart: | asflicense | 1m 1s | | The patch does not generate ASF License warnings. |
| | | 340m 2s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3646/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3646 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 804452b08978 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7457e09b25fece8192063a05305d8152d5f9b9ae |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3646/4/testReport/ |
| Max. process+thread count | 948 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-tools/hadoop-sls U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3646/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a change in pull request #3646: YARN-11003. Make RMNode aware of all (OContainer inclusive) allocated resources
Posted by GitBox <gi...@apache.org>.
goiri commented on a change in pull request #3646:
URL: https://github.com/apache/hadoop/pull/3646#discussion_r747780345
##########
File path: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java
##########
@@ -358,6 +360,94 @@ public void testContainerUpdate() throws InterruptedException{
.getContainerId());
}
+ /**
+ * Tests that allocated container resources are counted correctly in
+ * {@link org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNode}
+ * upon a node update. Resources should be counted for both GUARANTEED
+ * and OPPORTUNISTIC containers.
+ */
+ @Test (timeout = 5000)
+ public void testAllocatedContainerUpdate() {
+ NodeStatus mockNodeStatus = createMockNodeStatus();
+ //Start the node
+ node.handle(new RMNodeStartedEvent(null, null, null, mockNodeStatus));
+
+ NodeId nodeId = BuilderUtils.newNodeId("localhost:1", 1);
+
+ ApplicationId app0 = BuilderUtils.newApplicationId(0, 0);
+ ContainerId newContainerId = BuilderUtils.newContainerId(
+ BuilderUtils.newApplicationAttemptId(app0, 0), 0);
+ ContainerId runningContainerId = BuilderUtils.newContainerId(
+ BuilderUtils.newApplicationAttemptId(app0, 0), 1);
+ ContainerId newOppContainerId = BuilderUtils.newContainerId(
+ BuilderUtils.newApplicationAttemptId(app0, 0), 2);
+ ContainerId runningOppContainerId = BuilderUtils.newContainerId(
+ BuilderUtils.newApplicationAttemptId(app0, 0), 3);
+
+ rmContext.getRMApps().put(app0, Mockito.mock(RMApp.class));
+
+ RMNodeStatusEvent statusEventFromNode1 = getMockRMNodeStatusEvent(null);
+ ContainerStatus newContainerStatusFromNode = mock(ContainerStatus.class);
+ ContainerStatus runningContainerStatusFromNode =
+ mock(ContainerStatus.class);
+
+ final Resource newContainerCapability =
+ Resource.newInstance(100, 1);
+ final Resource runningContainerCapability =
+ Resource.newInstance(200, 2);
+ doReturn(newContainerId).when(newContainerStatusFromNode)
+ .getContainerId();
+ doReturn(ContainerState.NEW).when(newContainerStatusFromNode)
+ .getState();
+ doReturn(newContainerCapability).when(newContainerStatusFromNode)
+ .getCapability();
+ doReturn(runningContainerId).when(runningContainerStatusFromNode)
+ .getContainerId();
+ doReturn(ContainerState.RUNNING).when(runningContainerStatusFromNode)
+ .getState();
+ doReturn(runningContainerCapability).when(runningContainerStatusFromNode)
+ .getCapability();
+ doReturn(Arrays.asList(
+ newContainerStatusFromNode, runningContainerStatusFromNode))
+ .when(statusEventFromNode1).getContainers();
+ node.handle(statusEventFromNode1);
+ Assert.assertTrue(Resources.equals(
Review comment:
Resource has equals which actual is called by this, can't we just use assertEquals()?
##########
File path: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java
##########
@@ -358,6 +360,94 @@ public void testContainerUpdate() throws InterruptedException{
.getContainerId());
}
+ /**
+ * Tests that allocated container resources are counted correctly in
+ * {@link org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNode}
+ * upon a node update. Resources should be counted for both GUARANTEED
+ * and OPPORTUNISTIC containers.
+ */
+ @Test (timeout = 5000)
+ public void testAllocatedContainerUpdate() {
+ NodeStatus mockNodeStatus = createMockNodeStatus();
+ //Start the node
+ node.handle(new RMNodeStartedEvent(null, null, null, mockNodeStatus));
+
+ NodeId nodeId = BuilderUtils.newNodeId("localhost:1", 1);
+
+ ApplicationId app0 = BuilderUtils.newApplicationId(0, 0);
+ ContainerId newContainerId = BuilderUtils.newContainerId(
+ BuilderUtils.newApplicationAttemptId(app0, 0), 0);
+ ContainerId runningContainerId = BuilderUtils.newContainerId(
+ BuilderUtils.newApplicationAttemptId(app0, 0), 1);
+ ContainerId newOppContainerId = BuilderUtils.newContainerId(
+ BuilderUtils.newApplicationAttemptId(app0, 0), 2);
+ ContainerId runningOppContainerId = BuilderUtils.newContainerId(
+ BuilderUtils.newApplicationAttemptId(app0, 0), 3);
+
+ rmContext.getRMApps().put(app0, Mockito.mock(RMApp.class));
+
+ RMNodeStatusEvent statusEventFromNode1 = getMockRMNodeStatusEvent(null);
+ ContainerStatus newContainerStatusFromNode = mock(ContainerStatus.class);
+ ContainerStatus runningContainerStatusFromNode =
+ mock(ContainerStatus.class);
+
+ final Resource newContainerCapability =
+ Resource.newInstance(100, 1);
Review comment:
It would be nice to have something explaining this values; either a constant, a variable name or a comment.
##########
File path: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java
##########
@@ -358,6 +360,94 @@ public void testContainerUpdate() throws InterruptedException{
.getContainerId());
}
+ /**
+ * Tests that allocated container resources are counted correctly in
+ * {@link org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNode}
+ * upon a node update. Resources should be counted for both GUARANTEED
+ * and OPPORTUNISTIC containers.
+ */
+ @Test (timeout = 5000)
+ public void testAllocatedContainerUpdate() {
+ NodeStatus mockNodeStatus = createMockNodeStatus();
+ //Start the node
+ node.handle(new RMNodeStartedEvent(null, null, null, mockNodeStatus));
+
+ NodeId nodeId = BuilderUtils.newNodeId("localhost:1", 1);
+
+ ApplicationId app0 = BuilderUtils.newApplicationId(0, 0);
+ ContainerId newContainerId = BuilderUtils.newContainerId(
+ BuilderUtils.newApplicationAttemptId(app0, 0), 0);
+ ContainerId runningContainerId = BuilderUtils.newContainerId(
+ BuilderUtils.newApplicationAttemptId(app0, 0), 1);
+ ContainerId newOppContainerId = BuilderUtils.newContainerId(
+ BuilderUtils.newApplicationAttemptId(app0, 0), 2);
+ ContainerId runningOppContainerId = BuilderUtils.newContainerId(
+ BuilderUtils.newApplicationAttemptId(app0, 0), 3);
+
+ rmContext.getRMApps().put(app0, Mockito.mock(RMApp.class));
+
+ RMNodeStatusEvent statusEventFromNode1 = getMockRMNodeStatusEvent(null);
+ ContainerStatus newContainerStatusFromNode = mock(ContainerStatus.class);
+ ContainerStatus runningContainerStatusFromNode =
+ mock(ContainerStatus.class);
+
+ final Resource newContainerCapability =
+ Resource.newInstance(100, 1);
+ final Resource runningContainerCapability =
+ Resource.newInstance(200, 2);
+ doReturn(newContainerId).when(newContainerStatusFromNode)
+ .getContainerId();
+ doReturn(ContainerState.NEW).when(newContainerStatusFromNode)
+ .getState();
+ doReturn(newContainerCapability).when(newContainerStatusFromNode)
+ .getCapability();
+ doReturn(runningContainerId).when(runningContainerStatusFromNode)
+ .getContainerId();
+ doReturn(ContainerState.RUNNING).when(runningContainerStatusFromNode)
+ .getState();
+ doReturn(runningContainerCapability).when(runningContainerStatusFromNode)
+ .getCapability();
+ doReturn(Arrays.asList(
+ newContainerStatusFromNode, runningContainerStatusFromNode))
+ .when(statusEventFromNode1).getContainers();
+ node.handle(statusEventFromNode1);
+ Assert.assertTrue(Resources.equals(
+ node.getAllocatedContainerResource(),
+ Resource.newInstance(300, 3)));
+
+ RMNodeStatusEvent statusEventFromNode2 = getMockRMNodeStatusEvent(null);
+ ContainerStatus newOppContainerStatusFromNode = mock(ContainerStatus.class);
+ ContainerStatus runningOppContainerStatusFromNode =
+ mock(ContainerStatus.class);
+ doReturn(newOppContainerId).when(newOppContainerStatusFromNode)
+ .getContainerId();
+ doReturn(ContainerState.NEW).when(newOppContainerStatusFromNode)
+ .getState();
+ doReturn(newContainerCapability).when(newOppContainerStatusFromNode)
+ .getCapability();
+ doReturn(ExecutionType.OPPORTUNISTIC)
+ .when(newOppContainerStatusFromNode)
+ .getExecutionType();
+ doReturn(runningOppContainerId).when(runningOppContainerStatusFromNode)
+ .getContainerId();
+ doReturn(ContainerState.RUNNING).when(runningOppContainerStatusFromNode)
+ .getState();
+ doReturn(runningContainerCapability).when(runningOppContainerStatusFromNode)
+ .getCapability();
+ doReturn(ExecutionType.OPPORTUNISTIC)
+ .when(runningOppContainerStatusFromNode)
+ .getExecutionType();
+ doReturn(Arrays.asList(
+ newContainerStatusFromNode, runningContainerStatusFromNode,
+ newOppContainerStatusFromNode, runningOppContainerStatusFromNode))
+ .when(statusEventFromNode2).getContainers();
+
+ node.handle(statusEventFromNode2);
+ Assert.assertTrue(Resources.equals(
Review comment:
assertEquals()
##########
File path: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java
##########
@@ -358,6 +360,94 @@ public void testContainerUpdate() throws InterruptedException{
.getContainerId());
}
+ /**
+ * Tests that allocated container resources are counted correctly in
+ * {@link org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNode}
+ * upon a node update. Resources should be counted for both GUARANTEED
+ * and OPPORTUNISTIC containers.
+ */
+ @Test (timeout = 5000)
+ public void testAllocatedContainerUpdate() {
+ NodeStatus mockNodeStatus = createMockNodeStatus();
+ //Start the node
+ node.handle(new RMNodeStartedEvent(null, null, null, mockNodeStatus));
+
+ NodeId nodeId = BuilderUtils.newNodeId("localhost:1", 1);
+
+ ApplicationId app0 = BuilderUtils.newApplicationId(0, 0);
+ ContainerId newContainerId = BuilderUtils.newContainerId(
+ BuilderUtils.newApplicationAttemptId(app0, 0), 0);
+ ContainerId runningContainerId = BuilderUtils.newContainerId(
+ BuilderUtils.newApplicationAttemptId(app0, 0), 1);
+ ContainerId newOppContainerId = BuilderUtils.newContainerId(
+ BuilderUtils.newApplicationAttemptId(app0, 0), 2);
+ ContainerId runningOppContainerId = BuilderUtils.newContainerId(
+ BuilderUtils.newApplicationAttemptId(app0, 0), 3);
+
+ rmContext.getRMApps().put(app0, Mockito.mock(RMApp.class));
+
+ RMNodeStatusEvent statusEventFromNode1 = getMockRMNodeStatusEvent(null);
+ ContainerStatus newContainerStatusFromNode = mock(ContainerStatus.class);
Review comment:
Can we create this mock in a static method to make the test easier to read?
##########
File path: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java
##########
@@ -358,6 +360,94 @@ public void testContainerUpdate() throws InterruptedException{
.getContainerId());
}
+ /**
+ * Tests that allocated container resources are counted correctly in
+ * {@link org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNode}
+ * upon a node update. Resources should be counted for both GUARANTEED
+ * and OPPORTUNISTIC containers.
+ */
+ @Test (timeout = 5000)
+ public void testAllocatedContainerUpdate() {
+ NodeStatus mockNodeStatus = createMockNodeStatus();
+ //Start the node
+ node.handle(new RMNodeStartedEvent(null, null, null, mockNodeStatus));
Review comment:
Can we also assert for a couple more points?
Including when starts empty.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bibinchundatt commented on a change in pull request #3646: YARN-11003. Make RMNode aware of all (OContainer inclusive) allocated resources
Posted by GitBox <gi...@apache.org>.
bibinchundatt commented on a change in pull request #3646:
URL: https://github.com/apache/hadoop/pull/3646#discussion_r750070889
##########
File path: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
##########
@@ -464,6 +466,11 @@ public Resource getTotalCapability() {
return this.totalCapability;
}
+ @Override
Review comment:
We need to also update the data when we addNodeTransition
##########
File path: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java
##########
@@ -358,6 +375,119 @@ public void testContainerUpdate() throws InterruptedException{
.getContainerId());
}
+ /**
+ * Tests that allocated container resources are counted correctly in
+ * {@link org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNode}
+ * upon a node update. Resources should be counted for both GUARANTEED
+ * and OPPORTUNISTIC containers.
+ */
+ @Test (timeout = 5000)
+ public void testAllocatedContainerUpdate() {
+ NodeStatus mockNodeStatus = createMockNodeStatus();
+ //Start the node
+ node.handle(new RMNodeStartedEvent(null, null, null, mockNodeStatus));
Review comment:
Update the testcase with addNode resource update transition too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3646: YARN-11003. Make RMNode aware of all (OContainer inclusive) allocated resources
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3646:
URL: https://github.com/apache/hadoop/pull/3646#issuecomment-966789708
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 58s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 38s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 24m 35s | | trunk passed |
| +1 :green_heart: | compile | 23m 30s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 19m 50s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 55s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 45s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 31s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 22s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 0s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 28s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 50s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 22s | | the patch passed |
| +1 :green_heart: | compile | 22m 38s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 22m 38s | | the patch passed |
| +1 :green_heart: | compile | 20m 4s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 20m 4s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 3m 50s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3646/2/artifact/out/results-checkstyle-root.txt) | root: The patch generated 2 new + 87 unchanged - 2 fixed = 89 total (was 89) |
| +1 :green_heart: | mvnsite | 1m 45s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 27s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 23s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 19s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 53s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 104m 59s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 12m 13s | | hadoop-sls in the patch passed. |
| +1 :green_heart: | asflicense | 0m 50s | | The patch does not generate ASF License warnings. |
| | | 317m 40s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3646/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3646 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux f913f494955d 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b6ae804744b3ee540b8d289ece04527556f9586c |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3646/2/testReport/ |
| Max. process+thread count | 955 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-tools/hadoop-sls U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3646/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] afchung commented on pull request #3646: YARN-11003. Make RMNode aware of all (OContainer inclusive) allocated resources
Posted by GitBox <gi...@apache.org>.
afchung commented on pull request #3646:
URL: https://github.com/apache/hadoop/pull/3646#issuecomment-970389378
@bibinchundatt thanks for the review! Updated the PR to address your comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bibinchundatt commented on a change in pull request #3646: YARN-11003. Make RMNode aware of all (OContainer inclusive) allocated resources
Posted by GitBox <gi...@apache.org>.
bibinchundatt commented on a change in pull request #3646:
URL: https://github.com/apache/hadoop/pull/3646#discussion_r750058852
##########
File path: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNode.java
##########
@@ -104,6 +105,14 @@
*/
public Resource getTotalCapability();
+ /**
+ * the total allocated resources to containers.
Review comment:
Start with upper case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3646: YARN-11003. Make RMNode aware of all (OContainer inclusive) allocated resources
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3646:
URL: https://github.com/apache/hadoop/pull/3646#issuecomment-966724419
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 44s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 23m 8s | | trunk passed |
| +1 :green_heart: | compile | 23m 35s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 20m 13s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 53s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 3s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 50s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 41s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 22s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 39s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 21s | | the patch passed |
| +1 :green_heart: | compile | 24m 28s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 24m 28s | | the patch passed |
| +1 :green_heart: | compile | 20m 25s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 20m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 3m 58s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 55s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 46s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 39s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 28s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 8s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 96m 38s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 12m 37s | | hadoop-sls in the patch passed. |
| +1 :green_heart: | asflicense | 0m 59s | | The patch does not generate ASF License warnings. |
| | | 310m 0s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3646/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3646 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux c3b60e1341bd 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9e60671f3683ef9e3842bbabd59f37d4aa36c33d |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3646/1/testReport/ |
| Max. process+thread count | 976 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-tools/hadoop-sls U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3646/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bibinchundatt commented on a change in pull request #3646: YARN-11003. Make RMNode aware of all (OContainer inclusive) allocated resources
Posted by GitBox <gi...@apache.org>.
bibinchundatt commented on a change in pull request #3646:
URL: https://github.com/apache/hadoop/pull/3646#discussion_r750058852
##########
File path: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNode.java
##########
@@ -104,6 +105,14 @@
*/
public Resource getTotalCapability();
+ /**
+ * the total allocated resources to containers.
Review comment:
Start with upper case in comments.. This will include the sum of O+G containers queued + running + paused on the node.. Comment cane be more explanatory..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3646: YARN-11003. Make RMNode aware of all (OContainer inclusive) allocated resources
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3646:
URL: https://github.com/apache/hadoop/pull/3646#issuecomment-969502883
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 53s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 56s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 1s | | trunk passed |
| +1 :green_heart: | compile | 21m 57s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 19m 19s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 43s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 4s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 50s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 45s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 14s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 23s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 24s | | the patch passed |
| +1 :green_heart: | compile | 22m 23s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 22m 23s | | the patch passed |
| +1 :green_heart: | compile | 19m 9s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 19m 9s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 3m 37s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3646/3/artifact/out/results-checkstyle-root.txt) | root: The patch generated 2 new + 87 unchanged - 2 fixed = 89 total (was 89) |
| +1 :green_heart: | mvnsite | 2m 4s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 48s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 43s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 35s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 8s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 96m 4s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 12m 40s | | hadoop-sls in the patch passed. |
| +1 :green_heart: | asflicense | 1m 1s | | The patch does not generate ASF License warnings. |
| | | 303m 20s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3646/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3646 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 0dc4f999a317 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2ee6bf8f93f7c4b1825ec367d4fbf76ae66d2fc4 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3646/3/testReport/ |
| Max. process+thread count | 966 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-tools/hadoop-sls U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3646/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] afchung commented on a change in pull request #3646: YARN-11003. Make RMNode aware of all (OContainer inclusive) allocated resources
Posted by GitBox <gi...@apache.org>.
afchung commented on a change in pull request #3646:
URL: https://github.com/apache/hadoop/pull/3646#discussion_r750355189
##########
File path: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
##########
@@ -464,6 +466,11 @@ public Resource getTotalCapability() {
return this.totalCapability;
}
+ @Override
Review comment:
Good catch :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri merged pull request #3646: YARN-11003. Make RMNode aware of all (OContainer inclusive) allocated resources
Posted by GitBox <gi...@apache.org>.
goiri merged pull request #3646:
URL: https://github.com/apache/hadoop/pull/3646
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] afchung commented on pull request #3646: YARN-11003. Make RMNode aware of all (OContainer inclusive) allocated resources
Posted by GitBox <gi...@apache.org>.
afchung commented on pull request #3646:
URL: https://github.com/apache/hadoop/pull/3646#issuecomment-966657911
@goiri thanks for the review! Have pushed a new commit to address your comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a change in pull request #3646: YARN-11003. Make RMNode aware of all (OContainer inclusive) allocated resources
Posted by GitBox <gi...@apache.org>.
goiri commented on a change in pull request #3646:
URL: https://github.com/apache/hadoop/pull/3646#discussion_r749613001
##########
File path: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java
##########
@@ -358,6 +375,119 @@ public void testContainerUpdate() throws InterruptedException{
.getContainerId());
}
+ /**
+ * Tests that allocated container resources are counted correctly in
+ * {@link org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNode}
+ * upon a node update. Resources should be counted for both GUARANTEED
+ * and OPPORTUNISTIC containers.
+ */
+ @Test (timeout = 5000)
+ public void testAllocatedContainerUpdate() {
+ NodeStatus mockNodeStatus = createMockNodeStatus();
+ //Start the node
+ node.handle(new RMNodeStartedEvent(null, null, null, mockNodeStatus));
+
+ // Make sure that the node starts with no allocated resources
+ Assert.assertEquals(node.getAllocatedContainerResource(), Resources.none());
+
+ ApplicationId app0 = BuilderUtils.newApplicationId(0, 0);
+ final ContainerId newContainerId = BuilderUtils.newContainerId(
+ BuilderUtils.newApplicationAttemptId(app0, 0), 0);
+ final ContainerId runningContainerId = BuilderUtils.newContainerId(
+ BuilderUtils.newApplicationAttemptId(app0, 0), 1);
+
+ rmContext.getRMApps().put(app0, Mockito.mock(RMApp.class));
+
+ RMNodeStatusEvent statusEventFromNode1 = getMockRMNodeStatusEvent(null);
+
+ final List<ContainerStatus> containerStatuses = new ArrayList<>();
+
+ // Use different memory and VCores for new and running state containers
+ // to test that they add up correctly
+ final Resource newContainerCapability =
+ Resource.newInstance(100, 1);
+ final Resource runningContainerCapability =
+ Resource.newInstance(200, 2);
+ final Resource completedContainerCapability =
+ Resource.newInstance(50, 3);
+ final ContainerStatus newContainerStatusFromNode = getMockContainerStatus(
+ newContainerId, newContainerCapability, ContainerState.NEW);
+ final ContainerStatus runningContainerStatusFromNode =
+ getMockContainerStatus(runningContainerId, runningContainerCapability,
+ ContainerState.RUNNING);
+
+ containerStatuses.addAll(Arrays.asList(
+ newContainerStatusFromNode, runningContainerStatusFromNode));
+ doReturn(containerStatuses).when(statusEventFromNode1).getContainers();
+ node.handle(statusEventFromNode1);
+ Assert.assertEquals(node.getAllocatedContainerResource(),
+ Resource.newInstance(300, 3));
+
+ final ContainerId newOppContainerId = BuilderUtils.newContainerId(
+ BuilderUtils.newApplicationAttemptId(app0, 0), 2);
+ final ContainerId runningOppContainerId = BuilderUtils.newContainerId(
+ BuilderUtils.newApplicationAttemptId(app0, 0), 3);
+
+ // Use the same resource capability as in previous for opportunistic case
+ RMNodeStatusEvent statusEventFromNode2 = getMockRMNodeStatusEvent(null);
+ final ContainerStatus newOppContainerStatusFromNode =
+ getMockContainerStatus(newOppContainerId, newContainerCapability,
+ ContainerState.NEW, ExecutionType.OPPORTUNISTIC);
+ final ContainerStatus runningOppContainerStatusFromNode =
+ getMockContainerStatus(runningOppContainerId,
+ runningContainerCapability, ContainerState.RUNNING,
+ ExecutionType.OPPORTUNISTIC);
+
+ containerStatuses.addAll(Arrays.asList(
+ newOppContainerStatusFromNode, runningOppContainerStatusFromNode));
+
+ // Pass in both guaranteed and opportunistic container statuses
+ doReturn(containerStatuses).when(statusEventFromNode2).getContainers();
+
+ node.handle(statusEventFromNode2);
+
+ // The result here should be double the first check,
+ // since allocated resources are doubled, just
+ // with different execution types
+ Assert.assertEquals(node.getAllocatedContainerResource(),
+ Resource.newInstance(600, 6));
+
+ RMNodeStatusEvent statusEventFromNode3 = getMockRMNodeStatusEvent(null);
+ final ContainerId completedContainerId = BuilderUtils.newContainerId(
+ BuilderUtils.newApplicationAttemptId(app0, 0), 4);
+ final ContainerId completedOppContainerId = BuilderUtils.newContainerId(
+ BuilderUtils.newApplicationAttemptId(app0, 0), 5);
+ final ContainerStatus completedContainerStatusFromNode =
+ getMockContainerStatus(completedContainerId, completedContainerCapability,
+ ContainerState.COMPLETE, ExecutionType.OPPORTUNISTIC);
+ final ContainerStatus completedOppContainerStatusFromNode =
+ getMockContainerStatus(completedOppContainerId,
+ completedContainerCapability, ContainerState.COMPLETE,
+ ExecutionType.OPPORTUNISTIC);
+
+ containerStatuses.addAll(Arrays.asList(
+ completedContainerStatusFromNode, completedOppContainerStatusFromNode));
+
+ doReturn(containerStatuses).when(statusEventFromNode3).getContainers();
+ node.handle(statusEventFromNode3);
+
+ // Adding completed containers should not have changed
+ // the resources allocated
+ Assert.assertEquals(node.getAllocatedContainerResource(),
Review comment:
Usually we would put the expected first in the assertEquals
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org