You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2021/11/10 16:21:47 UTC
[GitHub] [hadoop] afchung opened a new pull request #3642: YARN-10760. Number of allocated OPPORTUNISTIC containers can dip below 0
afchung opened a new pull request #3642:
URL: https://github.com/apache/hadoop/pull/3642
### Description of PR
* Checks that we do not release containers that do not exist
* Checks that we increment recovered allocated OPPORTUNISTIC containers on
RM restart
* Adds corresponding test
### How was this patch tested?
* Unit test added
* Deployment to production environment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] afchung commented on a change in pull request #3642: YARN-10760. Number of allocated OPPORTUNISTIC containers can dip below 0
Posted by GitBox <gi...@apache.org>.
afchung commented on a change in pull request #3642:
URL: https://github.com/apache/hadoop/pull/3642#discussion_r746917343
##########
File path: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestOpportunisticContainerAllocatorAMService.java
##########
@@ -817,6 +825,56 @@ public void testOpportunisticSchedulerMetrics() throws Exception {
metrics.getAggregatedReleasedContainers());
}
+ /**
+ * Tests that, if a node has running opportunistic containers when the RM
+ * is down, RM is able to reflect the opportunistic containers
+ * in its metrics upon RM recovery.
+ */
+ @Test
+ public void testMetricsRetainsAllocatedOpportunisticAfterRMRestart()
+ throws Exception {
+ HashMap<NodeId, MockNM> nodes = new HashMap<>();
+ MockNM nm1 = new MockNM("h1:1234", 4096, rm.getResourceTrackerService());
+ nodes.put(nm1.getNodeId(), nm1);
+ final RMApp app = MockRMAppSubmitter.submit(rm,
+ MockRMAppSubmissionData.Builder.createWithMemory(GB, rm)
+ .withAppName("app")
+ .withUser("user")
+ .withAcls(null)
+ .withQueue("default")
+ .build());
+
+ final ApplicationAttemptId appAttemptId =
+ app.getCurrentAppAttempt().getAppAttemptId();
+
+ final ContainerId recoverContainerId = ContainerId.newContainerId(
+ appAttemptId, 2);
+
+ final Resource fakeResource = Resource.newInstance(1024, 1);
+ final String fakeDiagnostics = "recover container";
+ final Priority fakePriority = Priority.newInstance(0);
+
+ final NMContainerStatus recoverContainerReport =
+ NMContainerStatus.newInstance(
+ recoverContainerId, 0, ContainerState.RUNNING,
+ fakeResource, fakeDiagnostics, 0,
+ fakePriority, 0, null,
+ ExecutionType.OPPORTUNISTIC, -1);
+
+ rm.registerNode(
+ "h1:1234", 4096, 1,
+ Collections.singletonList(
+ appAttemptId.getApplicationId()),
+ Collections.singletonList(recoverContainerReport));
+
+ OpportunisticSchedulerMetrics metrics =
+ OpportunisticSchedulerMetrics.getMetrics();
+ Assert.assertEquals(1,
Review comment:
Makes sense. This helped me find that while the test passes when run on its own, it fails when run with the rest of the testts in the class since the metrics were not re-initialized between tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3642: YARN-10760. Number of allocated OPPORTUNISTIC containers can dip below 0
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3642:
URL: https://github.com/apache/hadoop/pull/3642#issuecomment-965838202
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 6s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 2s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 51s | | trunk passed |
| +1 :green_heart: | compile | 4m 48s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 3m 45s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 17s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 51s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 31s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 22s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 48s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 59s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 39s | | the patch passed |
| +1 :green_heart: | compile | 4m 37s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 4m 37s | | the patch passed |
| +1 :green_heart: | compile | 3m 40s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 3m 40s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 11s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 40s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 19s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 10s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 59s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 37s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 57s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | unit | 106m 18s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not generate ASF License warnings. |
| | | 237m 26s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3642/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3642 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux d42faf61da34 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4c1aa91163cb40881f7cd0cd358a1987e0a03330 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3642/2/testReport/ |
| Max. process+thread count | 938 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3642/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a change in pull request #3642: YARN-10760. Number of allocated OPPORTUNISTIC containers can dip below 0
Posted by GitBox <gi...@apache.org>.
goiri commented on a change in pull request #3642:
URL: https://github.com/apache/hadoop/pull/3642#discussion_r746869941
##########
File path: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java
##########
@@ -720,7 +726,10 @@ public void completedContainer(RMContainer rmContainer,
SchedulerApplicationAttempt schedulerAttempt =
getCurrentAttemptForContainer(containerId);
if (schedulerAttempt != null) {
- schedulerAttempt.removeRMContainer(containerId);
+ if (schedulerAttempt.removeRMContainer(containerId)) {
Review comment:
Is this an OPPORTUNISTIC container for sure?
##########
File path: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestOpportunisticContainerAllocatorAMService.java
##########
@@ -817,6 +825,56 @@ public void testOpportunisticSchedulerMetrics() throws Exception {
metrics.getAggregatedReleasedContainers());
}
+ /**
+ * Tests that, if a node has running opportunistic containers when the RM
+ * is down, RM is able to reflect the opportunistic containers
+ * in its metrics upon RM recovery.
+ */
+ @Test
+ public void testMetricsRetainsAllocatedOpportunisticAfterRMRestart()
+ throws Exception {
+ HashMap<NodeId, MockNM> nodes = new HashMap<>();
+ MockNM nm1 = new MockNM("h1:1234", 4096, rm.getResourceTrackerService());
+ nodes.put(nm1.getNodeId(), nm1);
+ final RMApp app = MockRMAppSubmitter.submit(rm,
+ MockRMAppSubmissionData.Builder.createWithMemory(GB, rm)
Review comment:
Extracting this is more readable.
##########
File path: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestOpportunisticContainerAllocatorAMService.java
##########
@@ -817,6 +825,56 @@ public void testOpportunisticSchedulerMetrics() throws Exception {
metrics.getAggregatedReleasedContainers());
}
+ /**
+ * Tests that, if a node has running opportunistic containers when the RM
+ * is down, RM is able to reflect the opportunistic containers
+ * in its metrics upon RM recovery.
+ */
+ @Test
+ public void testMetricsRetainsAllocatedOpportunisticAfterRMRestart()
+ throws Exception {
+ HashMap<NodeId, MockNM> nodes = new HashMap<>();
+ MockNM nm1 = new MockNM("h1:1234", 4096, rm.getResourceTrackerService());
+ nodes.put(nm1.getNodeId(), nm1);
+ final RMApp app = MockRMAppSubmitter.submit(rm,
+ MockRMAppSubmissionData.Builder.createWithMemory(GB, rm)
+ .withAppName("app")
+ .withUser("user")
+ .withAcls(null)
+ .withQueue("default")
+ .build());
+
+ final ApplicationAttemptId appAttemptId =
+ app.getCurrentAppAttempt().getAppAttemptId();
+
+ final ContainerId recoverContainerId = ContainerId.newContainerId(
+ appAttemptId, 2);
+
+ final Resource fakeResource = Resource.newInstance(1024, 1);
+ final String fakeDiagnostics = "recover container";
+ final Priority fakePriority = Priority.newInstance(0);
+
+ final NMContainerStatus recoverContainerReport =
+ NMContainerStatus.newInstance(
+ recoverContainerId, 0, ContainerState.RUNNING,
+ fakeResource, fakeDiagnostics, 0,
+ fakePriority, 0, null,
+ ExecutionType.OPPORTUNISTIC, -1);
+
+ rm.registerNode(
+ "h1:1234", 4096, 1,
+ Collections.singletonList(
+ appAttemptId.getApplicationId()),
+ Collections.singletonList(recoverContainerReport));
+
+ OpportunisticSchedulerMetrics metrics =
+ OpportunisticSchedulerMetrics.getMetrics();
+ Assert.assertEquals(1,
Review comment:
We probably want to assert before this too with a 0 and maybe one of the cases where we do not increment.
I have the feeling this test passes even without the code changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri merged pull request #3642: YARN-10760. Number of allocated OPPORTUNISTIC containers can dip below 0
Posted by GitBox <gi...@apache.org>.
goiri merged pull request #3642:
URL: https://github.com/apache/hadoop/pull/3642
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3642: YARN-10760. Number of allocated OPPORTUNISTIC containers can dip below 0
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3642:
URL: https://github.com/apache/hadoop/pull/3642#issuecomment-965691903
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 57s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 37m 9s | | trunk passed |
| +1 :green_heart: | compile | 1m 18s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 4s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 49s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 7s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 54s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 46s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 18s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 51s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 1s | | the patch passed |
| +1 :green_heart: | compile | 1m 12s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 0m 56s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 56s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 42s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 3s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 49s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 40s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 22s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 30s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 106m 54s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3642/1/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not generate ASF License warnings. |
| | | 210m 45s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestOpportunisticContainerAllocatorAMService |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3642/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3642 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 3a258a2f77b1 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / bb48aa53807d9efe85d083dca99adb054c895ff4 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3642/1/testReport/ |
| Max. process+thread count | 937 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3642/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] afchung commented on a change in pull request #3642: YARN-10760. Number of allocated OPPORTUNISTIC containers can dip below 0
Posted by GitBox <gi...@apache.org>.
afchung commented on a change in pull request #3642:
URL: https://github.com/apache/hadoop/pull/3642#discussion_r746917343
##########
File path: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestOpportunisticContainerAllocatorAMService.java
##########
@@ -817,6 +825,56 @@ public void testOpportunisticSchedulerMetrics() throws Exception {
metrics.getAggregatedReleasedContainers());
}
+ /**
+ * Tests that, if a node has running opportunistic containers when the RM
+ * is down, RM is able to reflect the opportunistic containers
+ * in its metrics upon RM recovery.
+ */
+ @Test
+ public void testMetricsRetainsAllocatedOpportunisticAfterRMRestart()
+ throws Exception {
+ HashMap<NodeId, MockNM> nodes = new HashMap<>();
+ MockNM nm1 = new MockNM("h1:1234", 4096, rm.getResourceTrackerService());
+ nodes.put(nm1.getNodeId(), nm1);
+ final RMApp app = MockRMAppSubmitter.submit(rm,
+ MockRMAppSubmissionData.Builder.createWithMemory(GB, rm)
+ .withAppName("app")
+ .withUser("user")
+ .withAcls(null)
+ .withQueue("default")
+ .build());
+
+ final ApplicationAttemptId appAttemptId =
+ app.getCurrentAppAttempt().getAppAttemptId();
+
+ final ContainerId recoverContainerId = ContainerId.newContainerId(
+ appAttemptId, 2);
+
+ final Resource fakeResource = Resource.newInstance(1024, 1);
+ final String fakeDiagnostics = "recover container";
+ final Priority fakePriority = Priority.newInstance(0);
+
+ final NMContainerStatus recoverContainerReport =
+ NMContainerStatus.newInstance(
+ recoverContainerId, 0, ContainerState.RUNNING,
+ fakeResource, fakeDiagnostics, 0,
+ fakePriority, 0, null,
+ ExecutionType.OPPORTUNISTIC, -1);
+
+ rm.registerNode(
+ "h1:1234", 4096, 1,
+ Collections.singletonList(
+ appAttemptId.getApplicationId()),
+ Collections.singletonList(recoverContainerReport));
+
+ OpportunisticSchedulerMetrics metrics =
+ OpportunisticSchedulerMetrics.getMetrics();
+ Assert.assertEquals(1,
Review comment:
Makes sense. This helped me find that while the test passes when run on its own, it fails when run with the rest of the tests in the class since the metrics were not re-initialized between tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] afchung commented on pull request #3642: YARN-10760. Number of allocated OPPORTUNISTIC containers can dip below 0
Posted by GitBox <gi...@apache.org>.
afchung commented on pull request #3642:
URL: https://github.com/apache/hadoop/pull/3642#issuecomment-965673846
@goiri thanks for the review! Pushed a new commit to address your comments, please have another look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] afchung commented on a change in pull request #3642: YARN-10760. Number of allocated OPPORTUNISTIC containers can dip below 0
Posted by GitBox <gi...@apache.org>.
afchung commented on a change in pull request #3642:
URL: https://github.com/apache/hadoop/pull/3642#discussion_r746877782
##########
File path: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java
##########
@@ -720,7 +726,10 @@ public void completedContainer(RMContainer rmContainer,
SchedulerApplicationAttempt schedulerAttempt =
getCurrentAttemptForContainer(containerId);
if (schedulerAttempt != null) {
- schedulerAttempt.removeRMContainer(containerId);
+ if (schedulerAttempt.removeRMContainer(containerId)) {
Review comment:
Yes, it's in the else clause of `if (rmContainer.getExecutionType() == ExecutionType.GUARANTEED)`, where for now there's only one other type of `ExecutionType`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org