You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2022/04/28 15:30:45 UTC
[GitHub] [hadoop] ashutoshcipher opened a new pull request, #4244: YARN-11119. Backport YARN-10538 to branch-2.10
ashutoshcipher opened a new pull request, #4244:
URL: https://github.com/apache/hadoop/pull/4244
### Description of PR
Backport YARN-10538 to branch-2.10
* JIRA: YARN-11119
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ashutoshcipher commented on pull request #4244: YARN-11119. Backport YARN-10538 to branch-2.10
Posted by GitBox <gi...@apache.org>.
ashutoshcipher commented on PR #4244:
URL: https://github.com/apache/hadoop/pull/4244#issuecomment-1114538988
@aajisaka - Please review this PR. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on a diff in pull request #4244: YARN-11119. Backport YARN-10538 to branch-2.10
Posted by GitBox <gi...@apache.org>.
aajisaka commented on code in PR #4244:
URL: https://github.com/apache/hadoop/pull/4244#discussion_r906887383
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRMRPCNodeUpdates.java:
##########
@@ -139,6 +146,48 @@ public void testAMRMDecommissioningNodes() throws Exception {
NodeUpdateType.NODE_DECOMMISSIONING, nr.getNodeUpdateType());
}
+ @Test
+ public void testAMRMRecommissioningNodes() throws Exception {
+ MockNM nm1 = rm.registerNode("127.0.0.1:1234", 10000);
+ MockNM nm2 = rm.registerNode("127.0.0.2:1234", 10000);
+ rm.drainEvents();
+
+ RMApp app1 = rm.submitApp(2000);
+
+ // Trigger the scheduling so the AM gets 'launched' on nm1
+ nm1.nodeHeartbeat(true);
+
+ RMAppAttempt attempt1 = app1.getCurrentAppAttempt();
+ MockAM am1 = rm.sendAMLaunched(attempt1.getAppAttemptId());
+
+ // register AM returns no unusable node
+ am1.registerAppAttempt();
+
+ // DECOMMISSION nm2
+ Integer decommissioningTimeout = 600;
+ syncNodeGracefulDecommission(nm2, decommissioningTimeout);
+
+ AllocateRequest allocateRequest1 =
+ AllocateRequest.newInstance(0, 0F, null, null, null);
+ AllocateResponse response1 =
+ allocate(attempt1.getAppAttemptId(), allocateRequest1);
+ List<NodeReport> updatedNodes = response1.getUpdatedNodes();
+ Assert.assertEquals(1, updatedNodes.size());
+
Review Comment:
Would you add the following lines which are contained in https://github.com/apache/hadoop/pull/2564/files ?
```
NodeReport nr = updatedNodes.iterator().next();
Assert.assertEquals(
decommissioningTimeout, nr.getDecommissioningTimeout());
Assert.assertEquals(
NodeUpdateType.NODE_DECOMMISSIONING, nr.getNodeUpdateType());
```
I'm +1 if that is addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4244: YARN-11119. Backport YARN-10538 to branch-2.10
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4244:
URL: https://github.com/apache/hadoop/pull/4244#issuecomment-1166668516
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 9m 22s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ branch-2.10 Compile Tests _ |
| +1 :green_heart: | mvninstall | 17m 4s | | branch-2.10 passed |
| +1 :green_heart: | compile | 1m 3s | | branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10 |
| +1 :green_heart: | compile | 0m 51s | | branch-2.10 passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| +1 :green_heart: | checkstyle | 0m 47s | | branch-2.10 passed |
| +1 :green_heart: | mvnsite | 1m 3s | | branch-2.10 passed |
| +1 :green_heart: | javadoc | 0m 51s | | branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10 |
| +1 :green_heart: | javadoc | 0m 37s | | branch-2.10 passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| +1 :green_heart: | spotbugs | 2m 13s | | branch-2.10 passed |
| -0 :warning: | patch | 2m 33s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 47s | | the patch passed |
| +1 :green_heart: | compile | 0m 50s | | the patch passed with JDK Azul Systems, Inc.-1.7.0_262-b10 |
| +1 :green_heart: | javac | 0m 50s | | the patch passed |
| +1 :green_heart: | compile | 0m 44s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| +1 :green_heart: | javac | 0m 44s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 28s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 48s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 35s | | the patch passed with JDK Azul Systems, Inc.-1.7.0_262-b10 |
| +1 :green_heart: | javadoc | 0m 29s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| +1 :green_heart: | spotbugs | 1m 45s | | the patch passed |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 65m 10s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not generate ASF License warnings. |
| | | 109m 55s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4244/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4244 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux a65a96b6fef0 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-2.10 / c19ccc4c7696918ecfc692f68e4d6c9a06ea1d89 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_262-b10 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4244/4/testReport/ |
| Max. process+thread count | 847 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4244/4/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ashutoshcipher commented on a diff in pull request #4244: YARN-11119. Backport YARN-10538 to branch-2.10
Posted by GitBox <gi...@apache.org>.
ashutoshcipher commented on code in PR #4244:
URL: https://github.com/apache/hadoop/pull/4244#discussion_r906870462
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRMRPCNodeUpdates.java:
##########
@@ -98,6 +111,47 @@ public AllocateResponse run() throws Exception {
});
}
+ @Test
+ public void testAMRMRecommissioningNodes() throws Exception {
+ MockNM nm1 = rm.registerNode("127.0.0.1:1234", 10000);
+ MockNM nm2 = rm.registerNode("127.0.0.2:1234", 10000);
+ rm.drainEvents();
+
+ RMApp app1 = rm.submitApp(2000);
+
+ // Trigger the scheduling so the AM gets 'launched' on nm1
+ nm1.nodeHeartbeat(true);
+
+ RMAppAttempt attempt1 = app1.getCurrentAppAttempt();
+ MockAM am1 = rm.sendAMLaunched(attempt1.getAppAttemptId());
+
+ // register AM returns no unusable node
+ am1.registerAppAttempt();
+
+ // DECOMMISSION nm2
+ syncNodeGracefulDecommission(nm2);
+
+ AllocateRequest allocateRequest1 =
+ AllocateRequest.newInstance(0, 0F, null, null, null);
+ AllocateResponse response1 =
+ allocate(attempt1.getAppAttemptId(), allocateRequest1);
+ List<NodeReport> updatedNodes = response1.getUpdatedNodes();
+ Assert.assertEquals(0, updatedNodes.size());
Review Comment:
@aajisaka - Updated the test case with `updatedNodes with 1 `and its passing - checked in my local after [YARN-6483](https://issues.apache.org/jira/browse/YARN-6483) is backport to branch-2.10
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ashutoshcipher commented on a diff in pull request #4244: YARN-11119. Backport YARN-10538 to branch-2.10
Posted by GitBox <gi...@apache.org>.
ashutoshcipher commented on code in PR #4244:
URL: https://github.com/apache/hadoop/pull/4244#discussion_r906888151
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRMRPCNodeUpdates.java:
##########
@@ -139,6 +146,48 @@ public void testAMRMDecommissioningNodes() throws Exception {
NodeUpdateType.NODE_DECOMMISSIONING, nr.getNodeUpdateType());
}
+ @Test
+ public void testAMRMRecommissioningNodes() throws Exception {
+ MockNM nm1 = rm.registerNode("127.0.0.1:1234", 10000);
+ MockNM nm2 = rm.registerNode("127.0.0.2:1234", 10000);
+ rm.drainEvents();
+
+ RMApp app1 = rm.submitApp(2000);
+
+ // Trigger the scheduling so the AM gets 'launched' on nm1
+ nm1.nodeHeartbeat(true);
+
+ RMAppAttempt attempt1 = app1.getCurrentAppAttempt();
+ MockAM am1 = rm.sendAMLaunched(attempt1.getAppAttemptId());
+
+ // register AM returns no unusable node
+ am1.registerAppAttempt();
+
+ // DECOMMISSION nm2
+ Integer decommissioningTimeout = 600;
+ syncNodeGracefulDecommission(nm2, decommissioningTimeout);
+
+ AllocateRequest allocateRequest1 =
+ AllocateRequest.newInstance(0, 0F, null, null, null);
+ AllocateResponse response1 =
+ allocate(attempt1.getAppAttemptId(), allocateRequest1);
+ List<NodeReport> updatedNodes = response1.getUpdatedNodes();
+ Assert.assertEquals(1, updatedNodes.size());
+
Review Comment:
Thanks, Updated the test case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4244: YARN-11119. Backport YARN-10538 to branch-2.10
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4244:
URL: https://github.com/apache/hadoop/pull/4244#issuecomment-1166722215
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 53s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ branch-2.10 Compile Tests _ |
| +1 :green_heart: | mvninstall | 16m 30s | | branch-2.10 passed |
| +1 :green_heart: | compile | 1m 12s | | branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10 |
| +1 :green_heart: | compile | 0m 50s | | branch-2.10 passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| +1 :green_heart: | checkstyle | 0m 43s | | branch-2.10 passed |
| +1 :green_heart: | mvnsite | 1m 3s | | branch-2.10 passed |
| +1 :green_heart: | javadoc | 0m 53s | | branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10 |
| +1 :green_heart: | javadoc | 0m 41s | | branch-2.10 passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| +1 :green_heart: | spotbugs | 2m 6s | | branch-2.10 passed |
| -0 :warning: | patch | 2m 28s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 45s | | the patch passed |
| +1 :green_heart: | compile | 0m 51s | | the patch passed with JDK Azul Systems, Inc.-1.7.0_262-b10 |
| +1 :green_heart: | javac | 0m 51s | | the patch passed |
| +1 :green_heart: | compile | 0m 43s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| +1 :green_heart: | javac | 0m 43s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 29s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 47s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 37s | | the patch passed with JDK Azul Systems, Inc.-1.7.0_262-b10 |
| +1 :green_heart: | javadoc | 0m 31s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| +1 :green_heart: | spotbugs | 1m 41s | | the patch passed |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 68m 42s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not generate ASF License warnings. |
| | | 104m 26s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4244/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4244 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 2140dda26254 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-2.10 / 68303792d305ba3f04bb97e8d86088a5f6512c6c |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_262-b10 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4244/5/testReport/ |
| Max. process+thread count | 902 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4244/5/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ashutoshcipher commented on a diff in pull request #4244: YARN-11119. Backport YARN-10538 to branch-2.10
Posted by GitBox <gi...@apache.org>.
ashutoshcipher commented on code in PR #4244:
URL: https://github.com/apache/hadoop/pull/4244#discussion_r869795198
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRMRPCNodeUpdates.java:
##########
@@ -98,6 +111,47 @@ public AllocateResponse run() throws Exception {
});
}
+ @Test
+ public void testAMRMRecommissioningNodes() throws Exception {
+ MockNM nm1 = rm.registerNode("127.0.0.1:1234", 10000);
+ MockNM nm2 = rm.registerNode("127.0.0.2:1234", 10000);
+ rm.drainEvents();
+
+ RMApp app1 = rm.submitApp(2000);
+
+ // Trigger the scheduling so the AM gets 'launched' on nm1
+ nm1.nodeHeartbeat(true);
+
+ RMAppAttempt attempt1 = app1.getCurrentAppAttempt();
+ MockAM am1 = rm.sendAMLaunched(attempt1.getAppAttemptId());
+
+ // register AM returns no unusable node
+ am1.registerAppAttempt();
+
+ // DECOMMISSION nm2
+ syncNodeGracefulDecommission(nm2);
+
+ AllocateRequest allocateRequest1 =
+ AllocateRequest.newInstance(0, 0F, null, null, null);
+ AllocateResponse response1 =
+ allocate(attempt1.getAppAttemptId(), allocateRequest1);
+ List<NodeReport> updatedNodes = response1.getUpdatedNodes();
+ Assert.assertEquals(0, updatedNodes.size());
Review Comment:
Raised PR for [YARN-6483](https://issues.apache.org/jira/browse/YARN-6483) - PR https://github.com/apache/hadoop/pull/4258
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ashutoshcipher commented on a diff in pull request #4244: YARN-11119. Backport YARN-10538 to branch-2.10
Posted by GitBox <gi...@apache.org>.
ashutoshcipher commented on code in PR #4244:
URL: https://github.com/apache/hadoop/pull/4244#discussion_r863294688
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRMRPCNodeUpdates.java:
##########
@@ -98,6 +111,47 @@ public AllocateResponse run() throws Exception {
});
}
+ @Test
+ public void testAMRMRecommissioningNodes() throws Exception {
+ MockNM nm1 = rm.registerNode("127.0.0.1:1234", 10000);
+ MockNM nm2 = rm.registerNode("127.0.0.2:1234", 10000);
+ rm.drainEvents();
+
+ RMApp app1 = rm.submitApp(2000);
+
+ // Trigger the scheduling so the AM gets 'launched' on nm1
+ nm1.nodeHeartbeat(true);
+
+ RMAppAttempt attempt1 = app1.getCurrentAppAttempt();
+ MockAM am1 = rm.sendAMLaunched(attempt1.getAppAttemptId());
+
+ // register AM returns no unusable node
+ am1.registerAppAttempt();
+
+ // DECOMMISSION nm2
+ syncNodeGracefulDecommission(nm2);
+
+ AllocateRequest allocateRequest1 =
+ AllocateRequest.newInstance(0, 0F, null, null, null);
+ AllocateResponse response1 =
+ allocate(attempt1.getAppAttemptId(), allocateRequest1);
+ List<NodeReport> updatedNodes = response1.getUpdatedNodes();
+ Assert.assertEquals(0, updatedNodes.size());
Review Comment:
Yeah @aajisaka - this makes sense. Let's try back porting [YARN-6483](https://issues.apache.org/jira/browse/YARN-6483) and see if there aren't much dependencies to be backported along with it and we can successfully do that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4244: YARN-11119. Backport YARN-10538 to branch-2.10
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4244:
URL: https://github.com/apache/hadoop/pull/4244#issuecomment-1112471513
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 12m 31s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ branch-2.10 Compile Tests _ |
| +1 :green_heart: | mvninstall | 16m 52s | | branch-2.10 passed |
| +1 :green_heart: | compile | 1m 3s | | branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10 |
| +1 :green_heart: | compile | 0m 55s | | branch-2.10 passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| +1 :green_heart: | checkstyle | 0m 45s | | branch-2.10 passed |
| +1 :green_heart: | mvnsite | 1m 3s | | branch-2.10 passed |
| +1 :green_heart: | javadoc | 0m 49s | | branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10 |
| +1 :green_heart: | javadoc | 0m 40s | | branch-2.10 passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| +1 :green_heart: | spotbugs | 2m 7s | | branch-2.10 passed |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 46s | | the patch passed |
| +1 :green_heart: | compile | 0m 50s | | the patch passed with JDK Azul Systems, Inc.-1.7.0_262-b10 |
| +1 :green_heart: | javac | 0m 50s | | the patch passed |
| +1 :green_heart: | compile | 0m 42s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| +1 :green_heart: | javac | 0m 42s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 29s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 47s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 36s | | the patch passed with JDK Azul Systems, Inc.-1.7.0_262-b10 |
| +1 :green_heart: | javadoc | 0m 30s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| +1 :green_heart: | spotbugs | 1m 40s | | the patch passed |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 68m 26s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not generate ASF License warnings. |
| | | 116m 21s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4244/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4244 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux a359ebc4d0c6 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-2.10 / ebc95e037b7450b6d229ec2c6d4d8a585fee8228 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_262-b10 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4244/1/testReport/ |
| Max. process+thread count | 902 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4244/1/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on a diff in pull request #4244: YARN-11119. Backport YARN-10538 to branch-2.10
Posted by GitBox <gi...@apache.org>.
aajisaka commented on code in PR #4244:
URL: https://github.com/apache/hadoop/pull/4244#discussion_r863034596
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRMRPCNodeUpdates.java:
##########
@@ -98,6 +111,47 @@ public AllocateResponse run() throws Exception {
});
}
+ @Test
+ public void testAMRMRecommissioningNodes() throws Exception {
+ MockNM nm1 = rm.registerNode("127.0.0.1:1234", 10000);
+ MockNM nm2 = rm.registerNode("127.0.0.2:1234", 10000);
+ rm.drainEvents();
+
+ RMApp app1 = rm.submitApp(2000);
+
+ // Trigger the scheduling so the AM gets 'launched' on nm1
+ nm1.nodeHeartbeat(true);
+
+ RMAppAttempt attempt1 = app1.getCurrentAppAttempt();
+ MockAM am1 = rm.sendAMLaunched(attempt1.getAppAttemptId());
+
+ // register AM returns no unusable node
+ am1.registerAppAttempt();
+
+ // DECOMMISSION nm2
+ syncNodeGracefulDecommission(nm2);
+
+ AllocateRequest allocateRequest1 =
+ AllocateRequest.newInstance(0, 0F, null, null, null);
+ AllocateResponse response1 =
+ allocate(attempt1.getAppAttemptId(), allocateRequest1);
+ List<NodeReport> updatedNodes = response1.getUpdatedNodes();
+ Assert.assertEquals(0, updatedNodes.size());
Review Comment:
I noticed the size of `updatedNodes` is 1 in #2564. It is because YARN-6483 is not in branch-2.10.
Now I'm thinking we need to backport YARN-6483 before YARN-10538. @ashutoshcipher what do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4244: YARN-11119. Backport YARN-10538 to branch-2.10
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4244:
URL: https://github.com/apache/hadoop/pull/4244#issuecomment-1166666886
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 12m 21s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ branch-2.10 Compile Tests _ |
| +1 :green_heart: | mvninstall | 16m 31s | | branch-2.10 passed |
| +1 :green_heart: | compile | 1m 1s | | branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10 |
| +1 :green_heart: | compile | 0m 53s | | branch-2.10 passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| +1 :green_heart: | checkstyle | 0m 46s | | branch-2.10 passed |
| +1 :green_heart: | mvnsite | 1m 3s | | branch-2.10 passed |
| +1 :green_heart: | javadoc | 0m 54s | | branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10 |
| +1 :green_heart: | javadoc | 0m 41s | | branch-2.10 passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| +1 :green_heart: | spotbugs | 2m 8s | | branch-2.10 passed |
| -0 :warning: | patch | 2m 30s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 47s | | the patch passed |
| +1 :green_heart: | compile | 0m 49s | | the patch passed with JDK Azul Systems, Inc.-1.7.0_262-b10 |
| +1 :green_heart: | javac | 0m 49s | | the patch passed |
| +1 :green_heart: | compile | 0m 43s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| +1 :green_heart: | javac | 0m 43s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 30s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 48s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 36s | | the patch passed with JDK Azul Systems, Inc.-1.7.0_262-b10 |
| +1 :green_heart: | javadoc | 0m 31s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| +1 :green_heart: | spotbugs | 1m 38s | | the patch passed |
|||| _ Other Tests _ |
| -1 :x: | unit | 68m 50s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4244/3/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not generate ASF License warnings. |
| | | 116m 15s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4244/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4244 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 4e5c4405262f 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-2.10 / c19ccc4c7696918ecfc692f68e4d6c9a06ea1d89 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_262-b10 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4244/3/testReport/ |
| Max. process+thread count | 888 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4244/3/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] aajisaka merged pull request #4244: YARN-11119. Backport YARN-10538 to branch-2.10
Posted by GitBox <gi...@apache.org>.
aajisaka merged PR #4244:
URL: https://github.com/apache/hadoop/pull/4244
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org