You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2022/12/22 23:43:47 UTC
[GitHub] [hadoop] lnbest0707 opened a new pull request, #5254: Yarn-11391 Add yarn RM DNS support
lnbest0707 opened a new pull request, #5254:
URL: https://github.com/apache/hadoop/pull/5254
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g., 'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
The patch reuses the resolver introduced from the HDFS side and apply similar logic to YARN RM service addresses resolving.
To utilize the YARN DNS support and use DNS as endpoint, simply upgrade the hadoop binary and revise configs from, for example:
<property>
<name>yarn.resourcemanager.address.rm1</name>
<value>rm1_address:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address.rm1</name>
<value>rm1_address:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address.rm1</name>
<value>rm1_address:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address.rm1</name>
<value>rm1_address:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm1</name>
<value>rm1_address:8088</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.https.address.rm1</name>
<value>rm1_address:8090</value>
</property>
<property>
<name>yarn.resourcemanager.address.rm2</name>
<value>rm2_address:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address.rm2</name>
<value>rm2_address:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address.rm2</name>
<value>rm2_address:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address.rm2</name>
<value>rm2_address:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm2</name>
<value>rm2_address:8088</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.https.address.rm2</name>
<value>rm2_address:8090</value>
</property>
<property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rm1,rm2</value>
</property>
to:
<property>
<name>yarn.resourcemanager.address.rm1</name>
<value>rm_multi_a_dns:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address.rm1</name>
<value>rm_multi_a_dns:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address.rm1</name>
<value>rm_multi_a_dns:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address.rm1</name>
<value>rm_multi_a_dns:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm1</name>
<value>rm_multi_a_dns:8088</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.https.address.rm1</name>
<value>rm_multi_a_dns:8090</value>
</property>
<property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rm1</value>
</property>
<property>
<name>yarn.resourcemanager.ha.resolve-needed</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.ha.resolver.useFQDN</name>
<value>true</value> # required in secure mode
</property>
<property>
<name>yarn.resourcemanager.ha.refresh-period-ms</name>
<value>180000</value> # 3 min
</property>
where rm_multi_a_dns is a multi-A DNS record for rm1_address and rm2_address. This means the following output on the terminal.
$ dig +short <rm_multi_a_dns> | xargs -n +1 dig +short -x | sort
<rm1_address>
<rm2_address>
For the newly introduced flags, please refer to yarn-default.xml.
### How was this patch tested?
### For code changes:
- [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5254: Yarn-11391 Add yarn RM DNS support
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #5254:
URL: https://github.com/apache/hadoop/pull/5254#issuecomment-1363584299
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 46s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 19s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 5s | | trunk passed |
| +1 :green_heart: | compile | 23m 11s | | trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 20m 24s | | trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 3m 50s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 48s | | trunk passed |
| -1 :x: | javadoc | 1m 16s | [/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5254/1/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt) | hadoop-common in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 2m 36s | | trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 6m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 31s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 17s | | the patch passed |
| +1 :green_heart: | compile | 22m 13s | | the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 22m 13s | | the patch passed |
| +1 :green_heart: | compile | 20m 35s | | the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 20m 35s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 3m 40s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5254/1/artifact/out/results-checkstyle-root.txt) | root: The patch generated 2 new + 184 unchanged - 0 fixed = 186 total (was 184) |
| +1 :green_heart: | mvnsite | 3m 52s | | the patch passed |
| -1 :x: | javadoc | 1m 5s | [/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5254/1/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt) | hadoop-common in the patch failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 2m 39s | | the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| -1 :x: | spotbugs | 2m 4s | [/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5254/1/artifact/out/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.html) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | shadedclient | 21m 41s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 18m 16s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 1m 16s | | hadoop-yarn-api in the patch passed. |
| +1 :green_heart: | unit | 5m 44s | | hadoop-yarn-common in the patch passed. |
| +1 :green_heart: | asflicense | 1m 2s | | The patch does not generate ASF License warnings. |
| | | 244m 14s | | |
| Reason | Tests |
|-------:|:------|
| SpotBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| | Inconsistent synchronization of org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider.rmServiceIds; locked 90% of time Unsynchronized access at RequestHedgingRMFailoverProxyProvider.java:90% of time Unsynchronized access at RequestHedgingRMFailoverProxyProvider.java:[line 147] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5254/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5254 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 0b852de2bbac 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e1d2a74aebc27acf70543e0d88e0208fd82cfa69 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5254/1/testReport/ |
| Max. process+thread count | 1988 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5254/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5254: Yarn-11391 Add yarn RM DNS support
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #5254:
URL: https://github.com/apache/hadoop/pull/5254#issuecomment-1370329102
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 12m 37s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 54s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 46s | | trunk passed |
| +1 :green_heart: | compile | 23m 59s | | trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 21m 44s | | trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 3m 40s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 30s | | trunk passed |
| -1 :x: | javadoc | 1m 9s | [/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5254/2/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt) | hadoop-common in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 2m 25s | | trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 7m 12s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 37s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 26s | | the patch passed |
| +1 :green_heart: | compile | 24m 12s | | the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 24m 12s | | the patch passed |
| +1 :green_heart: | compile | 21m 24s | | the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 21m 24s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 10s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5254/2/artifact/out/results-checkstyle-root.txt) | root: The patch generated 2 new + 184 unchanged - 0 fixed = 186 total (was 184) |
| +1 :green_heart: | mvnsite | 3m 28s | | the patch passed |
| -1 :x: | javadoc | 1m 2s | [/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5254/2/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt) | hadoop-common in the patch failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 2m 34s | | the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| -1 :x: | spotbugs | 2m 5s | [/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5254/2/artifact/out/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.html) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | shadedclient | 23m 25s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 18m 16s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 1m 12s | | hadoop-yarn-api in the patch passed. |
| +1 :green_heart: | unit | 5m 34s | | hadoop-yarn-common in the patch passed. |
| +1 :green_heart: | asflicense | 0m 53s | | The patch does not generate ASF License warnings. |
| | | 262m 32s | | |
| Reason | Tests |
|-------:|:------|
| SpotBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| | Inconsistent synchronization of org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider.rmServiceIds; locked 90% of time Unsynchronized access at RequestHedgingRMFailoverProxyProvider.java:90% of time Unsynchronized access at RequestHedgingRMFailoverProxyProvider.java:[line 147] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5254/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5254 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 42244b06485a 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 23002be02232c88745f77e9e551dfff5500b3dee |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5254/2/testReport/ |
| Max. process+thread count | 1291 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5254/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] lnbest0707 closed pull request #5254: Yarn-11391 Add yarn RM DNS support
Posted by GitBox <gi...@apache.org>.
lnbest0707 closed pull request #5254: Yarn-11391 Add yarn RM DNS support
URL: https://github.com/apache/hadoop/pull/5254
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org