You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/06/26 07:23:16 UTC
[GitHub] [spark] PavithraRamachandran opened a new pull request #28931: [SPARK-32103] [yarn] Handle host/port split in IPV6 in YarnRMClient
PavithraRamachandran opened a new pull request #28931:
URL: https://github.com/apache/spark/pull/28931
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
In IPv6 scenario, the current logic to split hostname and port is not correct.
### Why are the changes needed?
to support IPV6 deployment scenario
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
UT and IPV6 spark deployment with yarn
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-653936145
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #28931:
URL: https://github.com/apache/spark/pull/28931#discussion_r451632013
##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -1046,18 +1055,30 @@ private[spark] object Utils extends Logging {
return cached
}
- val indx: Int = hostPort.lastIndexOf(':')
- // This is potentially broken - when dealing with ipv6 addresses for example, sigh ...
- // but then hadoop does not support ipv6 right now.
- // For now, we assume that if port exists, then it is valid - not check if it is an int > 0
- if (-1 == indx) {
+ def setDefaultPortValue: (String, Int) = {
val retval = (hostPort, 0)
hostPortParseResults.put(hostPort, retval)
- return retval
+ retval
+ }
+
+ if (hostPort != null && hostPort.split(":").length > 2) {
Review comment:
Since the new changed logic (line 1064 ~ 1081), could you add a comment before this line?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-653932298
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-656663258
**[Test build #125609 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125609/testReport)** for PR 28931 at commit [`768e7c4`](https://github.com/apache/spark/commit/768e7c46bff5b5df76efa52dc30d432b45bb6f16).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-656663747
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-653118357
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28931: [SPARK-32103] Handle host/port split in IPV6 in YarnRMClient
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-650025864
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-655585627
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28931: [SPARK-32103] Handle host/port split in IPV6 in YarnRMClient
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-650480608
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124558/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] PavithraRamachandran commented on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
PavithraRamachandran commented on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-652342616
@dongjoon-hyun thank you for identifying the missed scenario, i shall handle it and update the PR
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28931: [SPARK-32103] Handle host/port split in IPV6 in YarnRMClient
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-650480574
**[Test build #124558 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124558/testReport)** for PR 28931 at commit [`3e2eb3a`](https://github.com/apache/spark/commit/3e2eb3a11ee993605ca94779e4b8642a9cad2f3f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-653221538
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28931: [SPARK-32103] Handle host/port split in IPV6 in YarnRMClient
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-650025864
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-653931939
**[Test build #124942 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124942/testReport)** for PR 28931 at commit [`becec4e`](https://github.com/apache/spark/commit/becec4e58158361a7edcda61f28ab4bbd73a18b3).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #28931:
URL: https://github.com/apache/spark/pull/28931#discussion_r451635049
##########
File path: core/src/test/scala/org/apache/spark/util/UtilsSuite.scala
##########
@@ -1309,6 +1309,97 @@ class UtilsSuite extends SparkFunSuite with ResetSystemProperties with Logging {
assert(Utils.buildLocationMetadata(paths, 15) == "[path0, path1, path2]")
assert(Utils.buildLocationMetadata(paths, 25) == "[path0, path1, path2, path3]")
}
+
+ test("checkHosts support IPV6 and IPV4") {
+ // IPV4 ips
+ Utils.checkHost("0.0.0.0")
+ intercept[AssertionError] {
+ Utils.checkHost("0.0.0.0:0")
+ }
+ intercept[AssertionError] {
+ Utils.checkHost("0.0.0.0:")
+ }
+
+ // IPV6 ips
+ Utils.checkHost("[::1]")
+ intercept[AssertionError] {
+ Utils.checkHost("[::1]:0")
+ }
+ intercept[AssertionError] {
+ Utils.checkHost("[::1]:")
+ }
+
+ // hostname
+ Utils.checkHost("localhost")
+ intercept[AssertionError] {
+ Utils.checkHost("localhost:0")
+ }
+ intercept[AssertionError] {
+ Utils.checkHost("localhost:")
+ }
+ }
+
+ test("checkHostPort support IPV6 and IPV4") {
+ // IPV4 ips
+ Utils.checkHostPort("0.0.0.0:0")
+ intercept[AssertionError] {
+ Utils.checkHostPort("0.0.0.0")
+ }
+
+ // IPV6 ips
+ Utils.checkHostPort("[::1]:0")
+ intercept[AssertionError] {
+ Utils.checkHostPort("[::1]")
+ }
+
+ // hostname
+ Utils.checkHostPort("localhost:0")
+ intercept[AssertionError] {
+ Utils.checkHostPort("localhost")
+ }
+ }
+
+ test("parseHostPort support IPV6 and IPV4") {
+ // IPV4 ips
+ var hostnamePort = Utils.parseHostPort("0.0.0.0:80")
+ assert(hostnamePort._1.equals("0.0.0.0"))
+ assert(hostnamePort._2 === 80)
+
+ hostnamePort = Utils.parseHostPort("0.0.0.0")
+ assert(hostnamePort._1.equals("0.0.0.0"))
+ assert(hostnamePort._2 === 0)
+
+ hostnamePort = Utils.parseHostPort("0.0.0.0:")
+ assert(hostnamePort._1.equals("0.0.0.0"))
+ assert(hostnamePort._2 === 0)
+
+
Review comment:
nit. repeated blank lines.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-656853891
**[Test build #125609 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125609/testReport)** for PR 28931 at commit [`768e7c4`](https://github.com/apache/spark/commit/768e7c46bff5b5df76efa52dc30d432b45bb6f16).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-655792102
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28931: [SPARK-32103] Handle host/port split in IPV6 in YarnRMClient
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-650478156
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28931: [SPARK-32103] Handle host/port split in IPV6 in YarnRMClient
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-650478040
**[Test build #124558 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124558/testReport)** for PR 28931 at commit [`3e2eb3a`](https://github.com/apache/spark/commit/3e2eb3a11ee993605ca94779e4b8642a9cad2f3f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-653118357
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-655792112
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125379/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28931: [SPARK-32103] Handle host/port split in IPV6 in YarnRMClient
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-650478156
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-653936114
**[Test build #124942 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124942/testReport)** for PR 28931 at commit [`becec4e`](https://github.com/apache/spark/commit/becec4e58158361a7edcda61f28ab4bbd73a18b3).
* This patch **fails to generate documentation**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] PavithraRamachandran commented on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
PavithraRamachandran commented on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-653555033
@dongjoon-hyun i am getting 502 proxy error when i try to access the CI to check for failures. Could u help me?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-655566387
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-653117779
**[Test build #124920 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124920/testReport)** for PR 28931 at commit [`becec4e`](https://github.com/apache/spark/commit/becec4e58158361a7edcda61f28ab4bbd73a18b3).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-655587771
**[Test build #125379 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125379/testReport)** for PR 28931 at commit [`4c9cb2e`](https://github.com/apache/spark/commit/4c9cb2e5cbc00a31f905102f1765f1a421a5ef01).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-653113732
**[Test build #124919 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124919/testReport)** for PR 28931 at commit [`98a737b`](https://github.com/apache/spark/commit/98a737bbbde84cdbbb19fab8c254ae36607a0e7a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #28931: [SPARK-32103][CORE] Support IPv6 host/port in core module
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-658553220
Hi, @gatorsmile . Technically, this only handles `host/port` parsing inside `core` module. I'm sure that this is a meaningful step inside Spark. However, we didn't test anything on IPv6. Like what we did for JDK11, I expect lots of hurdle both inside and outside Spark.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #28931: [SPARK-32103][CORE] Support IPv6 host/port in core module
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-658553220
Hi, @gatorsmile . Technically, this only handles `host/port` parsing inside `core` module. I'm sure that this is a meaningful step inside Spark. However, we didn't test anything on IPv6. Like JDK11, I expects lots of hurdle both inside and outside Spark.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-655588411
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28931: [SPARK-32103] Handle host/port split in IPV6 in YarnRMClient
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-650480605
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-656856757
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28931: [SPARK-32103] Handle host/port split in IPV6 in YarnRMClient
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-650480605
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-653113732
**[Test build #124919 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124919/testReport)** for PR 28931 at commit [`98a737b`](https://github.com/apache/spark/commit/98a737bbbde84cdbbb19fab8c254ae36607a0e7a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-655574967
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-653221941
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124920/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28931: [SPARK-32103] Handle host/port split in IPV6 in YarnRMClient
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-650026318
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-655588411
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #28931:
URL: https://github.com/apache/spark/pull/28931#discussion_r451631122
##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -1027,12 +1027,21 @@ private[spark] object Utils extends Logging {
}
def checkHost(host: String): Unit = {
Review comment:
This function semantic is changed to handle `IP`. Could you add a function description to explain it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-653936145
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-653114709
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-653931939
**[Test build #124942 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124942/testReport)** for PR 28931 at commit [`becec4e`](https://github.com/apache/spark/commit/becec4e58158361a7edcda61f28ab4bbd73a18b3).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #28931: [SPARK-32103][CORE] Support IPv6 host/port in core module
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #28931:
URL: https://github.com/apache/spark/pull/28931
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] PavithraRamachandran commented on a change in pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
PavithraRamachandran commented on a change in pull request #28931:
URL: https://github.com/apache/spark/pull/28931#discussion_r448274758
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnRMClient.scala
##########
@@ -107,7 +107,7 @@ private[spark] class YarnRMClient extends Logging {
// so not all stable releases have it.
val prefix = WebAppUtils.getHttpSchemePrefix(conf)
val proxies = WebAppUtils.getProxyHostsAndPortsForAmFilter(conf)
- val hosts = proxies.asScala.map(_.split(":").head)
+ val hosts = proxies.asScala.map(proxy => proxy.splitAt(proxy.lastIndexOf(":"))._1)
Review comment:
thank you for identifying this , i shall handle such scenarios too and correct it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] PavithraRamachandran commented on a change in pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
PavithraRamachandran commented on a change in pull request #28931:
URL: https://github.com/apache/spark/pull/28931#discussion_r448274758
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnRMClient.scala
##########
@@ -107,7 +107,7 @@ private[spark] class YarnRMClient extends Logging {
// so not all stable releases have it.
val prefix = WebAppUtils.getHttpSchemePrefix(conf)
val proxies = WebAppUtils.getProxyHostsAndPortsForAmFilter(conf)
- val hosts = proxies.asScala.map(_.split(":").head)
+ val hosts = proxies.asScala.map(proxy => proxy.splitAt(proxy.lastIndexOf(":"))._1)
Review comment:
I have identified few more places where IPV6 is not handled, i shall update my PR soon
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-655791528
**[Test build #125379 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125379/testReport)** for PR 28931 at commit [`4c9cb2e`](https://github.com/apache/spark/commit/4c9cb2e5cbc00a31f905102f1765f1a421a5ef01).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-653221538
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-655574978
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125377/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28931: [SPARK-32103] Handle host/port split in IPV6 in YarnRMClient
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-650477700
ok to test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-653221941
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124920/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-653209779
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-655792102
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-653209783
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124919/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-653207690
Retest this please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-655574967
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-653629276
GitHub Action CI is okay, but UCB AmbLab Jenkins CI seems to be completely down now. In this case, there is no way for us to do. Let's wait for a while.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #28931:
URL: https://github.com/apache/spark/pull/28931#discussion_r451634654
##########
File path: core/src/test/scala/org/apache/spark/util/UtilsSuite.scala
##########
@@ -1309,6 +1309,97 @@ class UtilsSuite extends SparkFunSuite with ResetSystemProperties with Logging {
assert(Utils.buildLocationMetadata(paths, 15) == "[path0, path1, path2]")
assert(Utils.buildLocationMetadata(paths, 25) == "[path0, path1, path2, path3]")
}
+
+ test("checkHosts support IPV6 and IPV4") {
Review comment:
- `checkHosts` -> `checkHost`
- `support` -> `supports`
- `IPV6 and IPV4` -> `both IPv4 and IPv6`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28931: [SPARK-32103] Handle host/port split in IPV6 in YarnRMClient
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #28931:
URL: https://github.com/apache/spark/pull/28931#discussion_r446475568
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnRMClient.scala
##########
@@ -107,7 +107,7 @@ private[spark] class YarnRMClient extends Logging {
// so not all stable releases have it.
val prefix = WebAppUtils.getHttpSchemePrefix(conf)
val proxies = WebAppUtils.getProxyHostsAndPortsForAmFilter(conf)
- val hosts = proxies.asScala.map(_.split(":").head)
+ val hosts = proxies.asScala.map(proxy => proxy.splitAt(proxy.lastIndexOf(":"))._1)
Review comment:
Hi, @PavithraRamachandran . If possible, could you check the code base if this is the last instance or not?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-653932298
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-653206625
**[Test build #124920 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124920/testReport)** for PR 28931 at commit [`becec4e`](https://github.com/apache/spark/commit/becec4e58158361a7edcda61f28ab4bbd73a18b3).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28931: [SPARK-32103] Handle host/port split in IPV6 in YarnRMClient
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-650478040
**[Test build #124558 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124558/testReport)** for PR 28931 at commit [`3e2eb3a`](https://github.com/apache/spark/commit/3e2eb3a11ee993605ca94779e4b8642a9cad2f3f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-656663258
**[Test build #125609 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125609/testReport)** for PR 28931 at commit [`768e7c4`](https://github.com/apache/spark/commit/768e7c46bff5b5df76efa52dc30d432b45bb6f16).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28931: [SPARK-32103] Handle host/port split in IPV6 in YarnRMClient
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-650026318
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-653936147
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124942/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #28931:
URL: https://github.com/apache/spark/pull/28931#discussion_r451633470
##########
File path: core/src/test/scala/org/apache/spark/util/UtilsSuite.scala
##########
@@ -1309,6 +1309,97 @@ class UtilsSuite extends SparkFunSuite with ResetSystemProperties with Logging {
assert(Utils.buildLocationMetadata(paths, 15) == "[path0, path1, path2]")
assert(Utils.buildLocationMetadata(paths, 25) == "[path0, path1, path2, path3]")
}
+
+ test("checkHosts support IPV6 and IPV4") {
+ // IPV4 ips
+ Utils.checkHost("0.0.0.0")
+ intercept[AssertionError] {
+ Utils.checkHost("0.0.0.0:0")
+ }
Review comment:
In this case, it would be great if we check the error message to prevent future regression.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-653117779
**[Test build #124920 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124920/testReport)** for PR 28931 at commit [`becec4e`](https://github.com/apache/spark/commit/becec4e58158361a7edcda61f28ab4bbd73a18b3).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] PavithraRamachandran commented on pull request #28931: [SPARK-32103][CORE] Support IPv6 host/port in core module
Posted by GitBox <gi...@apache.org>.
PavithraRamachandran commented on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-657359754
thank u for merging @dongjoon-hyun
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-655587771
**[Test build #125379 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125379/testReport)** for PR 28931 at commit [`4c9cb2e`](https://github.com/apache/spark/commit/4c9cb2e5cbc00a31f905102f1765f1a421a5ef01).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-653114709
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-653195895
**[Test build #124919 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124919/testReport)** for PR 28931 at commit [`98a737b`](https://github.com/apache/spark/commit/98a737bbbde84cdbbb19fab8c254ae36607a0e7a).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28931: [SPARK-32103][CORE] Support IPv6 host/port in core module
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-658553220
Hi, @gatorsmile . Technically, this only handles `host/port` parsing inside `core` module only. I'm sure that this is a meaningful step inside Spark. However, we didn't test anything on IPv6. Like JDK11, I expects lots of hurdle both inside and outside Spark.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-653209779
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-656663747
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-655566387
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gatorsmile commented on pull request #28931: [SPARK-32103][CORE] Support IPv6 host/port in core module
Posted by GitBox <gi...@apache.org>.
gatorsmile commented on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-658503381
Is it the only blocker for IPV6 support?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-655792112
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125379/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-656856757
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28931: [SPARK-32103][YARN] Handle IPv6 host/port split in YarnRMClient
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #28931:
URL: https://github.com/apache/spark/pull/28931#issuecomment-655585975
cc @dbtsai
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org