You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/06/21 04:29:55 UTC
[GitHub] [spark] rajatahujaatinmobi opened a new pull request #28880: [SPARK-32039][YARN][WEBUI][master] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
rajatahujaatinmobi opened a new pull request #28880:
URL: https://github.com/apache/spark/pull/28880
What changes were proposed in this pull request?
When a Spark Job launched in Cluster mode with Yarn, Application Master sets spark.ui.port port to 0 which means Driver's web UI gets any random port even if we want to explicitly set the Port range for Driver's Web UI
Why are the changes needed?
We access Spark Web UI via Knox Proxy, and there are firewall restrictions due to which we can not access Spark Web UI since Web UI port range gets random port even if we set explicitly.
This Change will check if there is a specified port range explicitly mentioned so that it does not assign a random port.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Local Tested.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-651977705
**[Test build #124685 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124685/testReport)** for PR 28880 at commit [`24ecfcf`](https://github.com/apache/spark/commit/24ecfcf1c06a4af8f15ce28e6cd9b559bba7643b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-647200351
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rajatahujaatinmobi commented on a change in pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
rajatahujaatinmobi commented on a change in pull request #28880:
URL: https://github.com/apache/spark/pull/28880#discussion_r443291133
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
##########
@@ -211,9 +211,11 @@ private[spark] class ApplicationMaster(
final def run(): Int = {
try {
val attemptID = if (isClusterMode) {
- // Set the web ui port to be ephemeral for yarn so we don't conflict with
- // other spark processes running on the same box
- System.setProperty(UI_PORT.key, "0")
+ // Set the web ui port to be ephemeral for yarn if not set explicitly
+ // so we don't conflict with other spark processes running on the same box
+ if (System.getProperty(UI_PORT.key) != null) {
Review comment:
Yes, changing the If condition.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-650505296
**[Test build #124562 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124562/testReport)** for PR 28880 at commit [`8fe0aa9`](https://github.com/apache/spark/commit/8fe0aa98a18edb579b7ee214b6c6dd9ba69b395b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28880: [SPARK-32039][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-647198022
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28880: [SPARK-32039][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-647198022
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-651989946
**[Test build #124687 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124687/testReport)** for PR 28880 at commit [`1896905`](https://github.com/apache/spark/commit/1896905d42ee71709fa0da0a80d84fe3eb026c6e).
* This patch **fails to generate documentation**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28880: [SPARK-32039][YARN][WEBUI][master] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-647197657
ok to test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rajatahujaatinmobi commented on a change in pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
rajatahujaatinmobi commented on a change in pull request #28880:
URL: https://github.com/apache/spark/pull/28880#discussion_r444552932
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
##########
@@ -211,9 +211,11 @@ private[spark] class ApplicationMaster(
final def run(): Int = {
try {
val attemptID = if (isClusterMode) {
- // Set the web ui port to be ephemeral for yarn so we don't conflict with
- // other spark processes running on the same box
- System.setProperty(UI_PORT.key, "0")
+ // Set the web ui port to be ephemeral for yarn if not set explicitly
+ // so we don't conflict with other spark processes running on the same box
+ if (System.getProperty(UI_PORT.key) != null) {
+ System.setProperty(UI_PORT.key, "0")
+ }
Review comment:
I blocked a set of Port range from 18000 to 18050 and configure web or to start at 18008 but it eventually Started at 18051
**Properties Set: spark.ui.port=18018 spark.port.maxRetries=60**
Successful Case:
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port 18018. Attempting port 18019.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port 18019. Attempting port 18020.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port 18020. Attempting port 18021.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port 18021. Attempting port 18022.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port 18022. Attempting port 18023.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port 18023. Attempting port 18024.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port 18024. Attempting port 18025.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port 18025. Attempting port 18026.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port 18026. Attempting port 18027.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port 18027. Attempting port 18028.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port 18028. Attempting port 18029.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port 18029. Attempting port 18030.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port 18030. Attempting port 18031.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port 18031. Attempting port 18032.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port 18032. Attempting port 18033.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port 18033. Attempting port 18034.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port 18034. Attempting port 18035.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port 18035. Attempting port 18036.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port 18036. Attempting port 18037.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port 18037. Attempting port 18038.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port 18038. Attempting port 18039.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port 18039. Attempting port 18040.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port 18040. Attempting port 18041.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port 18041. Attempting port 18042.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port 18042. Attempting port 18043.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port 18043. Attempting port 18044.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port 18044. Attempting port 18045.
20/06/23 22:18:44 WARN util.Utils: Service 'SparkUI' could not bind on port 18045. Attempting port 18046.
20/06/23 22:18:44 WARN util.Utils: Service 'SparkUI' could not bind on port 18046. Attempting port 18047.
20/06/23 22:18:44 WARN util.Utils: Service 'SparkUI' could not bind on port 18047. Attempting port 18048.
20/06/23 22:18:44 WARN util.Utils: Service 'SparkUI' could not bind on port 18048. Attempting port 18049.
20/06/23 22:18:44 WARN util.Utils: Service 'SparkUI' could not bind on port 18049. Attempting port 18050.
20/06/23 22:18:44 WARN util.Utils: Service 'SparkUI' could not bind on port 18050. Attempting port 18051.
20/06/23 22:18:44 INFO server.AbstractConnector: Started ServerConnector@6f330eb9{HTTP/1.1,[http/1.1]}{0.0.0.0:18051}
20/06/23 22:18:44 INFO util.Utils: Successfully started service 'SparkUI' on port 18051.
And Job succeeded**
Failure Case
**Properties Set: spark.ui.port=18018 spark.port.maxRetries=6**
20/06/23 22:16:15 WARN util.Utils: Service 'SparkUI' could not bind on port 18018. Attempting port 18019.
20/06/23 22:16:15 WARN util.Utils: Service 'SparkUI' could not bind on port 18019. Attempting port 18020.
20/06/23 22:16:15 WARN util.Utils: Service 'SparkUI' could not bind on port 18020. Attempting port 18021.
20/06/23 22:16:15 WARN util.Utils: Service 'SparkUI' could not bind on port 18021. Attempting port 18022.
20/06/23 22:16:15 WARN util.Utils: Service 'SparkUI' could not bind on port 18022. Attempting port 18023.
20/06/23 22:16:15 WARN util.Utils: Service 'SparkUI' could not bind on port 18023. Attempting port 18024.
**20/06/23 22:16:15 ERROR ui.SparkUI: Failed to bind SparkUI
java.net.BindException: Address already in use: Service 'SparkUI' failed after 6 retries (starting from 18018)! Consider explicitly setting the appropriate port for the service 'SparkUI' (for example spark.ui.port for SparkUI) to an available port or increasing spark.port.maxRetries.
at sun.nio.ch.Net.bind0(Native Method)
Job Failed ...**
@tgravescs @Ngone51
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-650487372
**[Test build #124559 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124559/testReport)** for PR 28880 at commit [`b42c3ed`](https://github.com/apache/spark/commit/b42c3ed7498bad6d3607978c0c19979828da9ef0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28880: [SPARK-32039][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-647077548
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-647264059
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-647264059
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rajatahujaatinmobi commented on a change in pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
rajatahujaatinmobi commented on a change in pull request #28880:
URL: https://github.com/apache/spark/pull/28880#discussion_r443860777
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
##########
@@ -211,9 +211,11 @@ private[spark] class ApplicationMaster(
final def run(): Int = {
try {
val attemptID = if (isClusterMode) {
- // Set the web ui port to be ephemeral for yarn so we don't conflict with
- // other spark processes running on the same box
- System.setProperty(UI_PORT.key, "0")
+ // Set the web ui port to be ephemeral for yarn if not set explicitly
+ // so we don't conflict with other spark processes running on the same box
+ if (System.getProperty(UI_PORT.key) != null) {
+ System.setProperty(UI_PORT.key, "0")
+ }
Review comment:
Correct me if I am wrong
As per the discussions that UI port is always get assigned in the range based on spark.port.maxRetries. So we can always mention the starting port and afterward, it will try to start service based in between (starting port, starting port +spark.port.maxRetries). So we are including the range implicitly.
Copying the comment
``other port types were spawned in the range (port_mentioned to port_mentioned+spark.port.maxRetries) except the UI port which started on random port.``
We have a Knox proxy that accesses UI via RM link. And Knox proxy has a certain range of ports that it can access on the cluster which is (18000-18159). Unless we start web UI on that range, we can not access the UI. and we can achieve above the range if we set the following properties
spark.ui.port=18000
spark.port.maxRetries=159
Hence setting up a spark.ui.port as starting port is enough instead of setting up a random port number in the range.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-651990013
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-651978360
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-651978360
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-647200351
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rajatahujaatinmobi commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
rajatahujaatinmobi commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-647203127
> @rajatahujaatinmobi . This is a partial revert of [SPARK-3627](https://issues.apache.org/jira/browse/SPARK-3627) and has been discussed before. Please see [SPARK-29465](https://issues.apache.org/jira/browse/SPARK-29465).
> The last advice was `port range` might be acceptable solution.
If spark.ui.port is Set then it will start from that port and will try web ui until spark.port.maxRetries and its default value is 16. So it takes care of range.
```
spark.port.maxRetries | 16 | Maximum number of retries when binding to a port before giving up. When a port is given a specific value (non 0), each subsequent retry will increment the port used in the previous attempt by 1 before retrying. This essentially allows it to try a range of ports from the start port specified to port + maxRetries.
-- | -- | --
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rajatahujaatinmobi commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
rajatahujaatinmobi commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-647234225
> Is that the same comment with https://issues.apache.org/jira/browse/SPARK-29465?focusedCommentId=16952608&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16952608 ?
Yes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28880: [SPARK-32039][YARN][WEBUI][master] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-647077486
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-647227424
Is that the same comment with https://issues.apache.org/jira/browse/SPARK-29465?focusedCommentId=16952608&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16952608 ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-650505296
**[Test build #124562 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124562/testReport)** for PR 28880 at commit [`8fe0aa9`](https://github.com/apache/spark/commit/8fe0aa98a18edb579b7ee214b6c6dd9ba69b395b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-647230029
Seems like the change is firstly introduced in https://github.com/apache/spark/commit/c6de982be69cd50e66375cfea3d6c3267a01583b rather than SPARK-3627.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rajatahujaatinmobi commented on a change in pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
rajatahujaatinmobi commented on a change in pull request #28880:
URL: https://github.com/apache/spark/pull/28880#discussion_r446481786
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
##########
@@ -211,9 +211,11 @@ private[spark] class ApplicationMaster(
final def run(): Int = {
try {
val attemptID = if (isClusterMode) {
- // Set the web ui port to be ephemeral for yarn so we don't conflict with
- // other spark processes running on the same box
- System.setProperty(UI_PORT.key, "0")
+ // Set the web ui port to be ephemeral for yarn if not set explicitly
+ // so we don't conflict with other spark processes running on the same box
Review comment:
@tgravescs Updated my comment. If that looks okay, can we merge this request ? And I would like to backport this change until spark 2.4
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-650867794
LGTM
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28880: [SPARK-32039][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #28880:
URL: https://github.com/apache/spark/pull/28880#discussion_r443269381
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
##########
@@ -211,9 +211,11 @@ private[spark] class ApplicationMaster(
final def run(): Int = {
try {
val attemptID = if (isClusterMode) {
- // Set the web ui port to be ephemeral for yarn so we don't conflict with
- // other spark processes running on the same box
- System.setProperty(UI_PORT.key, "0")
+ // Set the web ui port to be ephemeral for yarn if not set explicitly
+ // so we don't conflict with other spark processes running on the same box
+ if (System.getProperty(UI_PORT.key) != null) {
+ System.setProperty(UI_PORT.key, "0")
+ }
Review comment:
Hi, @rajatahujaatinmobi . While reviewing this PR, I noticed that SPARK-29465 is a previous one for this issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rajatahujaatinmobi edited a comment on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
rajatahujaatinmobi edited a comment on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-652690383
> this is odd, looks like those are all R tests for some reason failing. @dongjoon-hyun have you seen that before?
@dongjoon-hyun Any comments as to how can we proceed further?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-650508322
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-651981357
**[Test build #124687 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124687/testReport)** for PR 28880 at commit [`1896905`](https://github.com/apache/spark/commit/1896905d42ee71709fa0da0a80d84fe3eb026c6e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-650508289
**[Test build #124562 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124562/testReport)** for PR 28880 at commit [`8fe0aa9`](https://github.com/apache/spark/commit/8fe0aa98a18edb579b7ee214b6c6dd9ba69b395b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-650505500
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-647258791
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-651992022
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-647258791
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-647200321
**[Test build #124342 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124342/testReport)** for PR 28880 at commit [`ce4b6ea`](https://github.com/apache/spark/commit/ce4b6ea7a4f43b776a57dec348f0265b07c6ca23).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #28880:
URL: https://github.com/apache/spark/pull/28880#discussion_r444621048
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
##########
@@ -211,9 +211,11 @@ private[spark] class ApplicationMaster(
final def run(): Int = {
try {
val attemptID = if (isClusterMode) {
- // Set the web ui port to be ephemeral for yarn so we don't conflict with
- // other spark processes running on the same box
- System.setProperty(UI_PORT.key, "0")
+ // Set the web ui port to be ephemeral for yarn if not set explicitly
+ // so we don't conflict with other spark processes running on the same box
+ if (System.getProperty(UI_PORT.key) != null) {
+ System.setProperty(UI_PORT.key, "0")
+ }
Review comment:
I don't get your point here. For the successful case, I only see the port range is 18018 ~ 18078 and the port 18051 is valid. Where are those ports 18000, 18050, 18008?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rajatahujaatinmobi commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
rajatahujaatinmobi commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-652034754
> test this please
Jenkins jobs seems to be doing fine
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124664/
> **[Test build #124685 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124685/testReport)** for PR 28880 at commit [`24ecfcf`](https://github.com/apache/spark/commit/24ecfcf1c06a4af8f15ce28e6cd9b559bba7643b).
>
> * This patch passes all tests.
>
> * This patch merges cleanly.
>
> * This patch adds no public classes.
@tgravescs Looks like all the test cases pass in the Jenkins Build. Shall we go ahead and merge it ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-650487591
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124559/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rajatahujaatinmobi commented on a change in pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
rajatahujaatinmobi commented on a change in pull request #28880:
URL: https://github.com/apache/spark/pull/28880#discussion_r444552932
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
##########
@@ -211,9 +211,11 @@ private[spark] class ApplicationMaster(
final def run(): Int = {
try {
val attemptID = if (isClusterMode) {
- // Set the web ui port to be ephemeral for yarn so we don't conflict with
- // other spark processes running on the same box
- System.setProperty(UI_PORT.key, "0")
+ // Set the web ui port to be ephemeral for yarn if not set explicitly
+ // so we don't conflict with other spark processes running on the same box
+ if (System.getProperty(UI_PORT.key) != null) {
+ System.setProperty(UI_PORT.key, "0")
+ }
Review comment:
I blocked a set of Port range from 18000 to 18050 and configure web or to start at 18008 but it eventually Started at 18051
**Properties Set: spark.ui.port=18018 spark.port.maxRetries=60**
Successful Case:
20/06/23 21:55:30 WARN util.Utils: Service 'SparkUI' could not bind on port 18035. Attempting port 18036.
20/06/23 21:55:30 WARN util.Utils: Service 'SparkUI' could not bind on port 18036. Attempting port 18037.
20/06/23 21:55:30 WARN util.Utils: Service 'SparkUI' could not bind on port 18037. Attempting port 18038.
20/06/23 21:55:30 WARN util.Utils: Service 'SparkUI' could not bind on port 18038. Attempting port 18039.
20/06/23 21:55:30 WARN util.Utils: Service 'SparkUI' could not bind on port 18039. Attempting port 18040.
20/06/23 21:55:30 WARN util.Utils: Service 'SparkUI' could not bind on port 18040. Attempting port 18041.
20/06/23 21:55:30 WARN util.Utils: Service 'SparkUI' could not bind on port 18041. Attempting port 18042.
20/06/23 21:55:30 WARN util.Utils: Service 'SparkUI' could not bind on port 18042. Attempting port 18043.
20/06/23 21:55:30 WARN util.Utils: Service 'SparkUI' could not bind on port 18043. Attempting port 18044.
20/06/23 21:55:30 WARN util.Utils: Service 'SparkUI' could not bind on port 18044. Attempting port 18045.
20/06/23 21:55:30 WARN util.Utils: Service 'SparkUI' could not bind on port 18045. Attempting port 18046.
20/06/23 21:55:30 WARN util.Utils: Service 'SparkUI' could not bind on port 18046. Attempting port 18047.
20/06/23 21:55:30 WARN util.Utils: Service 'SparkUI' could not bind on port 18047. Attempting port 18048.
20/06/23 21:55:30 WARN util.Utils: Service 'SparkUI' could not bind on port 18048. Attempting port 18049.
20/06/23 21:55:30 WARN util.Utils: Service 'SparkUI' could not bind on port 18049. Attempting port 18050.
20/06/23 21:55:30 WARN util.Utils: Service 'SparkUI' could not bind on port 18050. Attempting port 18051.
20/06/23 21:55:30 INFO server.AbstractConnector: Started ServerConnector@7ba63fe5{HTTP/1.1,[http/1.1]}{0.0.0.0:18051}
**20/06/23 21:55:30 INFO util.Utils: Successfully started service 'SparkUI' on port 18051.
And Job succeeded**
Failure Case
**Properties Set: spark.ui.port=18018 spark.port.maxRetries=6**
20/06/23 22:16:15 WARN util.Utils: Service 'SparkUI' could not bind on port 18018. Attempting port 18019.
20/06/23 22:16:15 WARN util.Utils: Service 'SparkUI' could not bind on port 18019. Attempting port 18020.
20/06/23 22:16:15 WARN util.Utils: Service 'SparkUI' could not bind on port 18020. Attempting port 18021.
20/06/23 22:16:15 WARN util.Utils: Service 'SparkUI' could not bind on port 18021. Attempting port 18022.
20/06/23 22:16:15 WARN util.Utils: Service 'SparkUI' could not bind on port 18022. Attempting port 18023.
20/06/23 22:16:15 WARN util.Utils: Service 'SparkUI' could not bind on port 18023. Attempting port 18024.
**20/06/23 22:16:15 ERROR ui.SparkUI: Failed to bind SparkUI
java.net.BindException: Address already in use: Service 'SparkUI' failed after 6 retries (starting from 18018)! Consider explicitly setting the appropriate port for the service 'SparkUI' (for example spark.ui.port for SparkUI) to an available port or increasing spark.port.maxRetries.
at sun.nio.ch.Net.bind0(Native Method)
Job Failed ...**
@tgravescs @Ngone51
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-647263966
**[Test build #124349 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124349/testReport)** for PR 28880 at commit [`10b9f7e`](https://github.com/apache/spark/commit/10b9f7e3bbe3685ccd27e2d71ac50e40b17588f9).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-650508322
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-647258478
**[Test build #124349 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124349/testReport)** for PR 28880 at commit [`10b9f7e`](https://github.com/apache/spark/commit/10b9f7e3bbe3685ccd27e2d71ac50e40b17588f9).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #28880:
URL: https://github.com/apache/spark/pull/28880
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #28880:
URL: https://github.com/apache/spark/pull/28880#discussion_r443540504
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
##########
@@ -211,9 +211,11 @@ private[spark] class ApplicationMaster(
final def run(): Int = {
try {
val attemptID = if (isClusterMode) {
- // Set the web ui port to be ephemeral for yarn so we don't conflict with
- // other spark processes running on the same box
- System.setProperty(UI_PORT.key, "0")
+ // Set the web ui port to be ephemeral for yarn if not set explicitly
+ // so we don't conflict with other spark processes running on the same box
+ if (System.getProperty(UI_PORT.key) != null) {
+ System.setProperty(UI_PORT.key, "0")
+ }
Review comment:
yes, please see the discussions there, I think a range of ports would be much better. A single port set on a multi-tenant yarn cluster could easily end up with conflicts. What requirements do you have from Knox, a certain range or ports?
Normally on yarn I would expect you to use the resource manager web ui proxy on a secure cluster so you wouldn't be accessing the UI directly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #28880:
URL: https://github.com/apache/spark/pull/28880#discussion_r444902786
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
##########
@@ -211,9 +211,11 @@ private[spark] class ApplicationMaster(
final def run(): Int = {
try {
val attemptID = if (isClusterMode) {
- // Set the web ui port to be ephemeral for yarn so we don't conflict with
- // other spark processes running on the same box
- System.setProperty(UI_PORT.key, "0")
+ // Set the web ui port to be ephemeral for yarn if not set explicitly
+ // so we don't conflict with other spark processes running on the same box
+ if (System.getProperty(UI_PORT.key) != null) {
+ System.setProperty(UI_PORT.key, "0")
+ }
Review comment:
thanks for testing that, please update the comment like my below comment requests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-647227579
cc @tgravescs
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28880: [SPARK-32039][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-647197914
**[Test build #124342 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124342/testReport)** for PR 28880 at commit [`ce4b6ea`](https://github.com/apache/spark/commit/ce4b6ea7a4f43b776a57dec348f0265b07c6ca23).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28880: [SPARK-32039][YARN][WEBUI][master] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-647197686
Thank you for moving to `master`, @rajatahujaatinmobi .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #28880:
URL: https://github.com/apache/spark/pull/28880#discussion_r444225115
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
##########
@@ -211,9 +211,11 @@ private[spark] class ApplicationMaster(
final def run(): Int = {
try {
val attemptID = if (isClusterMode) {
- // Set the web ui port to be ephemeral for yarn so we don't conflict with
- // other spark processes running on the same box
- System.setProperty(UI_PORT.key, "0")
+ // Set the web ui port to be ephemeral for yarn if not set explicitly
+ // so we don't conflict with other spark processes running on the same box
+ if (System.getProperty(UI_PORT.key) != null) {
+ System.setProperty(UI_PORT.key, "0")
+ }
Review comment:
ah yes you are correct you can get range that way. the ports originally wouldn't do the retries or at least we didn't allow configuring maxRetries.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-650487589
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-650487532
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rajatahujaatinmobi commented on a change in pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
rajatahujaatinmobi commented on a change in pull request #28880:
URL: https://github.com/apache/spark/pull/28880#discussion_r444633578
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
##########
@@ -211,9 +211,11 @@ private[spark] class ApplicationMaster(
final def run(): Int = {
try {
val attemptID = if (isClusterMode) {
- // Set the web ui port to be ephemeral for yarn so we don't conflict with
- // other spark processes running on the same box
- System.setProperty(UI_PORT.key, "0")
+ // Set the web ui port to be ephemeral for yarn if not set explicitly
+ // so we don't conflict with other spark processes running on the same box
+ if (System.getProperty(UI_PORT.key) != null) {
+ System.setProperty(UI_PORT.key, "0")
+ }
Review comment:
I blocked ports from **18000 to 18050 s**o now spark.ui.port can not get port b/w that range. But in Spark properties, I mentioned **spark.ui.port as 18018 and spark.port.maxRetries=60** so now spark service will start trying from 18018 port until **18018 + 60** in an increment way but ports are blocked until 18050 so spark.ui.port will pick up **18051** port.
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
##########
@@ -211,9 +211,11 @@ private[spark] class ApplicationMaster(
final def run(): Int = {
try {
val attemptID = if (isClusterMode) {
- // Set the web ui port to be ephemeral for yarn so we don't conflict with
- // other spark processes running on the same box
- System.setProperty(UI_PORT.key, "0")
+ // Set the web ui port to be ephemeral for yarn if not set explicitly
+ // so we don't conflict with other spark processes running on the same box
+ if (System.getProperty(UI_PORT.key) != null) {
+ System.setProperty(UI_PORT.key, "0")
+ }
Review comment:
I blocked ports from **18000 to 18050 s**o now spark.ui.port can not get port b/w that range. But in Spark properties, I mentioned **spark.ui.port as 18018 and spark.port.maxRetries=60** so now spark service will start trying from 18018 port until **18018 + 60** in an increment way but ports are blocked until 18050 so spark.ui.port will pick up **18051** port. @Ngone51
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-651981357
**[Test build #124687 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124687/testReport)** for PR 28880 at commit [`1896905`](https://github.com/apache/spark/commit/1896905d42ee71709fa0da0a80d84fe3eb026c6e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-651990024
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124687/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-647238811
cc @tgravescs FYI
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-650487582
**[Test build #124559 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124559/testReport)** for PR 28880 at commit [`b42c3ed`](https://github.com/apache/spark/commit/b42c3ed7498bad6d3607978c0c19979828da9ef0).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #28880:
URL: https://github.com/apache/spark/pull/28880#discussion_r444234809
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
##########
@@ -211,9 +211,11 @@ private[spark] class ApplicationMaster(
final def run(): Int = {
try {
val attemptID = if (isClusterMode) {
- // Set the web ui port to be ephemeral for yarn so we don't conflict with
- // other spark processes running on the same box
- System.setProperty(UI_PORT.key, "0")
+ // Set the web ui port to be ephemeral for yarn if not set explicitly
+ // so we don't conflict with other spark processes running on the same box
Review comment:
lets update the comment to say if explicitly set, it acts as a port range based on spark.port.maxRetries so make sure that is sufficiently large for your setup to avoid port conflicts.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-650505500
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #28880:
URL: https://github.com/apache/spark/pull/28880#discussion_r443287907
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
##########
@@ -211,9 +211,11 @@ private[spark] class ApplicationMaster(
final def run(): Int = {
try {
val attemptID = if (isClusterMode) {
- // Set the web ui port to be ephemeral for yarn so we don't conflict with
- // other spark processes running on the same box
- System.setProperty(UI_PORT.key, "0")
+ // Set the web ui port to be ephemeral for yarn if not set explicitly
+ // so we don't conflict with other spark processes running on the same box
+ if (System.getProperty(UI_PORT.key) != null) {
Review comment:
Shouldn't it be `==`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-651126517
rekicked the checks, once they pass I can commit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rajatahujaatinmobi commented on a change in pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
rajatahujaatinmobi commented on a change in pull request #28880:
URL: https://github.com/apache/spark/pull/28880#discussion_r446745968
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
##########
@@ -211,9 +211,11 @@ private[spark] class ApplicationMaster(
final def run(): Int = {
try {
val attemptID = if (isClusterMode) {
- // Set the web ui port to be ephemeral for yarn so we don't conflict with
- // other spark processes running on the same box
- System.setProperty(UI_PORT.key, "0")
+ // Set the web ui port to be ephemeral for yarn if not set explicitly
+ // so we don't conflict with other spark processes running on the same box
+ if (System.getProperty(UI_PORT.key) != null) {
+ System.setProperty(UI_PORT.key, "0")
+ }
Review comment:
@tgravescs I have done that. Can we proceed further?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-652053160
this is odd, looks like those are all R tests for some reason failing. @dongjoon-hyun have you seen that before?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rajatahujaatinmobi commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
rajatahujaatinmobi commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-651695582
> > rekicked the checks, once they pass I can commit
>
> continuous-integration/appveyor/pr is failing. Not sure why is that so?
> rekicked the checks, once they pass I can commit
@tgravescs can you rerun the tests and commit/merge with master if it seems okay? I can not rerun the tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rajatahujaatinmobi commented on a change in pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
rajatahujaatinmobi commented on a change in pull request #28880:
URL: https://github.com/apache/spark/pull/28880#discussion_r444825174
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
##########
@@ -211,9 +211,11 @@ private[spark] class ApplicationMaster(
final def run(): Int = {
try {
val attemptID = if (isClusterMode) {
- // Set the web ui port to be ephemeral for yarn so we don't conflict with
- // other spark processes running on the same box
- System.setProperty(UI_PORT.key, "0")
+ // Set the web ui port to be ephemeral for yarn if not set explicitly
+ // so we don't conflict with other spark processes running on the same box
+ if (System.getProperty(UI_PORT.key) != null) {
+ System.setProperty(UI_PORT.key, "0")
+ }
Review comment:
Yes, that's expected. So I think we are good to go unless someone else has any other doubts.
@Ngone51 @tgravescs
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #28880:
URL: https://github.com/apache/spark/pull/28880#discussion_r443933600
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
##########
@@ -211,9 +211,11 @@ private[spark] class ApplicationMaster(
final def run(): Int = {
try {
val attemptID = if (isClusterMode) {
- // Set the web ui port to be ephemeral for yarn so we don't conflict with
- // other spark processes running on the same box
- System.setProperty(UI_PORT.key, "0")
+ // Set the web ui port to be ephemeral for yarn if not set explicitly
+ // so we don't conflict with other spark processes running on the same box
+ if (System.getProperty(UI_PORT.key) != null) {
+ System.setProperty(UI_PORT.key, "0")
+ }
Review comment:
yea, I think yet Spark could utilize the combination of `spark.ui.port` and `spark.port.maxRetries` to perform "range port".
What else the "range port" do you refer to? @tgravescs
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rajatahujaatinmobi commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
rajatahujaatinmobi commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-652690383
> this is odd, looks like those are all R tests for some reason failing. @dongjoon-hyun have you seen that before?
@dongjoon-hyun Any updates as to how can we proceed further?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-652725685
For me, it looks passed with the following options, @tgravescs and @rajatahujaatinmobi .
```
-Phadoop-3.2 -Phive-2.3 -Pyarn -Dtest.exclude.tags=org.apache.spark.tags.ExtendedHiveTest yarn/test network-yarn/test
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-652726758
Hi, @rajatahujaatinmobi . What is your Apache JIRA id? I need to assign SPARK-29465 to you.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28880: [SPARK-32039][YARN][WEBUI][master] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-647077548
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-650487589
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-647197914
**[Test build #124342 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124342/testReport)** for PR 28880 at commit [`ce4b6ea`](https://github.com/apache/spark/commit/ce4b6ea7a4f43b776a57dec348f0265b07c6ca23).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-647258478
**[Test build #124349 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124349/testReport)** for PR 28880 at commit [`10b9f7e`](https://github.com/apache/spark/commit/10b9f7e3bbe3685ccd27e2d71ac50e40b17588f9).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-651977705
**[Test build #124685 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124685/testReport)** for PR 28880 at commit [`24ecfcf`](https://github.com/apache/spark/commit/24ecfcf1c06a4af8f15ce28e6cd9b559bba7643b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-651990013
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-651991783
**[Test build #124685 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124685/testReport)** for PR 28880 at commit [`24ecfcf`](https://github.com/apache/spark/commit/24ecfcf1c06a4af8f15ce28e6cd9b559bba7643b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-647198770
@rajatahujaatinmobi . This is a partial revert of SPARK-3627 and has been discussed before. Please see SPARK-29465.
The last advice was `port range` might be acceptable solution.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-650487372
**[Test build #124559 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124559/testReport)** for PR 28880 at commit [`b42c3ed`](https://github.com/apache/spark/commit/b42c3ed7498bad6d3607978c0c19979828da9ef0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-651810749
test this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #28880:
URL: https://github.com/apache/spark/pull/28880#discussion_r444658287
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
##########
@@ -211,9 +211,11 @@ private[spark] class ApplicationMaster(
final def run(): Int = {
try {
val attemptID = if (isClusterMode) {
- // Set the web ui port to be ephemeral for yarn so we don't conflict with
- // other spark processes running on the same box
- System.setProperty(UI_PORT.key, "0")
+ // Set the web ui port to be ephemeral for yarn if not set explicitly
+ // so we don't conflict with other spark processes running on the same box
+ if (System.getProperty(UI_PORT.key) != null) {
+ System.setProperty(UI_PORT.key, "0")
+ }
Review comment:
I see. This is what we expected, right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-651992022
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28880: [SPARK-32039][YARN][WEBUI][master] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-647077486
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28880:
URL: https://github.com/apache/spark/pull/28880#issuecomment-650487532
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org