You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Benno Staebler (JIRA)" <ji...@apache.org> on 2017/05/24 07:13:04 UTC
[jira] [Reopened] (SPARK-20853) spark.ui.reverseProxy=true leads to
hanging communication to master
[ https://issues.apache.org/jira/browse/SPARK-20853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Benno Staebler reopened SPARK-20853:
------------------------------------
Alex, you are right - the docu does states what you have explained.
Unfortunately though there are three facts:
- we have a system running on Linux/Intel without a private local network, where the mentioned settings are running fine
- I did change spark.ui.reverseProxyUrl to various http://x.x.x.x settings on the POWER machine with private local network and the master WebUI using the local or external URL still hangs.
- not using mentioned settings everything works fine, but we have no external access to the worker UIs
I will dig into the spark code myself, but currently I am out of ideas, what could match the given facts and cause the issue.
> spark.ui.reverseProxy=true leads to hanging communication to master
> -------------------------------------------------------------------
>
> Key: SPARK-20853
> URL: https://issues.apache.org/jira/browse/SPARK-20853
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 2.1.0
> Environment: ppc64le GNU/Linux, POWER8, only master node is reachable externally other nodes are in an internal network
> Reporter: Benno Staebler
> Labels: network, web-ui
>
> When *reverse proxy is enabled*
> {quote}
> spark.ui.reverseProxy=true
> spark.ui.reverseProxyUrl=/
> {quote}
> first of all any invocation of the spark master Web UI hangs forever locally (e.g. http://192.168.10.16:25001) and via external URL without any data received.
> One, sometimes two spark applications succeed without error and than workers start throwing exceptions:
> {quote}
> Caused by: java.io.IOException: Failed to connect to /192.168.10.16:25050
> {quote}
> The application dies during creation of SparkContext:
> {quote}
> 2017-05-22 16:11:23 INFO StandaloneAppClient$ClientEndpoint:54 - Connecting to master spark://node0101:25000...
> 2017-05-22 16:11:23 INFO TransportClientFactory:254 - Successfully created connection to node0101/192.168.10.16:25000 after 169 ms (132 ms spent in bootstraps)
> 2017-05-22 16:11:43 INFO StandaloneAppClient$ClientEndpoint:54 - Connecting to master spark://node0101:25000...
> 2017-05-22 16:12:03 INFO StandaloneAppClient$ClientEndpoint:54 - Connecting to master spark://node0101:25000...
> 2017-05-22 16:12:23 ERROR StandaloneSchedulerBackend:70 - Application has been killed. Reason: All masters are unresponsive! Giving up.
> 2017-05-22 16:12:23 WARN StandaloneSchedulerBackend:66 - Application ID is not initialized yet.
> 2017-05-22 16:12:23 INFO Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 25056.
> .....
> Caused by: java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem
> {quote}
> *This definitively does not happen without reverse proxy enabled!*
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org