You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "György Süveges (JIRA)" <ji...@apache.org> on 2016/11/04 15:59:59 UTC
[jira] [Comment Edited] (SPARK-4563) Allow spark driver to bind to
different ip then advertise ip
[ https://issues.apache.org/jira/browse/SPARK-4563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15636802#comment-15636802 ]
György Süveges edited comment on SPARK-4563 at 11/4/16 3:59 PM:
----------------------------------------------------------------
+1
However, it's related to SPARK_LOCAL_IP and Spark UI, I think it's similar.
We use AWS EMR5 (Spark 2.0 over YARN)
While developing we use spark from intellij in yarn-client mode. To connect to AWS, OpenVPN is applied with biderectional routing so the executors in amazon can access the driver on the local workstation. With OpenVPN there are multiple network interfaces then, so SPARK_LOCAL_IP should be set. But it's really inconvenient to set SPARK_LOCAL_IP each time we connect to OpenVPN and get different IP.
So we decided to set it automatically from code. Since SPARK_LOCAL_IP is env var and env vars are inmutable in java, then we change the spark.driver.host corresponding java property. But as notices spark.driver.host is only second order citizen in spark, for some features only the SPARK_LOCAL_IP is taken into account. For example Spark UI listens on the host set in SPARK_LOCAL_IP, but ignores the spark.driver.host totally. I hope this improvement will fix this issue, as I saw in https://github.com/apache/spark/commit/2cd1bfa4f0c6625b0ab1dbeba2b9586b9a6a9f42
I think SPARK_LOCAL_IP and spark.driver.host should be equivalent counterparts, shouldn't they?
was (Author: gyorgy_suveges@epam.com):
+1
However, it's related to SPARK_LOCAL_IP and Spark UI, I think it's similar.
We use AWS EMR5 (Spark 2.0 over YARN)
While developing we use spark from intellij in yarn-client mode. To connect to AWS, OpenVPN is applied with biderectional routing so the executors in amazon can access the driver on the local workstation. With OpenVPN there are *multiple network interfaces* then, so SPARK_LOCAL_IP should be set. But it's really inconvenient to set SPARK_LOCAL_IP each time we connect to OpenVPN and get different IP.
So we decided to set it automatically from code. Since SPARK_LOCAL_IP is env var and env vars are inmutable in java, then we change the spark.driver.host corresponding java property. But as notices spark.driver.host is only second order citizen in spark, for some features only the SPARK_LOCAL_IP is taken into account. For example Spark UI listens on the host set in SPARK_LOCAL_IP, but ignores the spark.driver.host totally. I hope this improvement will fix this issue, as I saw in https://github.com/apache/spark/commit/2cd1bfa4f0c6625b0ab1dbeba2b9586b9a6a9f42
I think SPARK_LOCAL_IP and spark.driver.host should be equivalent counterparts, shouldn't they?
> Allow spark driver to bind to different ip then advertise ip
> ------------------------------------------------------------
>
> Key: SPARK-4563
> URL: https://issues.apache.org/jira/browse/SPARK-4563
> Project: Spark
> Issue Type: Improvement
> Components: Deploy
> Reporter: Long Nguyen
> Assignee: Marcelo Vanzin
> Priority: Minor
> Fix For: 2.1.0
>
>
> Spark driver bind ip and advertise is not configurable. spark.driver.host is only bind ip. SPARK_PUBLIC_DNS does not work for spark driver. Allow option to set advertised ip/hostname
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org