You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Aris Vlasakakis <ar...@vlasakakis.com> on 2014/07/11 01:04:12 UTC
Submitting to a cluster behind a VPN, configuring different IP address
Hi Spark folks,
So on our production Spark cluster, it lives in the data center and I need
to attach to a VPN from my laptop, so that I can then submit a Spark
application job to the Spark Master (behind the VPN).
However, the problem arises that I have a local IP address on the laptop
which is on a separate network segment than the VPN. I figured out that I
can set the SparkConf configuration property called "spark.driver.host" to
be my VPN IP address - this got me further, because I could initiate a
connection to the master.
However, it was still not working. I was hoping to set the SparkConf
variables "spark.fileserver.uri" and "spark.httpBroadcast.uri" to NOT be my
local IP addresses, but the VPN IP addresses, because I think these
configuration variables actually pass my application files to the master
(in this case, it is a python script). So my spark.fileserver.uri and the
Spark Master need to be on the same network segment (the VPN subnetwork ).
Am I on the right track? How can I set "spark.fileserver.uri" and
"spark.httpBroadcast.uri" ? I see that these are actually run by Jetty
server...any thoughts?
Thank you so much!
--
Άρης Βλασακάκης
Aris Vlasakakis
Re: Submitting to a cluster behind a VPN, configuring different IP address
Posted by jay vyas <ja...@gmail.com>.
yup ....a related JIRA is here
https://issues.apache.org/jira/browse/SPARK-5113 which you might want to
leave a comment in. This can be quite tricky we found ! but there are a
host of env variable hacks you can use when launching spark masters/slaves.
On Thu, Apr 2, 2015 at 5:18 PM, Michael Quinlan <mq...@gmail.com> wrote:
> I was able to hack this on my similar setup issue by running (on the
> driver)
>
> $ sudo hostname ip
>
> Where ip is the same value set in the "spark.driver.host" property. This
> isn't a solution I would use universally and hope the someone can fix this
> bug in the distribution.
>
> Regards,
>
> Mike
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Submitting-to-a-cluster-behind-a-VPN-configuring-different-IP-address-tp9360p22363.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
--
jay vyas
Re: Submitting to a cluster behind a VPN, configuring different IP
address
Posted by TimMalt <dw...@gmx.net>.
Hi, and what can I do when I am on Windows?
It does not allow me to set the hostname to some IP....
Thanks,
Tim
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Submitting-to-a-cluster-behind-a-VPN-configuring-different-IP-address-tp9360p22674.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Submitting to a cluster behind a VPN, configuring different IP
address
Posted by Michael Quinlan <mq...@gmail.com>.
I was able to hack this on my similar setup issue by running (on the driver)
$ sudo hostname ip
Where ip is the same value set in the "spark.driver.host" property. This
isn't a solution I would use universally and hope the someone can fix this
bug in the distribution.
Regards,
Mike
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Submitting-to-a-cluster-behind-a-VPN-configuring-different-IP-address-tp9360p22363.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Submitting to a cluster behind a VPN, configuring different IP
address
Posted by nunarob <ro...@nunahealth.com>.
Hi,
I'm having the exact same problem - I'm on a VPN and I'm trying to set the
proproperties spark.httpBroadcast.uri and spark.fileserver.uri so that they
bind to my VPN ip instead of my regular network IP. Were you ever able to
get this working?
Cheers,
-Rob
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Submitting-to-a-cluster-behind-a-VPN-configuring-different-IP-address-tp9360p11560.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Submitting to a cluster behind a VPN, configuring different IP address
Posted by Aris Vlasakakis <ar...@vlasakakis.com>.
Hello!
Just curious if anybody could respond to my original message, if anybody
knows about how to set the configuration variables that are handles by
Jetty and not Spark's native framework..which is Akka I think?
Thanks
On Thu, Jul 10, 2014 at 4:04 PM, Aris Vlasakakis <ar...@vlasakakis.com>
wrote:
> Hi Spark folks,
>
> So on our production Spark cluster, it lives in the data center and I need
> to attach to a VPN from my laptop, so that I can then submit a Spark
> application job to the Spark Master (behind the VPN).
>
> However, the problem arises that I have a local IP address on the laptop
> which is on a separate network segment than the VPN. I figured out that I
> can set the SparkConf configuration property called "spark.driver.host" to
> be my VPN IP address - this got me further, because I could initiate a
> connection to the master.
>
> However, it was still not working. I was hoping to set the SparkConf
> variables "spark.fileserver.uri" and "spark.httpBroadcast.uri" to NOT be my
> local IP addresses, but the VPN IP addresses, because I think these
> configuration variables actually pass my application files to the master
> (in this case, it is a python script). So my spark.fileserver.uri and the
> Spark Master need to be on the same network segment (the VPN subnetwork ).
>
> Am I on the right track? How can I set "spark.fileserver.uri" and
> "spark.httpBroadcast.uri" ? I see that these are actually run by Jetty
> server...any thoughts?
>
> Thank you so much!
>
> --
> Άρης Βλασακάκης
> Aris Vlasakakis
>
--
Άρης Βλασακάκης
Aris Vlasakakis