Running apps over a VPN

[Christopher Piggott <cp...@gmail.com>]

My setup is that I have a spark master (using the spark scheduler) and 32
workers registered with it but they are on a private network.  I can
connect to that private network via OpenVPN.

I would like to be able to run spark applications from a local (on my
desktop) IntelliJ but have them use the remote master/workers.

I thought this would allow that:

    sparkConf.set("spark.submit.deployMode", "cluster")

but when my job runs it still complains that there are not enough
resources/workers.  Connecting to the master, it shows that workers have
been assigned and are in the RUNNING state.  My local spark app doesn't
agree.  It's like the workers were assigned but the PC end doesn't know.

I can use spark-submit.sh but I was really hoping to be able to run Spark
Applications directly from IDEA.  Possible?