spark.master overwritten in standalone plus cluster deploy-mode

[shanson <sh...@bloomberg.net>]

Issue: 'spark.master' not updated in driver launch command when mastership
changes.

Settings: --deploy-mode cluster \
     --supervise \
     --master "spark://master1:6066,master2:6066,master3:6066" \
     --conf "spark.master=spark://master1:7077,master2:7077,master3:7077"
    ... java opts and program args


This works fine most of the time, except in the following scenario:
1. master1 is told to launch driver and picks workerA to do so.
2. master1 leaves the cluster and workerA recognizes master2 as the new
master
3. driver dies for some reason
4. workerA tries to relaunch driver with
"-Dspark.master=spark://master1:7077"
5. driver gets connection refused repeatedly and never finds the new master.

The issue seems to stem from the rest server unconditionally overwriting
spark.master witht the current master url and nothing ever refreshing it:
https://github.com/apache/spark/blob/v1.6.0/core/src/main/scala/org/apache/spark/deploy/rest/StandaloneRestServer.scala#L147

A hacky workaround that I found was to use "spark.driver.extraJavaOption" to
pass in "-Dspark.master=spark://master1:7077,master2:7077,master3:7077".
This causes the master to be properly found, although it gets a little
confusing on the command line to see spark.master specified so many times.

Does anyone know if this is purposeful or a bug?



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-master-overwritten-in-standalone-plus-cluster-deploy-mode-tp25980.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org