You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@spark.apache.org by "assaf.mendelson" <as...@rsa.com> on 2016/10/26 11:30:09 UTC

Using SPARK_WORKER_INSTANCES and SPARK-15781

As of applying SPARK-15781 the documentation of SPARK_WORKER_INSTANCES have been removed. This was due to a warning in spark-submit which suggested:
WARN SparkConf:
SPARK_WORKER_INSTANCES was detected (set to '4').
This is deprecated in Spark 1.0+.

Please instead use:
- ./spark-submit with --num-executors to specify the number of executors
- Or set SPARK_EXECUTOR_INSTANCES
- spark.executor.instances to configure the number of instances in the spark config.



The problem is that there is no replacement method to launch spark standalone with multiple workers per node. The options -num-executors and SPARK_EXECUTOR_INSTANCES configure the job rather than the resource manager behavior.

If I look at the spark standalone scripts, the only way to set multiple workers per node is the use of SPARK_WORKER_INSTANCES. The fixed in SPARK-15781 fixed the documentation without solving the problem.
A possible simple solution would be to add a SPARK_STANDALONE_WORKERS variable and add it to the start-slave.sh script and update the documentation accordingly.

Am I missing something here? Should I open a new JIRA issue?
Thanks,
                Assaf




--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Using-SPARK-WORKER-INSTANCES-and-SPARK-15781-tp19571.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.