You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Michael Gummelt (JIRA)" <ji...@apache.org> on 2016/07/13 21:32:20 UTC
[jira] [Commented] (SPARK-13258) --conf properties not honored in
Mesos cluster mode
[ https://issues.apache.org/jira/browse/SPARK-13258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15375797#comment-15375797 ]
Michael Gummelt commented on SPARK-13258:
-----------------------------------------
Nope, this is still a bug.
> --conf properties not honored in Mesos cluster mode
> ---------------------------------------------------
>
> Key: SPARK-13258
> URL: https://issues.apache.org/jira/browse/SPARK-13258
> Project: Spark
> Issue Type: Bug
> Components: Mesos
> Affects Versions: 1.6.0
> Reporter: Michael Gummelt
>
> Spark properties set on {{spark-submit}} via the deprecated {{SPARK_JAVA_OPTS}} are passed along to the driver, but those set via the preferred {{--conf}} are not.
> For example, this results in the URI being fetched in the executor:
> {{SPARK_JAVA_OPTS="-Dspark.mesos.uris=https://raw.githubusercontent.com/mesosphere/spark/master/README.md -Dspark.mesos.executor.docker.image=mesosphere/spark:1.6.0" ./bin/spark-submit --deploy-mode cluster --master mesos://10.0.78.140:7077 --class org.apache.spark.examples.SparkPi http://downloads.mesosphere.com.s3.amazonaws.com/assets/spark/spark-examples_2.10-1.5.0.jar}}
> This does not:
> {{SPARK_JAVA_OPTS="-Dspark.mesos.executor.docker.image=mesosphere/spark:1.6.0" ./bin/spark-submit --deploy-mode cluster --master mesos://10.0.78.140:7077 --conf spark.mesos.uris=https://raw.githubusercontent.com/mesosphere/spark/master/README.md --class org.apache.spark.examples.SparkPi http://downloads.mesosphere.com.s3.amazonaws.com/assets/spark/spark-examples_2.10-1.5.0.jar}}
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L369
> In the above line of code, you can see that SPARK_JAVA_OPTS is passed along to the driver, so those properties take effect.
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L373
> Whereas in this line of code, you see that {{--conf}} variables are set on {{SPARK_EXECUTOR_OPTS}}, which AFAICT has absolutely no effect because this env var is being set on the driver, not the executor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org