You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Stavros Kontopoulos (JIRA)" <ji...@apache.org> on 2017/08/31 13:39:00 UTC
[jira] [Commented] (SPARK-13258) --conf properties not honored in
Mesos cluster mode
[ https://issues.apache.org/jira/browse/SPARK-13258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16148982#comment-16148982 ]
Stavros Kontopoulos commented on SPARK-13258:
---------------------------------------------
SPARK_JAVA_OPTS has been removed from the code base (https://issues.apache.org/jira/browse/SPARK-14453). I verified that this is not a bug anymore. Ssh'd to a node in a dc/os cluster and run a job in cluster mode:
./bin/spark-submit --verbose --deploy-mode cluster --master mesos://spark.marathon.mesos:14003 --conf spark.mesos.uris=https://raw.githubusercontent.com/mesosphere/spark/master/README.md --conf spark.executor.memory=1g --conf spark.executor.cores=2 --conf spark.cores.max=4 --conf spark.mesos.executor.home=/opt/spark/dist --conf spark.mesos.executor.docker.image=mesosphere/spark:1.1.1-2.2.0-hadoop-2.6 --class org.apache.spark.examples.SparkPi https://s3-eu-west-1.amazonaws.com/fdp-stavros-test/spark-examples_2.11-2.1.1.jar 1000
By looking at the driver sandbox the README is fetched:
I0831 13:24:54.727625 11257 fetcher.cpp:580] Fetched 'https://raw.githubusercontent.com/mesosphere/spark/master/README.md' to '/var/lib/mesos/slave/slaves/53b59f05-430e-4837-aa14-658ca13a82d3-S0/frameworks/53b59f05-430e-4837-aa14-658ca13a82d3-0002/executors/driver-20170831132454-0007/runs/54527349-52de-49e2-b090-9587e4547b99/README.md'
I0831 13:24:54.976838 11268 exec.cpp:162] Version: 1.2.2
I0831 13:24:54.983006 11275 exec.cpp:237] Executor registered on agent 53b59f05-430e-4837-aa14-658ca13a82d3-S0
17/08/31 13:24:56 INFO SparkContext: Running Spark version 2.2.0
Also looking at the executor's sandbox:
W0831 13:24:58.799739 11234 fetcher.cpp:322] Copying instead of extracting resource from URI with 'extract' flag, because it does not seem to be an archive: https://raw.githubusercontent.com/mesosphere/spark/master/README.md
I0831 13:24:58.799767 11234 fetcher.cpp:580] Fetched 'https://raw.githubusercontent.com/mesosphere/spark/master/README.md' to '/var/lib/mesos/slave/slaves/53b59f05-430e-4837-aa14-658ca13a82d3-S2/frameworks/53b59f05-430e-4837-aa14-658ca13a82d3-0002-driver-20170831132454-0007/executors/3/runs/8aa9a13d-c6a6-4449-b37d-8b8ab32339d2/README.md'
I0831 13:24:59.033010 11244 exec.cpp:162] Version: 1.2.2
I0831 13:24:59.038556 11246 exec.cpp:237] Executor registered on agent 53b59f05-430e-4837-aa14-658ca13a82d3-S2
17/08/31 13:25:00 INFO CoarseGrainedExecutorBackend: Started daemon with process name: 7@ip-10-0-2-43.eu-west-1.compute.internal
[~susanxhuynh] [~arand] [~sowen] Not an issue anymore for versions >= 2.2.0 you just need to use the --conf option.
> --conf properties not honored in Mesos cluster mode
> ---------------------------------------------------
>
> Key: SPARK-13258
> URL: https://issues.apache.org/jira/browse/SPARK-13258
> Project: Spark
> Issue Type: Bug
> Components: Mesos
> Affects Versions: 1.6.0
> Reporter: Michael Gummelt
>
> Spark properties set on {{spark-submit}} via the deprecated {{SPARK_JAVA_OPTS}} are passed along to the driver, but those set via the preferred {{--conf}} are not.
> For example, this results in the URI being fetched in the executor:
> {{SPARK_JAVA_OPTS="-Dspark.mesos.uris=https://raw.githubusercontent.com/mesosphere/spark/master/README.md -Dspark.mesos.executor.docker.image=mesosphere/spark:1.6.0" ./bin/spark-submit --deploy-mode cluster --master mesos://10.0.78.140:7077 --class org.apache.spark.examples.SparkPi http://downloads.mesosphere.com.s3.amazonaws.com/assets/spark/spark-examples_2.10-1.5.0.jar}}
> This does not:
> {{SPARK_JAVA_OPTS="-Dspark.mesos.executor.docker.image=mesosphere/spark:1.6.0" ./bin/spark-submit --deploy-mode cluster --master mesos://10.0.78.140:7077 --conf spark.mesos.uris=https://raw.githubusercontent.com/mesosphere/spark/master/README.md --class org.apache.spark.examples.SparkPi http://downloads.mesosphere.com.s3.amazonaws.com/assets/spark/spark-examples_2.10-1.5.0.jar}}
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L369
> In the above line of code, you can see that SPARK_JAVA_OPTS is passed along to the driver, so those properties take effect.
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L373
> Whereas in this line of code, you see that {{--conf}} variables are set on {{SPARK_EXECUTOR_OPTS}}, which AFAICT has absolutely no effect because this env var is being set on the driver, not the executor.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org