You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Udbhav Agrawal (JIRA)" <ji...@apache.org> on 2019/04/05 08:53:00 UTC
[jira] [Commented] (SPARK-27289) spark-submit explicit
configuration does not take effect but Spark UI shows it's effective
[ https://issues.apache.org/jira/browse/SPARK-27289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16810627#comment-16810627 ]
Udbhav Agrawal commented on SPARK-27289:
----------------------------------------
[~KaiXu] for me it is coming correct and could not reproduce this
> spark-submit explicit configuration does not take effect but Spark UI shows it's effective
> ------------------------------------------------------------------------------------------
>
> Key: SPARK-27289
> URL: https://issues.apache.org/jira/browse/SPARK-27289
> Project: Spark
> Issue Type: Bug
> Components: Deploy, Documentation, Spark Submit, Web UI
> Affects Versions: 2.3.3
> Reporter: KaiXu
> Priority: Minor
> Attachments: Capture.PNG
>
>
> The [doc |https://spark.apache.org/docs/latest/submitting-applications.html]says that "In general, configuration values explicitly set on a {{SparkConf}} take the highest precedence, then flags passed to {{spark-submit}}, then values in the defaults file", but when setting spark.local.dir through --conf with spark-submit, it still uses the values from ${SPARK_HOME}/conf/spark-defaults.conf, what's more, the Spark runtime UI environment variables shows the value from --conf, which is really misleading.
> e.g.
> I set submit my application through the command:
> /opt/spark233/bin/spark-submit --properties-file /opt/spark.conf --conf spark.local.dir=/tmp/spark_local -v --class org.apache.spark.examples.mllib.SparseNaiveBayes --master spark://bdw-slave20:7077 /opt/sparkbench/assembly/target/sparkbench-assembly-7.1-SNAPSHOT-dist.jar hdfs://bdw-slave20:8020/Bayes/Input
>
> the spark.local.dir in ${SPARK_HOME}/conf/spark-defaults.conf is:
> spark.local.dir=/mnt/nvme1/spark_local
> when the application is running, I found the intermediate shuffle data was wrote to /mnt/nvme1/spark_local, which is set through ${SPARK_HOME}/conf/spark-defaults.conf, but the Web UI shows that the environment value spark.local.dir=/tmp/spark_local.
> The spark-submit verbose also shows spark.local.dir=/tmp/spark_local, it's misleading.
>
> !image-2019-03-27-10-59-38-377.png!
> spark-submit verbose:
> XXXX
> Spark properties used, including those specified through
> --conf and those from the properties file /opt/spark.conf:
> (spark.local.dir,/tmp/spark_local)
> (spark.default.parallelism,132)
> (spark.driver.memory,10g)
> (spark.executor.memory,352g)
> XXXXX
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org