Re: Do spark-submit overwrite the Spark session created manually?

[Neo Chien <se...@gmail.com>]

Hi

According to official spec as below, I think the SparkSession.builder gets
the highest priority to the configuration and it means the ‘spark-submit’
passing options would be ignored. Please correct me if I am wrong, many
thanks.

Properties set directly on the SparkConf take highest precedence, then
flags passed to spark-submit or spark-shell, then options in the
spark-defaults.conf file.

https://spark.apache.org/docs/latest/configuration.html

Hope this helps,
Neo


<em...@yeikel.com> 於 2019年1月1日 週二 上午6:04寫道：

> Hi Community ,
>
>
>
> When we submit a job using ‘spark-submit’ passing options like the ‘master
> url’ what should be the content of the main class?
>
>
>
> For example , if I create the session myself :
>
>
>
>     val spark = SparkSession.builder.
>
>       master("local[*]")
>
>       .appName("Console")
>
>       .config("spark.app.id", "spark-es")
>
>       .getOrCreate()
>
>
>
> Will spark-submit overwrite parameters like the master url , name , etc?
>
>
>
> Thank you
>



[image: Mailtrack]
<https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality5&>
Sender
notified by
Mailtrack
<https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality5&>
19/01/01
下午3:43:08