[jira] [Commented] (SPARK-21048) Add an option --merged-properties-file to distinguish the configuration loading behavior

["Yaqub Alwan (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-21048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16388045#comment-16388045 ] 

Yaqub Alwan commented on SPARK-21048:
-------------------------------------

Commenting here because SPARK-21023 was closed and contrary to the remark "I think this is confusing relative to any value it adds" I also have found this behaviour of completely ignoring the spark-defaults.conf to be counterintuitive (if I am being honest I would actually say I find it obnoxious, as they're hardly defaults if they get ignored when not explicitly overridden) when supplied with a properties file, and I would also like to see _some_ solution to this problem, as a application developer doesn't want or need to know about cluster level configuration in order to just set some application level properties.

I would prefer to see something like --merge-properties-with-defaults in combination with --properties-file but any implementation works. I am a little concerned seeing the resistance to having this implemented, when the current behaviour is not intuitive.

> Add an option --merged-properties-file to distinguish the configuration loading behavior
> ----------------------------------------------------------------------------------------
>
>                 Key: SPARK-21048
>                 URL: https://issues.apache.org/jira/browse/SPARK-21048
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Submit
>    Affects Versions: 2.1.1
>            Reporter: Lantao Jin
>            Priority: Minor
>
> The problem description is the same as [SPARK-21023|https://issues.apache.org/jira/browse/SPARK-21023]. But different with that ticket. The purpose is not making sure the default properties file always be loaded. Instead, just offering other option to let user choose what they want.
> {quote}
> {{\-\-properties-file}} user-specified properties file which will replace the default properties file. deprecated.
> {{\-\-replaced-properties-file}} new option which equals the {{\-\-properties-file}} but more friendly. 
> {{\-\-merged-properties-file}} user-specified properties file which will merge with the default properties file.
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org