[jira] [Updated] (SPARK-7037) Inconsistent behavior for non-spark config properties in spark-shell and spark-submit

["Sean Owen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen updated SPARK-7037:
-----------------------------
    Assignee: Cheolsoo Park

> Inconsistent behavior for non-spark config properties in spark-shell and spark-submit
> -------------------------------------------------------------------------------------
>
>                 Key: SPARK-7037
>                 URL: https://issues.apache.org/jira/browse/SPARK-7037
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Shell, Spark Submit
>    Affects Versions: 1.3.1
>         Environment: YARN
>            Reporter: Cheolsoo Park
>            Assignee: Cheolsoo Park
>            Priority: Minor
>             Fix For: 1.4.0
>
>
> When specifying non-spark properties (i.e. names don't start with {{spark.}}) in the command line and config file, {{spark-submit}} and {{spark-shell}} behave differently, causing confusion to users.
> Here is the summary-
> || || spark-submit || spark-shell ||
> || --conf k=v | silently ignored | applied |
> || spark-defaults.conf | show a warning message and ignored | show a warning message and ignored |
> I assume that ignoring non-spark properties is intentional. If so, it should always be ignored with a warning message in all cases.
> The reason why I bring this up is as follows. In my production Hadoop jobs, I set a couple of internal properties in the job config to keep track of extra information. In an attempt to do the same in Spark, I set a non-spark property in the command line and found that it doesn't work in spark-submit while it works in spark-shell. This was confusing to me at first, and I had to spend some time to fully understand all the different behaviors (as described above).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org