You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Eugen Cepoi <ce...@gmail.com> on 2014/06/03 16:47:33 UTC
spark 1.0 not using properties file from SPARK_CONF_DIR
Is it on purpose that when setting SPARK_CONF_DIR spark submit still loads
the properties file from SPARK_HOME/conf/spark-defauls.conf ?
IMO it would be more natural to override what is defined in SPARK_HOME/conf
by SPARK_CONF_DIR when defined (and SPARK_CONF_DIR being overriden by
command line args).
Eugen
Re: spark 1.0 not using properties file from SPARK_CONF_DIR
Posted by Eugen Cepoi <ce...@gmail.com>.
Here is the PR https://github.com/apache/spark/pull/997
2014-06-03 19:24 GMT+02:00 Patrick Wendell <pw...@gmail.com>:
> You can set an arbitrary properties file by adding --properties-file
> argument to spark-submit. It would be nice to have spark-submit also
> look in SPARK_CONF_DIR as well by default. If you opened a JIRA for
> that I'm sure someone would pick it up.
>
> On Tue, Jun 3, 2014 at 7:47 AM, Eugen Cepoi <ce...@gmail.com> wrote:
> > Is it on purpose that when setting SPARK_CONF_DIR spark submit still
> loads
> > the properties file from SPARK_HOME/conf/spark-defauls.conf ?
> >
> > IMO it would be more natural to override what is defined in
> SPARK_HOME/conf
> > by SPARK_CONF_DIR when defined (and SPARK_CONF_DIR being overriden by
> > command line args).
> >
> > Eugen
>
Re: spark 1.0 not using properties file from SPARK_CONF_DIR
Posted by Patrick Wendell <pw...@gmail.com>.
You can set an arbitrary properties file by adding --properties-file
argument to spark-submit. It would be nice to have spark-submit also
look in SPARK_CONF_DIR as well by default. If you opened a JIRA for
that I'm sure someone would pick it up.
On Tue, Jun 3, 2014 at 7:47 AM, Eugen Cepoi <ce...@gmail.com> wrote:
> Is it on purpose that when setting SPARK_CONF_DIR spark submit still loads
> the properties file from SPARK_HOME/conf/spark-defauls.conf ?
>
> IMO it would be more natural to override what is defined in SPARK_HOME/conf
> by SPARK_CONF_DIR when defined (and SPARK_CONF_DIR being overriden by
> command line args).
>
> Eugen