[jira] [Commented] (SPARK-2481) The environment variables SPARK_HISTORY_OPTS is covered in start-history-server.sh

["Masayoshi TSUZUKI (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-2481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14062870#comment-14062870 ] 

Masayoshi TSUZUKI commented on SPARK-2481:
------------------------------------------

spark-env.sh is loaded after these lines are evaluated. (It is loaded in the line spark-daemon.sh is called)
{code:title=sbin/start-history-server.sh|borderStyle=solid}
if [ $# != 0 ]; then
  echo "Using command line arguments for setting the log directory is deprecated. Please "
  echo "set the spark.history.fs.logDirectory configuration option instead."
  export SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS -Dspark.history.fs.logDirectory=$1"
fi
{code}

If you want to set SPARK_HISTORY_OPTS in spark-env.sh and use the deprecated option like above, you can write like following in spark-env.sh:
{code:title=spark-env.sh|borderStyle=solid}
SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS -Dspark.history.XX=XX"
{code}

This works fine in my environment.

> The environment variables SPARK_HISTORY_OPTS is covered in start-history-server.sh
> ----------------------------------------------------------------------------------
>
>                 Key: SPARK-2481
>                 URL: https://issues.apache.org/jira/browse/SPARK-2481
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>            Reporter: Guoqiang Li
>            Assignee: Guoqiang Li
>
> If we have the following code in the conf/spark-env.sh  
> {{export SPARK_HISTORY_OPTS="-DSpark.history.XX=XX"}}
> The environment variables SPARK_HISTORY_OPTS is covered in [start-history-server.sh|https://github.com/apache/spark/blob/master/sbin/start-history-server.sh]
>  
> {code}
> if [ $# != 0 ]; then
>   echo "Using command line arguments for setting the log directory is deprecated. Please "
>   echo "set the spark.history.fs.logDirectory configuration option instead."
>   export SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS -Dspark.history.fs.logDirectory=$1"
> fi
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)