[jira] [Commented] (SPARK-2051) spark.yarn.dist.* configs are not supported in yarn-cluster mode

["Thomas Graves (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-2051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14030612#comment-14030612 ] 

Thomas Graves commented on SPARK-2051:
--------------------------------------

Note that the configs weren't originally intended to be supported in yarn-cluster mode. They were just replacing the environment variables in yarn-client mode.  We should add them for yarn-cluster mode and we should also make sure to document them.

> spark.yarn.dist.* configs are not supported in yarn-cluster mode
> ----------------------------------------------------------------
>
>                 Key: SPARK-2051
>                 URL: https://issues.apache.org/jira/browse/SPARK-2051
>             Project: Spark
>          Issue Type: Improvement
>          Components: YARN
>            Reporter: Guoqiang Li
>            Assignee: Guoqiang Li
>
>   Spark configuration
> {{conf/spark-defaults.conf}}:
> {quote}
> spark.yarn.dist.archives     /toona/conf
> spark.executor.extraClassPath ./conf
> spark.driver.extraClassPath  ./conf
> {quote}
> ----
> HDFS directory
> {{hadoop dfs -cat /toona/conf/toona.conf}} :
> {quote}
>  redis.num=4
> {quote}
> ----
> The following command execution fails
> {code}
> YARN_CONF_DIR=/etc/hadoop/conf ./bin/spark-submit  --num-executors 2 --driver-memory 2g --executor-memory 2g --master yarn-cluster --class toona.DeployTest toona-assembly.jar  
> {code}
> ----
> The following is  the test code
> {code}
> package toona
> import com.typesafe.config.Config
> import com.typesafe.config.ConfigFactory
> object DeployTest {
>   def main(args: Array[String]) {
>     val conf = ConfigFactory.load("toona.conf")
>     val redisNum = conf.getInt("redis.num") // Here will throw an `ConfigException` exception
>     assert(redisNum == 4)
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)