[jira] [Created] (SPARK-2051) In yarn.ClientBase spark.yarn.dist.* do not work

["Guoqiang Li (JIRA)" <ji...@apache.org>]

Guoqiang Li created SPARK-2051:
----------------------------------

             Summary: In yarn.ClientBase spark.yarn.dist.* do not work
                 Key: SPARK-2051
                 URL: https://issues.apache.org/jira/browse/SPARK-2051
             Project: Spark
          Issue Type: Bug
          Components: YARN
            Reporter: Guoqiang Li


  Spark configuration
{{conf/spark-defaults.conf}}:
{quote}
spark.yarn.dist.archives     /toona/conf
spark.executor.extraClassPath ./conf
spark.driver.extraClassPath  ./conf
{quote}
----

HDFS directory
{{hadoop dfs -cat /toona/conf/toona.conf}} :
{quote}
 redis.num=4
{quote}
----
The following command execution fails
{code}
YARN_CONF_DIR=/etc/hadoop/conf ./bin/spark-submit  --num-executors 2 --driver-memory 2g --executor-memory 2g --master yarn-cluster --class toona.DeployTest toona-assembly.jar  
{code}

----
The following is testing the code
{code}
package toona
import com.typesafe.config.Config
import com.typesafe.config.ConfigFactory

object DeployTest {
  def main(args: Array[String]) {
    val conf = ConfigFactory.load("toona.conf")
    val redisNum = conf.getInt("redis.num") // Here will throw an `ConfigException` exception
    assert(redisNum == 4)

  }
}
{code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)