You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Guoqiang Li (JIRA)" <ji...@apache.org> on 2014/06/06 06:19:02 UTC
[jira] [Updated] (SPARK-2051) In yarn.ClientBase spark.yarn.dist.*
do not work
[ https://issues.apache.org/jira/browse/SPARK-2051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Guoqiang Li updated SPARK-2051:
-------------------------------
Description:
Spark configuration
{{conf/spark-defaults.conf}}:
{quote}
spark.yarn.dist.archives /toona/conf
spark.executor.extraClassPath ./conf
spark.driver.extraClassPath ./conf
{quote}
----
HDFS directory
{{hadoop dfs -cat /toona/conf/toona.conf}} :
{quote}
redis.num=4
{quote}
----
The following command execution fails
{code}
YARN_CONF_DIR=/etc/hadoop/conf ./bin/spark-submit --num-executors 2 --driver-memory 2g --executor-memory 2g --master yarn-cluster --class toona.DeployTest toona-assembly.jar
{code}
----
The following is the test code
{code}
package toona
import com.typesafe.config.Config
import com.typesafe.config.ConfigFactory
object DeployTest {
def main(args: Array[String]) {
val conf = ConfigFactory.load("toona.conf")
val redisNum = conf.getInt("redis.num") // Here will throw an `ConfigException` exception
assert(redisNum == 4)
}
}
{code}
was:
Spark configuration
{{conf/spark-defaults.conf}}:
{quote}
spark.yarn.dist.archives /toona/conf
spark.executor.extraClassPath ./conf
spark.driver.extraClassPath ./conf
{quote}
----
HDFS directory
{{hadoop dfs -cat /toona/conf/toona.conf}} :
{quote}
redis.num=4
{quote}
----
The following command execution fails
{code}
YARN_CONF_DIR=/etc/hadoop/conf ./bin/spark-submit --num-executors 2 --driver-memory 2g --executor-memory 2g --master yarn-cluster --class toona.DeployTest toona-assembly.jar
{code}
----
The following is testing the code
{code}
package toona
import com.typesafe.config.Config
import com.typesafe.config.ConfigFactory
object DeployTest {
def main(args: Array[String]) {
val conf = ConfigFactory.load("toona.conf")
val redisNum = conf.getInt("redis.num") // Here will throw an `ConfigException` exception
assert(redisNum == 4)
}
}
{code}
> In yarn.ClientBase spark.yarn.dist.* do not work
> ------------------------------------------------
>
> Key: SPARK-2051
> URL: https://issues.apache.org/jira/browse/SPARK-2051
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Reporter: Guoqiang Li
>
> Spark configuration
> {{conf/spark-defaults.conf}}:
> {quote}
> spark.yarn.dist.archives /toona/conf
> spark.executor.extraClassPath ./conf
> spark.driver.extraClassPath ./conf
> {quote}
> ----
> HDFS directory
> {{hadoop dfs -cat /toona/conf/toona.conf}} :
> {quote}
> redis.num=4
> {quote}
> ----
> The following command execution fails
> {code}
> YARN_CONF_DIR=/etc/hadoop/conf ./bin/spark-submit --num-executors 2 --driver-memory 2g --executor-memory 2g --master yarn-cluster --class toona.DeployTest toona-assembly.jar
> {code}
> ----
> The following is the test code
> {code}
> package toona
> import com.typesafe.config.Config
> import com.typesafe.config.ConfigFactory
> object DeployTest {
> def main(args: Array[String]) {
> val conf = ConfigFactory.load("toona.conf")
> val redisNum = conf.getInt("redis.num") // Here will throw an `ConfigException` exception
> assert(redisNum == 4)
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)