You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "troopson (JIRA)" <ji...@apache.org> on 2017/10/06 16:24:00 UTC
[jira] [Commented] (SPARK-11697) Spark submit can't load user
config file using Standalone cluster
[ https://issues.apache.org/jira/browse/SPARK-11697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16194800#comment-16194800 ]
troopson commented on SPARK-11697:
----------------------------------
I have same problem and take for a while to figure it out. At last I set the deploy mode to client, and avoid the problem.
I think it is a common requirement to load a config file when submit a application, and SparkFiles is not so convenient, perhaps spark submit can support a variable like configfile, so that application can get the file content no matter client or cluster mode.
> Spark submit can't load user config file using Standalone cluster
> -----------------------------------------------------------------
>
> Key: SPARK-11697
> URL: https://issues.apache.org/jira/browse/SPARK-11697
> Project: Spark
> Issue Type: Bug
> Components: Deploy
> Affects Versions: 1.5.1
> Reporter: Sam Fang
>
> When I use standalone cluster, I can't read the config file (new java.io.File(configFile) , and configFile is "my_phone.conf" ) in the VM. But I can read when switch to use yarn cluster.
> /opt/spark-1.5/bin/spark-submit --class com.example.Test1 --master spark://192.168.7.1:7077 --total-executor-cores 12 --executor-memory 12g --driver-memory 4g --driver-java-options "-XX:MaxPermSize=2048m" --files /home/test/my_phone.conf /home/test/test-1.1.jar --configFile my_phone.conf
> /opt/spark-1.5/bin/spark-submit --class com.example.Test1 --master yarn-cluster --num-executors 8 --executor-memory 12g --driver-memory 4g --driver-java-options "-XX:MaxPermSize=2048m" --files /home/test/my_phone.conf /home/test/test-1.1.jar --configFile my_phone.conf
> Exception in thread "main" com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'group_by_column'
> at com.typesafe.config.impl.SimpleConfig.findKey(SimpleConfig.java:124)
> at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:145)
> at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:159)
> at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:164)
> at com.typesafe.config.impl.SimpleConfig.getString(SimpleConfig.java:206)
> at com.example.Test1.main(Test1.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672)
> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org