How to add a typesafe config file which is located on HDFS to spark-submit (cluster-mode)?

[Jobs <jo...@gmail.com>]

Hi,
I have a Spark (Spark 1.5.2) application that streams data from Kafka to
HDFS. My application contains two Typesafe config files to configure certain
things like Kafka topic etc. 

Now I want to run my application with spark-submit (cluster mode) in a
cluster.
The jar file with all dependencies of my project is stored on HDFS.
As long as my config files are included in the jar file everything works
fine. But this is unpractical for testing purposes because I always have to
rebuild the jar ...

Therefore I excluded the config files of my project and I added them via
"driver-class-path". This worked on client mode but if I move the config
files now to HDFS and run my application in cluster mode it can't find the
settings. Below you can find my spark-submit command:

    /usr/local/spark/bin/spark-submit \
        --total-executor-cores 10 \
        --executor-memory 15g \
        --verbose \
        --deploy-mode cluster\
        --class com.hdp.speedlayer.SpeedLayerApp \
        --driver-class-path hdfs://iot-master:8020/user/spark/config \
        --master spark://spark-master:6066 \
        hdfs://iot-master:8020/user/spark/speed-layer-CONFIG.jar

Below you can also see the error I get when I run the driver program in
cluster mode:

    Exception in thread "main" java.lang.reflect.InvocationTargetException
    	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    	at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    	at java.lang.reflect.Method.invoke(Method.java:497)
    	at
org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:58)
    	at
org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
    Caused by: java.lang.ExceptionInInitializerError
    	at com.speedlayer.SpeedLayerApp.main(SpeedLayerApp.scala)
    	... 6 more
    Caused by: com.typesafe.config.ConfigException$Missing: No configuration
setting found for key 'application'
    	at com.typesafe.config.impl.SimpleConfig.findKey(SimpleConfig.java:124)
    	at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:145)
    	at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:159)
    	at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:164)
    ...



I already tried it with the --file parameter and I also changed the HDFS
path to "hdfs:///iot-master:8020//user//spark//config because I thought
there could be a issue with name resolving in Spark. But unfortunately that
also didn't work. Does anybody know how I can get my application run in
cluster-mode?






--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-add-a-typesafe-config-file-which-is-located-on-HDFS-to-spark-submit-cluster-mode-tp26287.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org