You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by hishamm <hi...@unige.ch> on 2015/06/28 19:44:14 UTC

spark-submit in deployment mode with the "--jars" option

Hi,

I want to deploy my application on a standalone cluster. 
Spark submit acts in strange way. When I deploy the application in
*"client"* mode, everything works well and my application can see the
additional jar files. 

Here is the command:
>   spark-submit --master spark://1.2.3.4:7077 --deploy-mode  client
> --supervise --jars $(echo /myjars/*.jar | tr ' ' ',')  --class
> com.algorithm /my/path/algorithm.jar 

However, when I submit the command in *"cluster"* deployment mode. The
driver can not see the additional jars.
I always get *java.lang.ClassNotFoundException*

Here is the command:
>   spark-submit --master spark://1.2.3.4:7077 --deploy-mode cluster
> --supervise --jars $(echo /myjars/*.jar | tr ' ' ',')  --class
> com.algorithm /my/path/algorithm.jar 


Do I miss something ?

thanks,
Hisham



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-submit-in-deployment-mode-with-the-jars-option-tp23519.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


RE: spark-submit in deployment mode with the "--jars" option

Posted by Hisham Mohamed <Hi...@unige.ch>.
Hi Akhil,

Thanks for your reply. 
Here it is

Launch Command: "/usr/lib/jvm/java-7-oracle/jre/bin/java" "-cp" "/etc/spark/:/opt/spark/lib/spark-assembly-1.4.0-hadoop2.3.0.jar:/opt/spark-1.4.0-bin-hadoop2.3/lib/datanucleus-rdbms-3.2.9.jar:/opt/spark-1.4.0-bin
-hadoop2.3/lib/datanucleus-api-jdo-3.2.6.jar:/opt/spark-1.4.0-bin-hadoop2.3/lib/datanucleus-core-3.2.10.jar" "-Xms1024M" "-Xmx1024M" "-Dspark.eventLog.enabled=true" "-Dakka.loglevel=WARNING" "-Dspark.serializer=org.apache.spark.serializer.KryoSerializer" "-Dspark.executor.memory=1024m" "-Dspark.master=spark://1.2.3.4:7077" "-Dspark.rpc.askTimeout=10" "-Dspark.app.name=com.scala.algorithm" "-Dspark.driver.supervise=true" "-Dspark.jars=,file:/root/libs/commons-codec-1.2.jar,file:/root/libs/commons-lang3-3.1.jar,file:/root/libs/commons-logging-1.1.1.jar,file:/root/libs/guava-16.0.1.jar,file:/root/libs/httpclient-4.2.5.jar,file:/root/libs/httpcore-4.2.4.jar,file:/root/libs/ivy-2.3.0.jar,file:/root/libs/joda-convert-1.6.jar,file:/root/libs/joda-time-2.3.jar,file:/root/libs/jsr166e-1.1.0.jar,file:/root/libs/kryo-3.0.2.jar,file:/root/libs/lift-json_2.10-2.6.2.jar,file:/root/libs/lz4-1.2.0.jar,file:/root/libs/metrics-core-3.0.2.jar,file:/root/libs/netty-3.9.0.Final.jar,file:/root/libs/reflectasm-1.09.jar,file:/root/libs/slf4j-api-1.7.5.jar,file:/root/libs/snappy-java-1.0.5.jar,file:/root/spark-algorithm-build/0.1.1/algorithm.jar" "-Dspark.driver.memory=1g" "-XX:MaxPermSize=128m" "org.apache.spark.deploy.worker.DriverWrapper" "akka.tcp://sparkWorker@1.2.3.4:50911/user/Worker" "/srv/spark/work/driver-20150628164945-0020/analytics-0.1.1.jar" "com.scala.algorithm"
========================================

Exception in thread "main" java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:58)
	at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
Caused by: java.lang.NoClassDefFoundError: net/liftweb/json/DefaultFormats$
	at com.scala.utilities.dal.IOUtil$.parseConfigFile(IOUtil.scala:64)
	at com.scala.algorithmFramework$.main(algorithmfm.scala:20)
	at com.scala.algorithmFramework.main(algorithmfm.scala)
	... 6 more
Caused by: java.lang.ClassNotFoundException: net.liftweb.json.DefaultFormats$
	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
	... 9 more



________________________________________
From: Akhil Das [akhil@sigmoidanalytics.com]
Sent: 29 June 2015 09:43
To: Hisham Mohamed
Cc: user@spark.apache.org
Subject: Re: spark-submit in deployment mode with the "--jars" option

Can you paste the stacktrace? Looks like you are missing few jars.

Thanks
Best Regards

On Sun, Jun 28, 2015 at 11:14 PM, hishamm <hi...@unige.ch>> wrote:
Hi,

I want to deploy my application on a standalone cluster.
Spark submit acts in strange way. When I deploy the application in
*"client"* mode, everything works well and my application can see the
additional jar files.

Here is the command:
>   spark-submit --master spark://1.2.3.4:7077<http://1.2.3.4:7077> --deploy-mode  client
> --supervise --jars $(echo /myjars/*.jar | tr ' ' ',')  --class
> com.algorithm /my/path/algorithm.jar

However, when I submit the command in *"cluster"* deployment mode. The
driver can not see the additional jars.
I always get *java.lang.ClassNotFoundException*

Here is the command:
>   spark-submit --master spark://1.2.3.4:7077<http://1.2.3.4:7077> --deploy-mode cluster
> --supervise --jars $(echo /myjars/*.jar | tr ' ' ',')  --class
> com.algorithm /my/path/algorithm.jar


Do I miss something ?

thanks,
Hisham



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-submit-in-deployment-mode-with-the-jars-option-tp23519.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org<ma...@spark.apache.org>
For additional commands, e-mail: user-help@spark.apache.org<ma...@spark.apache.org>


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org