You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Han JU <ju...@gmail.com> on 2014/12/09 16:11:48 UTC
Submit application to spark on mesos cluster
Hi,
I have a little problem in submitting our application to a mesos cluster.
Basically the mesos cluster is configured and I'm able to have spark-shell
working correctly. Then I tried to launch our application jar (a uber, sbt
assembly jar with all deps):
bin/spark-submit --master mesos://10.192.222.232:5050 --jars
$ADDITIONAL_JARS --class my.package.BenchmarkDriver
file:///home/jclouds/spark-benchmark-assembly-0.1.0-SNAPSHOT.jar
--application_configs
So you can see I follow the documentation, provide the application jar in
file://... format and I make sure that the application jar is available in
this path in each worker of the cluster. Plus I provide the application jar
with --jars. However there's always ClassNotFound exception in worker:
java.lang.ClassNotFoundException: my.package.UrlLink
or, if I tried custom kryo serializer:
org.apache.spark.SparkException: Failed to invoke
my.package.CustomKryoRegistrator
I've tried using hdfs://... for the application jar, but it seems ignored
completely by spark-submit.
I'm using spark 1.1.1 on hadoop 2.4.
Any suggestions? How should I submit the application jar?
--
*JU Han*
Data Engineer @ Botify.com
+33 0619608888