You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Alex Dzhagriev <dz...@gmail.com> on 2016/05/03 18:11:18 UTC
--jars for mesos cluster
Hello all,
In the Mesos related spark docs (
http://spark.apache.org/docs/1.6.0/running-on-mesos.html#cluster-mode) I
found this statement:
Note that jars or python files that are passed to spark-submit should be
> URIs reachable by Mesos slaves, as the Spark driver doesn’t automatically
> upload local jars.
However, when I'm trying to submit the job using:
./spark-submit --class MyDriver --master mesos://my-mesos:7077
--deploy-mode cluster --supervise --executor-memory 2G --jars
http://3rd-party.jar http://my.jar
I don't see the --jars on the slave machine:
sh -c './bin/spark-submit --name MyDriver --master mesos://my-mesos:7077
--driver-cores 1.0 --driver-memory 1024M --class MyDriver --executor-memory
2G ./my.jar '
And of course I'm running into the ClassNotFoundException as the 3rd party
library is not there.
Can someone, please, help how to specify the --jars correctly?
Thanks, Alex.