You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Alan Braithwaite <al...@cloudflare.com> on 2015/08/07 02:21:47 UTC
Spark-submit fails when jar is in HDFS
Hi All,
We're trying to run spark with mesos and docker in client mode (since mesos
doesn't support cluster mode) and load the application Jar from HDFS. The
following is the command we're running:
/usr/local/spark/bin/spark-submit --master mesos://mesos.master:5050 --conf
spark.mesos.executor.docker.image=docker.repo/spark:latest --class
org.apache.spark.examples.SparkPi
hdfs://hdfs1/tmp/spark-examples-1.4.1-hadoop2.6.0-cdh5.4.4.jar 100
We're getting the following warning before an exception from that command:
Warning: Skip remote jar
hdfs://hdfs1/tmp/spark-examples-1.4.1-hadoop2.6.0-cdh5.4.4.jar.
java.lang.ClassNotFoundException: org.apache.spark.examples.SparkPi
Before I debug further, is this even supported? I started reading the code
and it wasn't clear that it's possible to load a remote jar in client mode
at all. I did see a related issue in [2] but it didn't quite clarify
everything I was looking for.
Thanks,
- Alan
[1] https://spark.apache.org/docs/latest/submitting-applications.html
[2]
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-submit-not-working-when-application-jar-is-in-hdfs-td21840.html
Re: Spark-submit fails when jar is in HDFS
Posted by Alan Braithwaite <al...@cloudflare.com>.
>
> Did you try this way?
/usr/local/spark/bin/spark-submit --master mesos://mesos.master:5050 --conf
> spark.mesos.executor.docker.image=docker.repo/spark:latest --class
> org.apache.spark.examples.SparkPi --jars
> hdfs://hdfs1/tmp/spark-examples-1.4.1-hadoop2.6.0-cdh5.4.4.jar 100
I did, and got the same error (I verified again right now too).
Also, Spark on Mesos supports cluster mode:
> http://spark.apache.org/docs/latest/running-on-mesos.html#cluster-mode
Oh cool! Looks like this page needs to be updated then:
http://spark.apache.org/docs/latest/submitting-applications.html
Thanks!
- Alan
Re: Spark-submit fails when jar is in HDFS
Posted by Dean Wampler <de...@gmail.com>.
Also, Spark on Mesos supports cluster mode:
http://spark.apache.org/docs/latest/running-on-mesos.html#cluster-mode
Dean Wampler, Ph.D.
Author: Programming Scala, 2nd Edition
<http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
Typesafe <http://typesafe.com>
@deanwampler <http://twitter.com/deanwampler>
http://polyglotprogramming.com
On Sun, Aug 9, 2015 at 4:30 AM, Akhil Das <ak...@sigmoidanalytics.com>
wrote:
> Did you try this way?
>
> /usr/local/spark/bin/spark-submit --master mesos://mesos.master:5050
> --conf spark.mesos.executor.docker.image=docker.repo/spark:latest --class
> org.apache.spark.examples.SparkPi *--jars hdfs://hdfs1/tmp/spark-*
> *examples-1.4.1-hadoop2.6.0-**cdh5.4.4.jar* 100
>
> Thanks
> Best Regards
>
> On Fri, Aug 7, 2015 at 5:51 AM, Alan Braithwaite <al...@cloudflare.com>
> wrote:
>
>> Hi All,
>>
>> We're trying to run spark with mesos and docker in client mode (since
>> mesos doesn't support cluster mode) and load the application Jar from
>> HDFS. The following is the command we're running:
>>
>> /usr/local/spark/bin/spark-submit --master mesos://mesos.master:5050
>> --conf spark.mesos.executor.docker.image=docker.repo/spark:latest --class
>> org.apache.spark.examples.SparkPi
>> hdfs://hdfs1/tmp/spark-examples-1.4.1-hadoop2.6.0-cdh5.4.4.jar 100
>>
>> We're getting the following warning before an exception from that command:
>>
>> Warning: Skip remote jar
>> hdfs://hdfs1/tmp/spark-examples-1.4.1-hadoop2.6.0-cdh5.4.4.jar.
>> java.lang.ClassNotFoundException: org.apache.spark.examples.SparkPi
>>
>> Before I debug further, is this even supported? I started reading the
>> code and it wasn't clear that it's possible to load a remote jar in client
>> mode at all. I did see a related issue in [2] but it didn't quite clarify
>> everything I was looking for.
>>
>> Thanks,
>> - Alan
>>
>> [1] https://spark.apache.org/docs/latest/submitting-applications.html
>>
>> [2]
>> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-submit-not-working-when-application-jar-is-in-hdfs-td21840.html
>>
>
>
Re: Spark-submit fails when jar is in HDFS
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
Did you try this way?
/usr/local/spark/bin/spark-submit --master mesos://mesos.master:5050 --conf
spark.mesos.executor.docker.image=docker.repo/spark:latest --class
org.apache.spark.examples.SparkPi *--jars hdfs://hdfs1/tmp/spark-*
*examples-1.4.1-hadoop2.6.0-**cdh5.4.4.jar* 100
Thanks
Best Regards
On Fri, Aug 7, 2015 at 5:51 AM, Alan Braithwaite <al...@cloudflare.com>
wrote:
> Hi All,
>
> We're trying to run spark with mesos and docker in client mode (since
> mesos doesn't support cluster mode) and load the application Jar from
> HDFS. The following is the command we're running:
>
> /usr/local/spark/bin/spark-submit --master mesos://mesos.master:5050
> --conf spark.mesos.executor.docker.image=docker.repo/spark:latest --class
> org.apache.spark.examples.SparkPi
> hdfs://hdfs1/tmp/spark-examples-1.4.1-hadoop2.6.0-cdh5.4.4.jar 100
>
> We're getting the following warning before an exception from that command:
>
> Warning: Skip remote jar
> hdfs://hdfs1/tmp/spark-examples-1.4.1-hadoop2.6.0-cdh5.4.4.jar.
> java.lang.ClassNotFoundException: org.apache.spark.examples.SparkPi
>
> Before I debug further, is this even supported? I started reading the
> code and it wasn't clear that it's possible to load a remote jar in client
> mode at all. I did see a related issue in [2] but it didn't quite clarify
> everything I was looking for.
>
> Thanks,
> - Alan
>
> [1] https://spark.apache.org/docs/latest/submitting-applications.html
>
> [2]
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-submit-not-working-when-application-jar-is-in-hdfs-td21840.html
>